Introduction

The Delphi technique is a method for collecting and organizing informed opinions from a group of experts who are knowledgeable in a specialized area [3]. The name was derived from Greek mythology: the oracle of Delphi, valued for its wisdom, wealth, and culture, became Apollo’s center for forecasting the future and was considered most truthful [17, 24].

Using the Delphi method, a panel of experts is questioned about specific items or issues, usually involving several iterations (‘rounds’) of a structured questionnaire [13]. Information is collected individually from each expert so that the responses are anonymous, and participation is kept confidential. The anonymous responses are analyzed and summarized by the researcher and returned to the respondents for further consideration and response [3]. The eventual outcome is to obtain converging consensus on a given subject [3]. Subsequent iterations of the questionnaires generally include items on which consensus was not reached, accompanied by the corresponding anonymous feedback [6]. The replies are collated again, and the process repeated until consensus is reached [25].

The “standard Delphi format” is described as consisting of four rounds of anonymous surveys with a single expert panel [3]. However, there are no formal, universally agreed-upon guidelines for the application of the Delphi method, and several variations have been described regarding the process of selection of experts, the size of the expert panel, and the definition of consensus [16]. Thus, the design and format of the technique often vary depending on the objectives and circumstances of a study, which has led to an array of heterogeneous studies in the literature.

The Delphi technique was initially developed in 1953 to obtain consensus within a group of military defense experts on military strategies [3]. Since its inception, it has been used in diverse fields of government, industry, business, education, and health for a range of purposes. Common examples are: for a group to obtain broad input and consensus on defining its research priorities [5]; to develop clinical practice guidelines [20]; to identify education priorities [9]; and to develop questionnaires, surveys, or instruments [10, 15].

Within the measurement field, the Delphi technique is a well-recognized and commonly used approach for gathering validity evidence using expert panels. Review panels are typically asked to evaluate an instrument in terms of predetermined criteria, including relevancy and representativeness of the items, testing procedures, administrative conditions, and item-scoring criteria [19]. The Delphi technique is particularly useful when collective subjective assessments are needed [18], it is difficult to organize face-to-face meetings [18], or individual opinions may be masked by more vocal subgroups [7, 24].

The Delphi method in palliative care research: a review of the published literature

Despite its extensive use in the measurement area, the application of the Delphi method to palliative care tool development is recent. The authors conducted a literature review to identify studies that used the Delphi method in palliative care research. The databases MEDLINE, CINAHL, EMBASE, and PsychINFO were searched in October 2006 using the following terms: (a) “Delphi” + “palliative”, (b) “Delphi” + “cancer pain”, (c) “Delphi” + “cancer” + “pain”, and (d) “Delphi” + “end of life”. The search identified 44 unique papers. Abstracts were reviewed and coded independently by two of the authors to identify how the Delphi process was used in each study; full articles were obtained and reviewed when necessary. The following inclusion criteria were used: (a) original research (b) use of the Delphi technique, and (c) study pertaining to palliative care. Papers were excluded from the sample if they did not include a Delphi study (n = 1), did not fall within the realm of palliative care (n = 2), or were not original research (n = 1). Of the 40 remaining papers, the Delphi method was used for a variety of purposes (Table 1).

Table 1 Summary of use of the Delphi method in palliative care research based on a literature review of published articles (n = 40)
Full size table

Only 2 of the 40 studies used the Delphi method for the purpose of tool development in palliative care [21, 22]. In the study by Nekolaichuk et al. [21], the Delphi method was used to gather validity evidence for a pain classification system, using two expert panels. Oxenham et al. [22] described a modified Delphi technique for the development of a clinical audit tool for monitoring cancer-pain management in a community-based sample. Thus, the Delphi process has been used for a range of purposes in palliative care but not commonly for tool development.

The purpose of this manuscript is to (a) report on the authors’ experience with using the Delphi technique in the development of two pain assessment measures in the palliative-care setting; (b) identify challenges in applying this technique including sampling, study design, and consensus setting; and (c) suggest approaches that can add to its effectiveness in supporting national and international collaboration in palliative-care research. The intended outcome of this report is to support the uptake of the Delphi technique to further promote international collaboration in tool development in palliative care.

The Delphi method in palliative care assessment tool validation: case studies

In response to the lack of widely accepted, reliable, validated assessment tools in the area of cancer pain, investigators belonging to a Canadian Institutes of Health Research New Emerging Team in Difficult Cancer Pain have developed two pain-assessment tools: the Edmonton Classification System for Cancer Pain (ECS-CP) for classifying cancer pain and the Alberta Breakthrough Pain Assessment Tool for Research (ABPAT-R) to support research in breakthrough pain. Both tools are available at http://www.cancerpainnet.ca/. The ECS-CP (originally called the Edmonton Staging System) was developed as a pain-classification system for cancer-pain management to characterize the various elements that contribute to the complexity of a given pain syndrome and predict the likelihood of achieving good pain control [11, 21]. The ECS-CP consists of five features: mechanism of pain, incident pain, psychological distress, addictive behavior, and cognitive function. The ABPAT-R was developed as a data-acquisition tool to support a program of clinical research evaluating novel interventions for breakthrough pain [12]. However, its sensitivity to change has not yet been validated for that purpose. It is anticipated that with further use and evaluation, the tool will be shortened and adapted for clinical use.

Implementation of the Delphi technique

As part of the validation process, the Delphi technique involving expert panel review was used to gather content-related evidence of construct validity for each of these tools [12, 21]. Table 2 outlines the stepwise approach that was used in the application of the Delphi process for obtaining validity evidence for these two instruments.

Table 2 A case illustration of a stepwise approach for gathering validity evidence using the Delphi technique
Full size table

Timelines for applying the Delphi technique

A timeline for planning and conducting the ECS-CP study appears in Fig. 1. The overall time for study completion was approximately 14 months. During this time period, the research team members met on a regular (usually every 2 weeks) basis to review the study process, synthesize findings, revise the instrument and manual, and prepare surveys for the next iteration. A timeline for planning and conducting the ABPAT-R study is summarized in Fig. 2. In total, planning and conducting the study took approximately 6 months from start (beginning with initial discussions on how to implement the methodology) to completion (analysis of results from the international panel review and revision of the ABPAT-R). The study team met via teleconference approximately one to two times per month throughout the planning and conduct of the study to discuss findings and to revise the ABPAT-R and the evaluation survey. Meetings were fairly brief (30–60 min) and were guided by a focused, action-oriented agenda. We found the use of Gantt charts [1] provided motivation and helped team members to adhere to the study timelines. These charts provided a visual guide for groups when planning sequences of events (Figs. 1 and 2).

Fig. 1
figure 1

Gantt chart of ECS-CP study timelines

Full size image
Fig. 2
figure 2

Gantt chart of ABPAT-R study timelines

Full size image

The Delphi method in palliative care assessment tool validation: challenges and lessons learned

In the development of the ECS-CP [21] and the ABPAT-R [12], we encountered a number of challenges in applying the Delphi technique. The following is a summary of these challenges related to sampling, study and survey design, consensus setting, and response rates.

Sampling

In the ECS-CP study, the use of two expert panels strengthened the validation process, as the first panel was more familiar with the use of an earlier version of the instrument clinically, whereas the second panel was less likely to have had this level of familiarity. In both the ECS-CP and ABPAT-R studies, the involvement of national and international panel members provided input regarding the use of these tools beyond the regional level and offered opportunities to identify potentially culturally sensitive issues and differences across palliative settings around the world. We were surprised to note that international involvement resulted in spontaneous expression of interest and collaboration in a follow-up international validation study of the ECS-CP in different countries and practice settings, which is now currently in progress. The sizes of the panels in the ECS-CP study (i.e. n = 18 and n = 52) and the ABPAT-R study (n = 16 and n = 22) were moderate to large for a Delphi process, potentially making it more difficult to achieve consensus. However, the selection was based on what we judged appropriate to adequately represent expertise regionally, nationally, and internationally, which we believed to strengthen the findings. It turned out that consensus was readily achieved in both studies.

Study design

In the ECS-CP study, both a mail-out and e-mail format of the evaluation survey were sent to panel members. A letter of introduction along with the survey was sent to each identified expert of the national panel (panel A). The same process was followed for the international panel (panel B). The majority of participants opted to return their responses by mail. Response rates were 67% (panel A, first round), 39% (panel A, second round), 56% (panel B, first round), and 64% (panel B, second round). The limitation of this approach was that it added to the turn-around time for receiving completed mail-out surveys, particularly for participants in distant geographical locations. In the ABPAT-R study, evaluation surveys were sent to each identified expert of the national panel with an invitation to participate. International panel members were first queried for their interest in and commitment to participating in the review process; then, surveys were sent only to those panel members who agreed to participate. Participants received and returned surveys by e-mail. Response rates were 56% (national panel) and 73% (international panel), indicating that the electronic format was acceptable, with the added benefit of being expeditious and inexpensive.

We reasoned that the use of two separate panels, as well as two iterations per panel in the ECS-CP study, added to the credibility of the studies. However, the number of iterations required to achieve consensus potentially lengthens the study process, with additional time requirements for synthesizing findings and developing follow-up surveys. The use of a videoconference between the two iterations for the ECS-CP panel A provided a further opportunity for participants to be involved in the process of data synthesis. This approach worked well given the close geographic proximity of panel A members. The logistics of videoconference discussion groups (particularly time zones and scheduling) may be predicted to be more difficult and expensive with panel members who are dispersed geographically.

Survey design

We encountered a few technical barriers related to the computer-based survey in the ABPAT-R study. For reasons not clear to us, one blank survey was returned to us in both the national and international Delphi rounds, perhaps highlighting the ease with which electronic files can be wrongly attached or misnamed with e-mail transmission. We noticed that a few panel members failed to respond to certain questions. In an item-by-item analysis, we observed it often was the third of three items on a page (i.e. furthest to the right when printed in Landscape mode) that was missed. We speculated that three items per page was too many, with reviewers advancing to the next page before completing all items. Electronic completion of the evaluation survey itself highlighted other issues. The format of the first evaluation survey (sent to the national panel) was not ‘locked’; when panelists typed their comments, the comment boxes expanded and effectively altered the format of the survey, such that some evaluation items were spread over two pages instead of one. We hypothesized that this may have been another underlying reason for panelists missing questions during the first round. After these formatting issues were addressed, missed questions were far less common in the second iteration.

Consensus setting

We were challenged by the absence of a widely accepted benchmark within the research community of what constitutes an adequate level of consensus within a Delphi process. The selection of percent agreement (>50%) in both studies was a conservative guideline that acknowledged the complexity and degree of potentially dissenting views regarding the topics of interest (e.g., definitions of mechanism of pain, incident pain, psychological distress, addictive behavior, cognitive status, breakthrough pain). Rather than expecting 100% consensus, this process enabled many differing perspectives to be identified and shared with other participants through a second iteration. The use of two iterations per panel in the ECS-CP study enhanced the achievement of consensus, as indicated by a positive response shift (i.e., higher percentage of agreement) from one iteration to the next. In the ABPAT-R study, the Delphi process revealed substantial consensus on the content of the tool, which also increased between rounds of review. The overall level of agreement with the tool, averaged over the four evaluated aspects of all items, was 80% among national panelists and 88% among international panelists. Panelists provided a wide range of constructive, descriptive comments, and several reviewers spontaneously offered similar comments which recommended the same revisions. As a result of these observations, we felt there was a sufficiently high level of consensus that subsequent rounds of review by each panel appeared to not be required.

Response rates

As reported earlier, response rates were moderate to moderately high. Given the more prolonged time frame of the ECS-CP study, a major challenge was to maintain the momentum and interest of participants. Participants were provided with periodic updates regarding the study process by e-mail. However, a more consistent approach, such as a regular e-mail update reporting current response rates and study progress, could have been helpful.

We did not follow up with nonresponders, and therefore, we cannot be certain of their reasons for not participating. In a recent Delphi study involving intensive care clinicians at Canadian academic centers, lack of time was cited as the main reason for refusal to participate by the majority of nonresponders who were followed [14]. Keeney et al. [16] cite examples to suggest that personal contact might help to enhance response rates by building rapport and by informing participants to enhance personal ownership of the project. Others suggest that rigorous follow-up, particularly by telephone rather than by letter, can help to boost response rates [3]. We opted not to identify and pursue nonresponders in an effort to assure anonymity. Also, we did not want to pressure nonrespondents into returning the questionnaire, potentially resulting in suboptimal feedback.

One particularly helpful aspect of the ABPAT-R study design was to invite international panel members to participate in the process first before being sent the survey. Sending the evaluation surveys only to those experts who agreed to participate in the Delphi process likely bolstered the response rate of this panel. This process of preselecting expert panel members was not carried out for the ECS-CP study nor for the first round (national panel) of the ABPAT-R study. As questionnaire research in general is subject to low response rates, for the purposes of a Delphi study, preselecting panel members may help to increase response rates by allowing for the formation of a more motivated, committed panel of experts.

Several of the international experts approached to evaluate the ABPAT-R expressed a desire to participate because they had been considering developing a similar tool themselves or because they believed that developing such a tool was of high priority. The great need for a standardized assessment tool for breakthrough pain and the identification of experts sharing this view may have been another underlying factor in the higher response rate of the international panelists.

Discussion

The Delphi process is a technique that can facilitate national and international collaboration in palliative care tool development. This creates an environment where international communication can occur with the ease of e-mail and videoconferencing and enhances its speed and affordability. We were heartened by the positive response from the large majority of colleagues from around the world who were approached with nothing more than an offer to collaborate and the reward solely of the knowledge of having contributed. Panelists provided many thoughtful and constructive suggestions. In general, the response from those panelists who participated was positive and enthusiastic, particularly among the international panelists, many of whom expressed an interest in collaboration for further testing of the tools. Perhaps one explanation for our positive experience is the inherent legitimacy of the underlying methodology: the Delphi technique is a process of peer review. That being said, limitations of this methodology have been described as: the lack of specific criteria for selection of the ‘expert’ panel (definition of expert as well as size of panel) [25]; the potential for manipulation of consensus [3]; the questionable reliability of the process [25]; and the arbitrary level of consensus that is considered acceptable by different researchers [25].

The proliferation of transfer of knowledge challenges the palliative care community to develop a common language, including common dictionaries. The goal to establish standard definitions challenged us to achieve consensus for potentially disparate phenomenology that would be broadly applicable in as many countries, cultures, and practice scenarios as possible.

We suggest the Delphi process may facilitate more rapid knowledge transfer and uptake of novel tools: we observed that engaging national and international opinion leaders in the validation of tools resulted in both solicited and unsolicited offers to continue collaborative research with the tools. Uptake of innovations within a health-care community is known to be greatly facilitated through support by “early adopters” such as academics who have published in the area of interest [23]. A bibliometric study can trace the diffusion of ideas or innovations over time or across disciplines [2, 4, 8], which may be a surrogate marker of knowledge transfer. Further research in this area could inform thinking around tactics to facilitate more rapid diffusion of palliative care tools by the academic community [23].

Summary

We recommend the Delphi process as an effective and efficient means to obtain broad, expert input in the process of tool development in palliative care. The Delphi method may be used in the process of generating or prioritizing instrument items and in ongoing instrument validation. The implementation of this process may lead to the establishment of international tool development working groups for further research and implementation of the tool in clinical practice. The Delphi method lends itself well to application in an electronic-based communication process. It contributes to making a tool more usable in a range of countries and cultures and may thereby facilitate international collaborations in palliative care research.