Article Text
PDF
Abstract
Introduction Older adults are at high risk of developing delirium in the emergency department (ED). Delirium associated with an ED visit is independently linked to poorer outcomes such as increased length of hospital stay and mortality. Performance measures (PMs) are needed to identify variations in the quality of delirium care to help focus improvement efforts where they are most needed. A preliminary list of 11 quality statements and 24 PMs was developed based on a synthesis of high-quality clinical practice guidelines. The purpose of this study is to gain consensus on a subset of PMs that can be used to evaluate delirium care quality for older ED patients.
Methods and analysis This protocol for a modified e-Delphi study is informed by the Guidance on Conducting and REporting DElphi Studies. Clinical experts from across Canada and internationally will be recruited through peer referral, professional organisations and social media calls for expressions of interest. A minimum of 17 participants will be recruited. The primary survey for each round will consist of closed-ended questions with the opportunity to provide comments to justify decisions and clarify understanding. Using 9-point Likert scales, participants will rate each quality statement according to the concepts of importance and actionability, then its associated PMs according to the concept of necessity. Results will be fed back to participants in subsequent rounds. A priori stopping criteria have been defined in terms of consensus and stability. A minimum of three rounds will be undertaken to allow participants to have feedback, revise previous responses, then stabilise responses.
Ethics and dissemination Ethical approval was provided at the University of Manitoba Health Research Ethics Board (ID HS25728 (H2022:340)). Informed consent will be obtained electronically using the Research Electronic Data Capture secure online platform. Knowledge translation and dissemination will be done through traditional (eg, conference presentations, peer-reviewed publications) and non-traditional (eg, ED Grand Rounds) strategies.
- delirium & cognitive disorders
- accident & emergency medicine
- aged
- quality in health care
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
STRENGTHS AND LIMITATIONS OF THIS STUDY
To ensure rigour and transparency in our research, this protocol is informed by the Guidance on Conducting and REporting DElphi Studies, which provides criteria in a similar format to other reporting checklists and is endorsed by the EQUATOR Network.
A priori criteria for consensus and stability have been defined for this study to reduce the subjectivity of stopping the Delphi process.
Recruiting a heterogeneous sample of clinical experts will help enrich the results and increase the credibility and acceptance of Performance measures (PMs) developed.
This study will use a formal consensus building process to develop a set of guideline-based PMs that are necessary to evaluate the quality of delirium care older emergency department patients receive.
Situational and personal biases can influence differences in how experts make judgements when using the Delphi method, which is a limitation we have attempted to reduce by planning to limit time between rounds, providing detailed background information and clear definitions for all concepts, providing quantitative and qualitative personalised feedback, as well as clearly defining consensus and stopping criteria.
Introduction
Older adults (ie, people 65 years of age and older) often use the emergency department (ED) as their first point of contact with the acute healthcare system.1–4 Older ED patients are at high risk of developing delirium in this care environment5–8; however, it often goes undetected or undertreated.9 10 Delirium is defined as a reversible ‘syndrome of abrupt onset, fluctuating course, with prominent cognitive symptoms including decreased attention and awareness, additional deficits such as memory, or disorientation and evidence of an underlying physiological cause’.11 Delirium associated with an ED visit is independently linked to poorer outcomes such as loss of independence, increased length of hospital stay, and mortality.8 12–17 A major barrier in improving care for older ED patients is the underlying knowledge gaps and lack of practice standards for assessing, recognising and managing delirium in the ED.18 Mechanisms to evaluate ED practice performance are needed first to identify gaps and variations in quality care to focus delirium care improvement strategies where they are most needed.
Much of the research on healthcare quality is based on the seminal work of Donabedian, who defines healthcare provision in terms of structures (ie, conditions under which care is provided), processes (ie, diagnosis, treatment, rehabilitation and prevention of health conditions) and outcomes (ie, changes in an individual or population attributable to healthcare).19 The Institute of Medicine defines quality of healthcare as ‘the degree to which health services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge’.20 This definition implies a dynamic association between structure, process and outcome.20–22 The Donabedian structure-process-outcome framework has guided the development of most existing quality measurement systems because it provides a comprehensive, easily understandable model for classifying different types of measures.23 Performance measures (PMs) are tools to quantify measurable aspects of practice performance (ie, structures and processes).21 24 The extent that PMs are observed in practice provides an indication of the quality of the care provided and the likelihood of attaining optimal patient outcomes.20 24 25
There have been few attempts to establish PMs to evaluate quality of care for older ED patients in relation to delirium. The PMs that exist have been found to be of low methodological quality and predominately based on pre-existing metrics.26–29 PMs are only as good as the evidence and methods used to develop them.23 30 Poorly developed PMs can lead to unintended consequences by providing misleading information to guide decision-making, policy development and quality improvement efforts.23 There is general agreement in the ED quality of care literature that there is a need to rigorously develop new evidence-based PMs instead of basing work on pre-existing metrics.26 31 32
Numerous researchers and organisations assert that clinical practice guidelines (CPGs) are an essential first step in developing quality statements (ie, concise statements defining best practice in a specific context), which in turn provide the basis for identifying PMs.21 24 25 33–35 High-quality CPGs have the power to translate the complexity of scientific evidence into recommendations for practice, and therefore, are essential for developing appropriate metrics to evaluate care quality.33 In the past decade, work has been done to increase the methodological rigour of developing guideline-based PMs.21 24 36 37 Nothacker et al, as part of the Guidelines International Network, developed a framework for generating guideline-based PMs.21 A summary of criteria articulated by Nothacker et al and how these align with our research is provided in online supplemental file 1.
Supplemental material
Prior to the current study, we developed a preliminary set of PMs based on the results of an umbrella review which synthesised recommendations from delirium CPGs appraised to be high quality.38 These recommendations were grouped into four categories of delirium care: screening, diagnosis, risk reduction and management. The synthesised recommendations provided an evidence-based foundation for the creation of a set of quality statements (N=11), and subsquent PMs (N=24), for delirium care in the ED. The next step in establishing a set of PMs for use, is to conduct a formal consensus process with a diverse panel of experts to finalise a set of PMs from the transformed recommendations.21
The purpose of this study is to gain consensus on a set of guideline-based PMs to monitor and evaluate delirium care quality for older ED patients. To achieve this, a modified electronic (e)-Delphi study will be conducted to reach consensus among key experts, including ED clinicians, ED decision-makers and geriatric specialists, on a set of ED PMs.
Methods and analysis
Delphi methods are used to synthesise knowledge and build consensus from a diverse group of experts through an iterative process of surveys and feedback.39–42 A modified e-Delphi process is the most appropriate method for several reasons: (1) independent and anonymous participation reduces group bias, pressure to conform by dominate personalities and groupthink, (2) controlled feedback between rounds provides participants the opportunity to refine their judgements and promotes consensus, (3) including a heterogeneous sample of experts can improve process and results and (4) using an online platform with electronic surveys increases geographical diversity at an acceptable cost and reduces practical barriers to participating.41 42
Study steering group
We convened a steering group to provide study oversight with members consisting of coauthors (with methodological and clinical expertise), two patient/family representatives, and two further clinical experts. The purpose of including individuals beyond the coauthors in this group is to: (1) provide feedback on Delphi survey development, structure and clarity, (2) help identify potential Delphi participants and (3) provide additional important insights throughout the Delphi process as needed. Including patient/family representatives in the steering group helps support the patient-centredness of the quality statements and PMs, such as, by ensuring they contain aspects of care important to patients and families, and suggesting alternative terminology that better reflects patient views. Members of the steering group who are not coauthors were identified through our existing contacts and networks. These members will not have access to raw study data or be able to influence the study process.
Study design
This e-Delphi study is informed by the Guidance on Conducting and REporting DElphi Studies43 (online supplemental file 2), which is endorsed by the EQUATOR Network,44 and other recommended criteria.39 45 A summary of the proposed e-Delphi approach is presented in figure 1. Data collection is planned between April and July 2023. Participant fatigue and attrition tends to increase after three rounds using the Delphi method,41 therefore, three Delphi rounds are proposed, however, a fourth round may be considered if a priori stopping criteria are not met. All rounds will be completed anonymously and electronically using the Research Electronic Data Capture (REDCap) secure online platform for building and managing online surveys46 through the University of Manitoba licensing agreement.47 In every round, Delphi participants will rate the importance and actionability of the quality statements, and the necessity of the related PMs using a 9-point Likert scale. Each participant will be sent a unique REDcap link to allow tracking of responses and participation in each round.
Supplemental material
Delphi study flow. PMs, performance measures.
Study sample
Selection and identification of expert panel
Consistent with the goals of enriching the Delphi consensus results and increasing the credibility and acceptance of PMs developed through this process,21 30 35–37 39 48 a heterogeneous sample of experts consisting of ED clinicians, clinical decision-makers, as well as geriatric specialists will be selected. System-level decision-makers (eg, regional directors or managers) and patients are not included as experts because the completion of the surveys requires specific clinical knowledge. Therefore, an ‘expert’ in this study is defined as one with clinical knowledge in the care of older adults in the ED. Participants will be considered for inclusion if they are able to read and write in English, willing to participate and meet one or more of the following criteria: clinical experience in a relevant field to the ED care of older adults for 5 or more years postbasic graduation, hold postgraduate qualifications or credentials relevant to the management of delirium in older adults, or are recognised by peers as an expert in the area (eg, member of a relevant organisation or network). Potential participants will be excluded if they have insufficient experience in a relevant area or are unable to commit to be available for the entire process.
Panel size and composition
There are no agreed criteria for the ideal sample size for a Delphi study; however, a larger sample size will increase the reliability of the group judgement.39 41 42 It has been asserted that reliability decreases significantly in sample sizes of approximately six or less, and improvements in reliability are subject to diminishing returns in sample sizes greater than approximately 12.41 42 Under the assumption that attrition is common in Delphi studies,41 our minimum sample size (N=17) was calculated to allow for approximately 30% attrition and ensure at least 12 responses in the final round.
Recruitment
Experts will be recruited from across Canada, and internationally, using a multipronged approach (see figure 2), including identification of individuals by coauthors. Recruitment will be augmented by encouraging identified experts to snowball the invitation with other eligible participants. Relevant national professional associations and networks (eg, Canadian Association of Emergency Physicians and National Emergency Nurses Association) and international associations (eg, iDelirium) will be approached to request their support to distribute an advertisement for recruitment to the expert panel via social media (ie, LinkedIn, Twitter and/or Facebook) and email. To prevent over-representation from one expert group, recruitment will be monitored to ensure a heterogeneous sample of experts is recruited. Recruitment will be planned over a 4-week period, with a reminder sent after 2 weeks. If there is no response after 4 weeks, the invited expert will not be contacted again. If the minimum target number of potential participants is not recruited after the 4-week period (N=17), the recruitment timeline will be extended for another 4 weeks.
Recruitment process.
Screening for eligibility
All identified experts will be invited to participate. The potential participants will be sent materials to inform them of the study objectives and design, as well as the commitment required for participation. Potential participants will be screened for eligibility and asked to nominate peers that may be interested in participating at this point. Experts who confirm their interest and eligibility will be provided with a unique link to the REDCap system at the University of Manitoba, where a participant information sheet and consent form will be hosted. The participant information sheet will include information such as the purpose of the study, a clear explanation of the anticipated process, and an explanation that participation will be required over a period of 4 months with three rounds of questioning and feedback.41 The information sheet will also reiterate the eligibility criteria, as well as explain anonymity and the withdrawal process. Participants will be asked to commit to participate in all rounds of the Delphi process, however, they will be allowed to withdraw at any time during data collection. Due to the nature of the Delphi process, responses will be used up to the point of withdrawal. Participants will be able to withdraw from the study by contacting the principal investigator (SF). Consent to participate will be obtained through REDCap.
Enhancing response rate
We will aim to achieve a 100% response rate in each round, however, recognising attrition and participant fatigue is common in studies using the Delphi method, a round will be accepted as valid if there is a response rate of 70%.41 We will employ methods to minimise attrition. Participants will be invited to an initial interview with the option for it to be conducted virtually (ie, video) or by telephone to discuss initial questions, as well as to develop rapport.41 For each round, the online survey will remain active for 2 weeks. After 1 week, an email reminder or thank-you will be sent.
Survey design and development
This study is a modified e-Delphi study, meaning participants are given preselected ‘issues’ (ie, quality statements and potential PMs) on which to make a judgement based on the current evidence.41 Therefore, the primary survey will consist of closed-ended questions with the opportunity to provide comments for each quality statement and related PMs to justify decisions and clarify understanding. The survey will be developed in REDCap. Along with the survey, each round of the study will be accompanied by a cover letter to refamiliarise participants to the study, state the intentions of the round, and provide definitions for key concepts.
Development of the survey was informed by PM assessment tools (ie, AIRE49 and QUALIFY50 instruments), PM criteria used by organisations that develop and implement PMs (ie, the National Quality Forum51 and NICE25), as well as syntheses of these sources.21 23 30 As the PMs are the operationalisable form of the quality statements,21 24 25 33–35 both will be judged by participants. Participants will be asked to judge each quality statement according to its importance (ie, relevant and of crucial value to the care of older adults in the ED) and actionability (ie, care can be done by providers in the ED setting with appropriate resources and tools). Then, they will be asked to judge each related PM according to its necessity (ie, the PM is necessary to evaluate the associated quality statement). The quality statements and PMs will be scored using these dimensions, on a Likert scale ranging from 1 to 9. Delphi participants will be advised to think of each 9-point scale being made up of three parts (ie, tertiles), where 1–3 could be used to record low ratings (ie, not at all important, actionable and/or necessary), 4–6 record average ratings (ie, somewhat important, actionable and/or necessary) and 7–9 record high ratings (ie, very important, actionable and/or necessary). For participants to provide an informed judgement, a rationale for the quality statement and PMs, including a brief summary of the evidence, will be provided.
Prior to implementation, the Delphi survey was piloted with clinical expert steering group members. The purpose of this process was to ensure that the PMs are clearly and precisely worded with unambiguous language, and that each set of PMs reflect the quality statement they are meant to measure. Those involved in the pilot were also asked questions such as, what did you like, what didn’t you like, and how long did the survey take to complete? Adjustments were made to the survey based on feedback from all steering group members and agreed on by study coauthors. For example, as quality statements are defined as ‘concise statements defining best practice in a specific context’ it was agreed to call the quality statements ‘best practices’ in the Delphi survey to decrease the number of new concepts introduced to participants and to increase comprehensibility and readability.
Procedures
A minimum of three rounds will be undertaken to allow participants to receive feedback, revise previous responses, then stabilise responses.41 52 A decision will be made after the third round if it is appropriate to stop according to a priori definitions of consensus and stability, due to the need to balance a high response rate with the need to achieve stability and consensus.
Defining consensus and stability
Establishing a priori criteria for defining consensus and stability reduces the subjectivity of stopping criteria.43 45 52 53 There is no universal definition of consensus,45 however, it is important to define the criteria for consensus that are suitable for the study objectives a priori.43 In concordance with PM development frameworks,30 54 we will use the RAND criteria for agreement to define consensus.55 Consensus is defined as 80% of ratings within the 3-point tertile of the overall median. The lower tertile1–3 represents scores that are ‘not at all’, the middle tertile4–6 represents scores that are ‘somewhat’, and the upper tertile7–9 represents scores that are ‘very’ important, actionable and/or necessary.
Consensus has been stated to be meaningless if stability is not also achieved.52 56 Therefore, a measurement of stability will be used as a stopping criterion for the Delphi process. In this study, stability will be defined as the consistency of responses between successive rounds (ie, no meaningful change).41 52 56 Meaningful change is defined as a median change between tertiles and a greater than 15% change in the percentage of participants whose scores changed tertiles.52 56
Stopping and PM removal criteria
For the overall study, the criterion to stop the Delphi process is defined as no meaningful change in scores between the current and preceding round on at least 75% of quality statements and PMs assessed. Using a cut-off of stability in three-quarters of the quality statements and PMs acknowledges achieving 100% stability may not be feasible, and provides flexibility to balance participant workload.41 52 56 Additionally, criteria for PM removal will be considered after the second and third rounds: To be removed from the process, a PM’s scores must show no meaningful change from the previous round, and there must be consensus that the PM is not necessary (ie, overall panel median of 1 to 3, with 80% of ratings in the lower tertile). Regardless of scores, all quality statements will remain in all rounds.
Round 1
The objective of round 1 is to obtain participant demographic data to describe the sample and collect data on initial judgements on the quality statements and potential PMs. Participants will complete a Participant Details Form collecting information on age, sex, geographical location, professional background, highest level of education, work setting and years of experience. All PMs assessed in the first round will be retained for the next round.
Round 2
The objective of round 2 is to reach consensus on quality statements and PMs by enabling participants to reflect on their score considering the viewpoint of other experts in the group. All participants will be invited to round 2, including those who did not complete round 1, unless they have withdrawn from the study. This provides participants the opportunity to continue their involvement if they choose.41 A personalised survey will be sent to each participant with: quantitative group results (ie, median, minimum and maximum ratings) presented numerically and graphically, qualitative feedback (ie, summary of participants’ comments), and the participant’s own response to illustrate their position in relation to the group. The PMs that meet stopping criteria will be removed from the process after round 2. All other PMs will remain in the third round to achieve or strengthen consensus on a final set of PMs.
Round 3
The objective of round 3 is to strengthen consensus on a final set of quality statements and PMs. All participants will be invited to round 3, unless they have withdrawn from the study. Similar to the second round, a new personalised survey will be sent to each participant again with the revised: quantitative group results presented numerically and graphically, qualitative feedback and the participant’s own response to illustrate their position in relation to the group.
Potential subsequent round
If the stopping criteria are not met after three rounds, a fourth round will be considered. This will be done to gain stability in responses, as well as reach consensus or gain greater understanding about the lack of consensus. The Delphi will be stopped if responses are stable and consensus is still not achieved after the third round, signifying persistent disagreement, as these results may also provide informative insights to guide future research.43
Data analysis
Data will be analysed during the Delphi procedure to provide participants with feedback, to exclude PMs from subsequent rounds, and to inform the need for a fourth round. The final Delphi results will then be analysed to directly achieve the study purpose.
Data analyses conducted during the Delphi process
There are two types of data analysis that will be conducted during the Delphi process. First, data will be analysed after each round to generate feedback reports for participants as described above. Descriptive statistics will be used to summarise responses to be fed-back to participants. Content analysis will be used to summarise participants’ comments into themes to be fed-back to the group.41 57 Original wording from one expert that best represents the wording across participants with similar statements will be used where possible.41
Second, data will be analysed to assess stability as a criterion for stopping the Delphi process, as well as for potentially removing PMs after the second and third rounds. Data will be descriptively analysed with frequency distributions generated to determine if there is meaningful change in the central tendency (ie, median change between tertiles) or dispersion of scores (ie, >15% change in participants responses between tertiles) between the current and preceding round. Results will then be examined to determine if stopping or PM removal criteria are met.
Analysing final Delphi results
The primary analytical goal is to identify quality statements that are important and actionable for quality delirium care of older ED patients and generate a set of related PMs necessary to monitor and evaluate this care. This will be done using the concepts of importance, actionability and necessity to reach consensus. Descriptive statistics will be used to evaluate consensus. To be selected for preliminary testing in future research, a quality statement and related PMs must reach consensus in the upper tertile (ie, overall panel median of 7–9, with 80% of ratings within the 3-point tertile of the overall median).
Once the Delphi is stopped and results are analysed, quality statements that are very important and actionable and related PMs that are very necessary according to group consensus will be selected for preliminary testing in future research. Those that achieve consensus just below the a priori thresholds for final selection but are believed to be relevant will be considered a posteriori as long as there is sufficient justification to do so.45 This will be done by members of the study steering group by examining the analysed qualitative data, along with the quantitative results to help make these determinations.
Patient and public involvement
Patients and/or the public were involved in the design, or conduct, or reporting, or dissemination plans of this research. Refer to the ‘study steering group’ subsection for further details.
Ethics and dissemination
This study received approval from the University of Manitoba Health Research Ethics Board (ID HS25728 (H2022:340)). All data will be collected and stored online through the REDCap Survey server, which is a secure platform that abides by Canadian privacy laws and implements several security safeguards. Informed consent will be obtained from all participants before completing any surveys.
Engagement with diverse experts throughout this study will help to ensure that our results are translatable to everyday ED care. Knowledge translation and dissemination will be done through traditional (eg, conference presentations, peer-reviewed publications) and non-traditional (eg, ED Grand Rounds) strategies. This study will provide a set of quality statements and related PMs that are based on scientifically sound evidence, and are important, actionable and necessary to a diverse group of clinical experts. Results will be used in future research to test the feasibility of using the PMs to evaluate delirium care quality older ED patients receive and guide improvement efforts.
Ethics statements
Patient consent for publication
Acknowledgments
The authors wish to thank and acknowledge Jessica Leigh RN, and Dr. Kerri Onotera (clinical experts), as well as Carli Rossall and one member who has chosen to remain anonymous (patient and family representatives) who agreed to be members of the study steering group.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Twitter @filiatrs
Contributors SF conceived the research idea under the supervision of MD, SK, JG and AC. The manuscript was first drafted by SF and MD, additional content and draft reviews were provided by SK, JG and AC. All authors reviewed and approved the final version.
Funding SF: The Canadian Institutes of Health Research (CIHR) Frederick Banting and Charles Best Canada Graduate Scholarships-Doctoral Research Award (CGS-D) (Award #: FRN-175944), and the Canadian Nurses Foundation (CNF) Dr. Ann C. Beckingham award (Award #: N/A). JG: Canada Research Chair in Health Knowledge Transfer and Uptake (Grant #: 950-200986) and CIHR Foundation grant (Grant #: FDN-143269). SK: Manitoba Research Chair in Health System Innovation (Research Manitoba; Grant #: N/A).
Competing interests None declared.
Patient and public involvement Patients and/or the public were involved in the design, or conduct, or reporting, or dissemination plans of this research. Refer to the Methods section for further details.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.