Article Text
PDF
Abstract
Background The trigger tool methodology uses clinical algorithms applied electronically to ‘flag’ medical records where adverse events (AEs) have most likely occurred. The authors sought to create surgical triggers to detect AEs in the ambulatory care setting.
Methods Four consecutive steps were used to develop ambulatory surgery triggers. First, the authors conducted a comprehensive literature review for surgical triggers. Second, a series of multidisciplinary focus groups (physicians, nurses, pharmacists and information technology specialists) provided user input on trigger selection. Third, a clinical advisory panel designed an initial set of 10 triggers. Finally, a three-phase Delphi process (surgical and trigger tool experts) evaluated and rated the suggested triggers.
Results The authors designed an initial set of 10 surgical triggers including five global triggers (flagging medical records for the suspicion of any AE) and five AE-specific triggers (flagging medical records for the suspicion of specific AEs). Based on the Delphi rating of the trigger's utility for system-level interventions, the final triggers were: (1) emergency room visit(s) within 21 days from surgery; (2) unscheduled readmission within 30 days from surgery; (3) unscheduled procedure (interventional radiological, urological, dental, cardiac or gastroenterological) or reoperation within 30 days from surgery; (4) unplanned initial hospital length of stay more than 24 h; and (5) lower-extremity Doppler ultrasound order entry and ICD code for deep vein thrombosis or pulmonary embolus within 30 days from surgery.
Conclusion The authors therefore propose a systematic methodology to develop trigger tools that takes into consideration previously published work, end-user preferences and expert opinion.
- Quality of care
- surgery
- ambulatory care
- adverse event
- information technology
Statistics from Altmetric.com
Introduction
Ambulatory or same-day surgeries have become increasingly more frequent in the last two decades, accounting for a greater proportion of surgeries performed in most medical centres and practises. Ambulatory surgery has gained more acceptance by both patients and providers mainly due to safer anaesthetic techniques and less invasive surgical techniques. In fact, multiple studies in the last few years suggest that ambulatory surgery is associated with minimal patient morbidity and that major postdischarge complications remain infrequent and rarely life-threatening.1–7 In a recent study from Denmark,8 where authors evaluated the causes of hospital readmissions following same-day surgeries, postoperative haematoma or haemorrhage (0.4%) and postoperative infections (0.29%) were the two most commonly encountered complications resulting in readmissions. When the same authors looked at the two most common same-day procedures performed, hernia repair and knee arthroscopy, these were associated with patient morbidity in 1:39 patients and 1:220 patients, respectively. Another 3-year study of complications following ambulatory surgery found that 0.15% of patients had an unplanned readmission or return visit within 30 days for complications related to their surgery.9 As same-day surgeries become more complex, and involve patients with advanced age and more preoperative comorbidities, the incidence of postoperative adverse events (AEs) in this setting will probably increase.
The trigger tool methodology uses surveillance algorithms (triggers) derived from clinical logic to ‘flag’ medical records where AEs have most likely occurred. These triggers or clues can be applied manually or electronically to clinical or administrative data. A Global (or General) Trigger identifies medical records with a high probability of occurrence of one of a broad spectrum of AEs. For example, frequent clinic visits following a patient's laparoscopic cholecystectomy can serve as a trigger or clue indicating that the postoperative course is not proceeding according to plan, possibly due to the occurrence of a postoperative AE. A Specific Trigger identifies medical records with a high probability of occurrence of one type of AE or a small family of AEs. For example, a progressive elevation of creatinine levels 1 week after surgery might identify the occurrence of postoperative renal failure. Most trigger algorithms and resultant alerts in use across the country are focused on adverse drug events. More recently, the Institute of Healthcare Improvement (IHI) published the results of a set of trigger tools targeting inpatient surgical AEs.10 In this latter study, 14.6% of the patients whose charts were flagged by the IHI trigger tool had an actual surgical AE, with most of these events not reported by any of the traditional or pre-existing surveillance methods. To the best of our knowledge, the trigger tool methodology has been focused primarily on the inpatient setting, and has only recently been attempted in the ambulatory care setting. For these reasons, we sought to design surgical trigger tools applicable to ambulatory care; in this specific paper, we describe a formal methodology to create these trigger algorithms that is based on the evidence in literature, end users' suggestions and experts' opinion. Given the amount of time, cost and resource commitment needed to electronically translate and then test trigger algorithms, a standardised design methodology is essential to minimise the risk of serious trigger design flaws.
Methods
Four consecutive steps (literature review, multidisciplinary focus groups, clinical advisory panel and three-phase Delphi process) were used to develop ambulatory surgery triggers as illustrated in figure 1. This methodology was designed specifically to reflect not only researchers' and experts' opinions, but also previous relevant literature in the area and the input of actual end users.
Study design.
Literature review
To collect information related to existing trigger tools in ambulatory care (including ambulatory surgery), we conducted a comprehensive search of multiple literature databases using standardised keyword searches. We limited the search to articles in the English language that were published since 1990. The databases searched included Medline/PubMed, Cochrane, EBM Reviews, Medline Silver Platter and PsychINFO. Since the trigger tool field is relatively new, and the words ‘trigger’ and ‘trigger tool’ are limited in use, we used multiple keyword combinations of more commonly used terms, such as ‘adverse event AND alert’ (box 1).
Box 1 Examples of keyword combinations used in the literature review Trigger tools
Adverse event and surveillance
Adverse event surveillance
Adverse event and signal
Adverse event and alert
(Medical errors OR medication errors OR postoperative complications OR iatrogenic disease) AND (Safety management OR risk management OR quality assurance) AND (Data collection OR decision support systems, clinical OR reminder systems OR medication systems, hospital OR medical records systems, computerised OR monitoring, physiological OR drug monitoring OR adverse drug reaction reporting systems OR clinical laboratory information systems OR user-computer interface) AND (Ambulatory care OR primary healthcare OR outpatients OR inpatients OR emergency services, hospital OR hospital)
Relevant articles were reviewed for the following information: AE setting (eg, ambulatory, inpatient, emergency room), trigger type (global, specific), trigger test characteristics (sensitivity, specificity and positive predictive value) and detection time frame (retrospective, concurrent). These fields were collected systematically using a Microsoft Access (Microsoft, Seattle, Washington) database specifically designed for this purpose.11
Focus groups
Focus groups were conducted at three different medical centres to obtain input from potential end users on: (1) users' perceptions of the relative incidence and impact of different AEs in the ambulatory setting, (2) priorities and preferences for potential areas of focus for ambulatory triggers and (3) triggers' clinical relevance, utility and ease of implementation. The 18 participants in the three focus groups included eight practising physicians (internal medicine, primary care, geriatrics, surgery and emergency medicine), three nurses (including quality improvement and infection control nurses), two clinical pharmacists, two information technology or informatics staff, and three administrative staff from clinical or quality-management departments.
Clinical advisory panel
The clinical advisory panel consisted of clinician members of the research team. Based on the determined gaps in literature and the insights provided by the focus groups, a preliminary set of 10 trigger rules was suggested.
Delphi process
In order to collate the expert feedback in a structured manner and formulate a consensus judgement12 13 on the choice of triggers, we established a Delphi panel that consisted of 11 experts including three surgeons with national expertise in surgical AE detection and surveillance and eight experts in patient safety and/or trigger tool design. These experts came from several geographically distinct areas of the USA. We used a modified RAND/UCLA Appropriateness Method that was conducted in three rounds.14 In the first round, panellists evaluated and rated the initial set of 10 surgical trigger tools on a scale from 1 to 9 (1: high; 9: low). In addition, panellists provided their input and suggestions to improve the test characteristics of each trigger algorithm. After incorporating the panel recommendations, a second Delphi round was conducted, where the triggers were rated again. In the third and final round, the rating results (median and range) from round two were shared with the panellists with specific probing to highlight areas of disagreement among panellists, particularly when one panellist's rating clearly deviated from the median rating. Subsequently, the panellists rerated the list of triggers.
Analysis
For all three Delphi rounds, we calculated the mean, median and range of each trigger rating on utility for system-level intervention. We also analysed the trigger ratings from the surgical panellists separately from the rest of the respondents to identify whether there were any major differences in rating based on the nature of training of the panellists (medical vs surgical).
Results
Literature review
A total of 745 trigger algorithms were found. These included 497 triggers related to adverse drug events, 166 triggers focused on clinical decision-making related events, 53 surgical triggers and 29 miscellaneous triggers tackling areas such as neonatal intensive care and vaccination. All 53 surgical triggers were retrospective and targeted inpatient AEs, with absence of any trigger algorithms specifically targeting outpatient AEs that are of a slightly different nature and severity than inpatient AEs. Of the 53 identified surgical triggers, 34 were global (including 24 originating from the IHI inpatient trigger tool), and 19 were AE-specific triggers. Fourteen of the 53 surgical triggers relied on simple word searches using natural language processing.
Focus groups
Input from the focus groups highlighted three important areas to consider when designing a trigger tool: (1) clinical relevance, (2) utility and (3) feasibility of implementation. The clinical relevance criteria considered by the participants as essential included the targeted AE incidence and preventability, the strength of association between the AE and the trigger, the degree and cost of patient harm resulting from the AE, and the trigger face validity (ie, whether the trigger make clinical sense to end users?). Participants also discussed utility criteria, which included trigger test characteristics (sensitivity, specificity and positive predictive value), likelihood of preventing harm (system or patient level intervention), usefulness for quality improvement, local actionability, ability to detect gaps in clinical care that frequently ‘fall through the cracks’ and cost–benefit (value added to systems already in place). Implementation concerns raised by the focus groups included physician alert or ‘pop-up’ fatigue (if the triggers are real-time or concurrent), potential for integration with existing systems such as the radiology or lab reporting systems in hospitals, timing of the trigger (eg, would it fire on interim instead of final radiological reports, thus creating false alerts?) and intensity of resource utilisation.
Clinical advisory panel
The preliminary set of triggers was edited and refined until consensus was reached on the list of 10 trigger tools. The set of triggers, along with relevant literature review material, were then presented to the Delphi panel (table 1).
Preliminary list of potential surgical trigger tools
Delphi panel
The three Delphi rounds were conducted consecutively over a period of 6 weeks. Round 1 showed significant divergence on priority ratings among the panellists reflecting different perspectives on the usefulness of the surgical triggers. Table 2 presents the median and range of rating of each trigger for its utility for system-level interventions from all three rounds. As the trigger algorithms were revised to create their final versions, and as the preliminary rating results were shared with the panellists, the rating of the surgical triggers showed an expected convergence with narrower ranges of rating in the final round. For example, the range of panellists' ratings of the trigger rule targeting postoperative urinary-tract infection changed from 1 to 9 in round one to a narrower range of 3–8 in round 3, indicating a gradual move towards consensus judgement. There was no significant difference in ratings between the surgical and non-surgical panellists.
Delphi ratings of utility of surgical triggers for system-level interventions
Final set of trigger tools
Round 3 median ratings were used for the selection of the final list of trigger tools to be tested. When the median rating was equivalent among two or more triggers, the trigger with the narrowest range of rating was chosen. Table 3 lists the final list of surgical triggers selected for testing against systematic medical record review. This list includes four global triggers and one specific trigger to detect postoperative pulmonary embolism or deep vein thrombosis.
Final set of trigger tools
Discussion
Development methodology
We therefore propose a four-step process to design trigger tools for the detection of AEs in ambulatory surgery. The literature review served as an essential step to comprehend and build on the clinical logic involved in trigger tools already in use. The focus groups provided invaluable insight on end users' priorities (such as the need to detect AE that are frequently missed in routine ambulatory care) and concerns (such as clinical alert fatigue). Both the literature review results and the focus groups' suggestions were used by the clinical advisory panel to present the Delphi Panel with a tentative set of triggers to be revised and rated. The Delphi process is a structured and reliable method to collate the opinions of a group of experts in order to formulate a consensus decision.14 Although the name Delphi implies some level of forecasting (in reference to the Delphi Oracle in classical Greece), the Delphi process stands as one of the more objective and reproducible methodologies used to explore the human judgement in the setting of an inexact or imperfect science.15 The main advantages of a Delphi technique, as compared with face-to-face interviews or group interviews are: (1) its relative immunity to classical group dynamics-related bias (eg, domineering vs suggestible personalities); (2) the structured and systematic flow of information; (3) the controlled feedback of panellists; and (4) the statistical analysis of expert opinion (or rating).12–15 On the other hand, the Delphi process has been criticised since its first use in the mid 20th century for its sensitivity to the choice of the participating experts (illusory expertise), its sensitivity to the vagueness and format of the questionnaire (format bias) and its tendency to oversimplify the subjects being discussed (simplification urge).12–15 In our study, we used an objective, systematic and computer-based Delphi method; in addition, we believe that we succeeded in gathering a panel of geographically disperse panellists representing distinct areas of surgical interests, patient safety experience and trigger tool expertise.
Trigger tool implications
One of the potential applications of the trigger tool methodology in surgical care is to enable the choice of a more focused sample to detect same-day surgery related AEs. Specifically, the National Surgical Quality Improvement Program (NSQIP) and the American College of Surgeons-National Surgical Quality Improvement Programs16 17 currently constitute the most reliable, validated and efficient method for surveillance of surgical AEs. Both use a systematic sample that includes only the first 36 consecutive cases and only the first five high-volume cases (such as hernia repair, or laparoscopic cholecystectomy procedures) in an 8-day review cycle to survey for AEs. The limitations of this systematic sampling are evident in high-volume hospitals where many more than 36 surgical procedures are performed in any 8-day cycle; given the rare incidence of AEs encountered in an ambulatory care setting, significant AEs might be missed with such a sampling methodology. With respect to AE detection in the ambulatory surgery specific setting, the list of NSQIP postoperative complications is relatively less relevant and not comprehensive. Ideally, the trigger tool methodology could be used to complement NSQIP by providing a ‘richer’ sample of medical records to be reviewed. Therefore, to screen for surgical AEs, the NSQIP personnel would review only medical records that were ‘flagged’ by the trigger algorithms rather than a random sample of ambulatory surgery records, as is currently done in NSQIP. A flagged medical record will have, at least, a higher likelihood of occurrence of an AE than a randomly chosen medical record.
Another potential use of trigger tools relates to the detection of same-day surgery complications across different providers. It is a relatively common practise for a surgeon to see the complications of same-day procedures that were performed by other surgeons; this can be explained in part by the fact that same-day patients who are discharged and who later present with AEs often opt to pursue care by another surgeon or facility different from the one that, in their mind, ‘caused’ the AE in the first place. A trigger-based Targeted Injury Detection System established in a large medical centre, or a combination of surgical practises or multiple hospitals spanning the same patient population, might prove essential for an accurate and complete follow-up of same-day surgery outcomes. In addition, AE-specific trigger tools that have reasonable positive predictive value and sensitivity can be used as a practical automated method to detect specific surgical AEs, such as postoperative pulmonary embolus or deep vein thrombosis. With the adoption of comprehensive electronic medical records, modifying these trigger algorithms for real-time AE detection will be feasible and could help prevent or mitigate individual patient harm.
Trigger tools test characteristics
The final list of trigger tools for ambulatory surgery is currently being tested on ambulatory surgeries performed in public, private and Veteran Affair medical centres, for both their positive predictive values (retrospective review of flagged medical records) and their sensitivities (retrospective review of a sample of non-flagged medical records). The low incidence of AEs in same-day surgeries warrants the review of a large number of medical records to reach statistically and clinically significant results in our study. However, we believe that the relative infrequency of these AEs should not be a deterrent to such work, but a motivator to design methods to increase efficiency in detecting AEs, and subsequently improve the quality of ambulatory surgery.
In conclusion, we developed trigger tools for ambulatory surgery taking into consideration previously published work, end-user preferences and expert opinion. Global triggers will serve to potentially provide a more convenient sample for medical record review in the search for AEs. In addition, specific triggers with reasonable sensitivity and specificity will be used for AE rate estimation and assessment of the effectiveness of quality-improvement interventions targeting the safety of ambulatory surgery.
References
Footnotes
Funding Funding was provided by the Agency for Healthcare Research and Quality (AHRQ), contract no HHSA290200600012540.
Competing interests None.
Ethics approval Ethics approval was provided by the IRB from all participating institutions was obtained for the project, but this manuscript did not invlove human subjects itself.
Provenance and peer review Not commissioned; externally peer reviewed.