Objectives To provide an overview of the available evidence regarding the safety of in situ simulation (ISS) in the emergency department (ED).
Design Scoping review.
Methods Original articles published before March 2021 were included if they investigated the use of ISS in the field of emergency medicine.
Information sources MEDLINE, EMBASE, Cochrane and Web of Science.
Results A total of 4077 records were identified by our search strategy and 2476 abstracts were screened. One hundred and thirty full articles were reviewed and 81 full articles were included. Only 33 studies (40%) assessed safety-related issues, among which 11 chose a safety-related primary outcome. Latent safety threats (LSTs) assessment was conducted in 24 studies (30%) and the cancellation rate was described in 9 studies (11%). The possible negative impact of ISS on real ED patients was assessed in two studies (2.5%), through a questionnaire and not through patient outcomes.
Conclusion Most studies use ISS for systems-based or education-based applications. Patient safety during ISS is often evaluated in the context of identifying or mitigating LSTs and rarely on the potential impact and risks to patients simultaneously receiving care in the ED. Our scoping review identified knowledge gaps related to the safe conduct of ISS in the ED, which may warrant further investigation.
- ACCIDENT & EMERGENCY MEDICINE
- MEDICAL EDUCATION & TRAINING
- TRAUMA MANAGEMENT
Data availability statement
No data are available.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.
Strengths and limitations of this study
First complete overview of safety of conducting in situ simulation (ISS) in emergency medicine.
The methodology includes a detailed search strategy, clear inclusion and exclusion criteria, duplicate screening and data extraction by independent reviewers.
The main limitation is the widespread design variations of the included studies and of the terminology and concepts related to ISS.
Isolating the impact of ISS from various potential confounding factors remains a major challenge.
Our study identified major knowledge gaps related to the safe conduct of ISS in emergency medicine, which require further investigation.
Emergency medicine (EM) is a complex and challenging specialty requiring the mastery of numerous technical and non-technical skills. Most emergency departments (ED) are overcrowded, chaotic environments with reported rates of errors up to 10%.1 Regular simulation is needed to establish and maintain skills competency for ED professionals.2 3 However, recent literature suggests a bias towards simulations for rare and unexpected cases (cardiac arrest, difficult airway, disaster medicine and/or rare cases) rather than for more common, routine cases.4–6 The retention rate of acquired knowledge and skills can also be quite low.7 8
In situ simulation (ISS) offers an optimal training solution for these practical EM-related issues.9 ISS is defined as simulation taking place in the participant’s everyday work environment.10 11 The ED environment is very different from a simulation centre and this can greatly impact the learning process.12 The value of ISS is predicated on principles from the ‘situated learning theory’ which states that learning is closely linked to the context of the experience.12 ISS is a ‘point of care’ type of simulation training used to respond to specific local teaching and training needs and, as a result, may enhance patient safety by exposing latent safety threats (LSTs) and trialling mitigation strategies.10 13
As with any intervention (or any clinical research project),14 15 there are potential harms linked to ISS including inadvertent use of simulation equipment/medications for real patient care.16–18 There is a potential for tension between the present and future state of ED care related to the delivery of ISS. During an ISS session, patients in the ED may be at risk of negative impacts because of the misdirection of resources towards the ISS. The potential risks associated with ISS and the published no go criteria seem to have been reached empirically through expert consensus.16–18 The main objective of this scoping review is to provide an overview of the available evidence regarding the safety of ISS in the ED. Our secondary objective is to explore the benefits of ISS on all levels of Kirkpatrick’s pyramid in EM.19 20
We followed published guidance for conducting a scoping review.21
The results of this scoping review are reported as per the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines.22
Patient and public involvement
Patients or the public were not involved in the design or the conduct of this study.
We used broad inclusion criteria to present a comprehensive overview of ISS in EM, without any language or date limitations. Original studies were included if they investigated the use of ISS in the ED. Specifically, this comprised studies pertaining to clinician education, system/process evaluation, patient outcomes and patient safety.
We included single-group pretest, non-randomised and randomised studies, parallel-group and cross-over designs, and studies of ‘adjuvant’ instruction in which simulation was added to other instructions common to all learners.
Protocols, commentaries, conference abstracts, posters and correspondences were excluded. This review excludes systematic reviews or meta-analyses and editorial-style reviews. However, those reviews (invited, narrative or systematic), editorials and letters to the editor were scrutinised and citation tracking was conducted to retrieve potentially relevant studies.
Search strategy and information sources
An experienced librarian designed a search strategy (online supplemental eAppendix 1) to search MEDLINE, EMBASE, Cochrane and Web of Science, using keywords specific to each database from inception to January 2020. The search was updated in March 2021.
Since there is no MESH keyword for ‘ISS’, a manual search was performed in our four targeted databases.
All identified studies were exported to Endnote V.X9 (Clarivate Analytics, Philadelphia, Pennsylvania, USA) and duplicates were then removed. Four authors (JT, WL, GM and EJ) independently assessed the eligibility of each study. Title, abstract and full text were first screened and then cross-referenced between reviewers. Disagreements were resolved by consensus. Any unresolved disagreement was discussed with a third researcher with experience in the field of simulation studies (ER-D).
Given the nature of a scoping review, risk of bias and quality appraisals were not performed.21
Four independent reviewers completed data extraction, ensuring a double reviewing for each article.
The main outcome, safety, is defined in this study as the absence of incidents, accidents or a state with the minimal acceptable level of risk. This definition of safety has been used in other simulation studies.23–25 We selected specific criteria such as wait time, patients leaving without being seen or accident reports, which are based on literature on safety metrics in EM.26 27 We explored two safety concepts: ‘ongoing’ safety of patients being managed while ISS was ongoing in the ED (measured by ISS cancellation rate, accident reports, ED median wait time, number of patients who left without being seen) and ‘future patient’ safety with more long-term benefits from ISS (measured by LST evaluation). We sought to explore the impact of ISS on patients’ safety throughout time. Indeed, future patients may benefit from ISS if its output translates into improved skills, clinical care and patient safety.
Secondary outcomes were assessed using the validated Kirkpatrick pyramid, which is a tool used to rank educational interventions on a scale of 1–5 (figure 1).19 Because of the specificity of ISS, we also distinguished studies that used ISS to teach and train from those that aimed to explore procedures and systems.
Descriptive statistics and percentages were used to describe our results. No statistical analysis, quantitative meta-analysis or pooling was conducted due to the heterogeneous study designs and methodologies.
A total of 4077 records were identified by our initial search strategy. After removing 1601 duplicates, 2476 titles and abstracts were screened. Of those, 2366 did not meet our eligibility criteria. A second search was conducted in March 2021 after which seven original articles were added to our review. A total of 81 full articles were analysed (online supplemental eAppendix 2).
General characteristics of included studies
All reviewed articles were published in English between 2006 and 2021 in specialised journals (81%, n=66).
Most included studies used a prospective design (94%; n=76), 47 of which were observational (58%). Only 3 studies were randomised controlled trials and 28 were pre/post intervention studies (35%). For the most part, the included studies were single centre (80%; n=65) and quantitative (67%; n=54). A high majority included interprofessional participants, defined as >2 professions (84%; n=68).
The clinical topics explored in the studies included ED care (77%; n=62) and extrahospital care settings (such as HEMS, prehospital care, etc) (22%; n=18). Most studies focused on ISS training for advanced life support/basic life support (n=44, 54%). The remainder focused on a variety of clinical topics including trauma (n=35, 43%) and airway management (n=21, 26%). Three studies particularly focused on COVID-19 (4%). Very few studies used consistent scenarios and as a result more detailed thematic analysis was not performed. Ethical approval was obtained for 51 studies (63%). The full characteristics of the included studies are presented in table 1.
Is ISS safe for patients in EM?
We found that 33 studies (40%) evaluated safety issues, among which 12 (15%) chose a safety-related primary outcome (table 2).
As for ongoing safety of patients during ISS, none of the included studies assessed the following: (1) The number of patients in ED who left without being seen during the period of the ISS training, or (2) The impact of ISS on patient wait times and (3) Safety-related quality parameters in the ED.
No studies included official institutional accident reports, but three studies included surveys on accident reports.28–30 No accident was reported during ISS.10 31 32 The cancellation rate of ISS was described in nine studies (11%).10 31–38 One study included analyses on feasibility (based on department status, such as patient waiting to be seen) in their secondary outcomes. The authors showed that 3 ISS were delayed, and none were cancelled but no other patient-related outcome was presented.39
The impact of ISS on real ED patients was assessed in two studies. In one multicentre clinical trial, ISS was used to prepare and facilitate the identification and mitigation of threats to study participation and patient safety.26 In this case, ISS was not really used to teach and directly improve skills but rather to prepare teams for the implementation of a clinical trial. As an ongoing research project can also jeopardise the safety of other patients by redirecting attention on the study and not on real patients, safety was analysed with parameters such as latent threats identification, mitigation and protocol errors and deviation. Patterson et al’s work on LST identification also addressed the impact of ISS training on patient care,10 which was assessed using a questionnaire. Four participants out of 118 reported that the simulation was disruptive or affected the participants in a negative way. These are the only studies exploring the impact of ISS on patients being managed in the ED.
Future safety of patients
The LST assessment was conducted in 24 studies (30%).
Kirkpatrick evaluation level
Most studies employed Kirkpatrick-related endpoints (n=75; 92%). Fourteen studies (17%) reported comparison with a preintervention assessment or a control group using knowledge as the outcome (KP1), mostly through a pre/post design. The assessment of KP2 level was provided for 10 studies (12%), and 27 (33%) using ISS as an intervention in EM to assess a level 3 outcome.
Twenty-two studies assessed a level 4 outcome. We considered LST identification as a level 4, because of the direct impact of this methodology on patient care. This classification was decided on consensus among the authors. For instance, in the study by Gray et al (38), the authors showed that the identification of various LSTs regarding massive blood transfusion led to improved patient outcomes (measured by a decrease in the delays between massive haemorrhage protocol activation and blood component administration). We identified three studies with a direct impact on patient outcomes (morbidity, mortality, length of stay). Using a pre–post study design, Steinemann et al demonstrated a significant improvement in mean teamwork scores of real patient care in addition to a 16% reduction in the mean resuscitation time in the ED.40 The other study that included a patient outcome analysis examined the effect of introducing a paediatric medical emergency team combined with regular ISS training on patient and system-level outcomes.41 Even though the decrease in paediatric intensive care unit mortality was nonsignificant, the authors found an association between ISS and a reduction of healthcare-related costs (level 5). Using a prospective observational design, Theilen et al showed that regular ISS training of ED healthcare professionals was associated with a reduction in costs and overall patient mortality. Another more recent study showed direct patient outcome improvement.42 The authors showed that a novel ISS-based quality improvement (QI) intervention for blood component administration in bleeding trauma patients led to a 21% mean reduction in time between massive haemorrhage protocol activation and blood component administration. The impact of changing the institution’s massive haemorrhage protocol after the identification of LSTs during ISS was explored in this study using a pre/post design.
No studies showed a negative impact when exploring the improvement of knowledge or skills. The efficacy of ISS was always superior when compared with any another training method or to a control group. We were unable to categorise 10 of our included studies (12.3%) within the 5 levels of KP, either because the studies assessed guideline adherence38 43 or because ISS was employed to assess a process or a system (table 3).44 45
Indeed, for 24 studies (29%), ISS was used to explore and assess a procedure, an organisation, or guideline adherence and not to teach or train. Seven studies (8.6%) assessed the feasibility of ISS itself without exploring learning or safety outcomes.
This scoping review assessed the safety of ISS in EM.
ISS and patient safety
Conducting ISS within the chaotic and busy ED environment may raise concerns for the ongoing care of real patients. However, we found that few studies evaluated the impact of ISS on the safety of actual ED patients. Our scoping review focused on the available evidence regarding two safety concepts: ongoing ED patient safety during ISS, and the safety of future patients (long-term benefits of ISS). It is important for educators and ED professionals to be aware of the associated risks of ISS. Specific, rigorous guidelines are needed to help create a framework aiming to enhance ISS ‘ongoing’ safety. Surprisingly, we did not find any studies evaluating the impact of ISS on ongoing patient care, clinical adverse events, wait times or patients leaving without being seen. Some editorials warn against these risks,16–18 but we found no studies exploring nor demonstrating the potential consequences or risks of ISS in a busy ED for the ongoing care of patients. Most studies seemingly overvalue the ‘future’ safety of patients while undervaluing the ‘ongoing’ safety of ED patients during ISS. This seems crucial because ISS is increasingly used to teach and train EM professionals. As for any method aiming at improving the quality of care, the risks for patients should be controlled. In that sense, any type of clinical research may jeopardise patient safety. For example, the time it takes to consent and enrol a patient in a therapeutic trial could impact ED safety measures. The time-sensitive nature of EM research has been reported in research primers.14 Our scoping review shows the absence of scientific evidence from studies with rigorous methodology to confirm the absence of risks for patients. The Foundation for Healthcare Simulation Safety published a 10-item ‘pledge’ of ‘best practices’ for simulation programmes to reduce simulation-related hazards.17 46 However, even if those principles seem logical, they are not evidence based.
Educators and simulation teams should consider guidelines from existing literature on QI to ensure the sustainability of these interventions. In the case of ISS, the ongoing safety of patients could be referred to as a ‘balancing measure’. In the QI literature, balancing measures represent checkpoints that ensure there are no potential unintended consequences (risk to ongoing patient safety) resulting from an intervention (ISS).37 47–51
Patient-reported measures, which appear to be underutilised based on our study results, represent a useful lens to understand the value of ISS more comprehensively. For example, at one investigator’s institution (AP), a Code Orange simulation during which dedicated volunteers checked in with patients waiting and patients had positive perceptions regarding this ongoing training (unpublished data).
One would think that being closer to the patient and to the patient care facility would make it easier and more intuitive to assess ISS-related patient outcomes, but only two studies assessed direct patient outcomes.41 42 This raises the question of the relevance of the Kirkpatrick pyramid. Indeed, we deemed appropriate to classify the identification of LST as a level 4 on the KP scale, because it is directly linked to improving the quality of care. This decision was reached on consensus among the authors and is debatable because direct patient outcomes are not measured. However, we found data exploring the improvement of care following the identification of LSTs to validate our choice of classification.41 42 We believe even though this was not directly measured in every study, the identification of LSTs leads to improved patient outcomes by reducing medical errors.52 We were unable to classify a great number of studies using this tool. As other authors have previously suggested, it may be wise to adopt a new lexicon and new endpoints to assess the impact of ISS.53 54 Some studies did, however, explore level KP4 and even KP5.41 Their inspiring results should encourage researchers to pursue their efforts and lead projects aiming at exploring higher levels of the KP pyramid.
Study methodology limitations
As highlighted in previous studies, studies exploring the impact of simulation-based interventions are still needed.2 55–57 We observed that even though the included studies were quite recent, as ISS seems to have emerged in the last 20 years, the vast majority used observational prospective designs. With ISS, the proximity between the intervention and the possible patient-centred outcomes is inspiring and educators should seek collaboration with methodologists to elaborate research protocols leaning to prove the translational dimension of ISS. Simulation should be about improving patient outcomes, and the causative effect of isolating ISS should be a priority.
Strengths and limitations
Because there is no mesh word for ‘ISS’ we conducted a search on simulation and EM followed by more detailed reviews for ‘in situ’ specific articles, which made the screening process more complex. Furthermore, the terms used to describe ISS are highly heterogeneous. It is possible that this resulted in missed articles that otherwise should have been included. However, the corollary is this process required a close assessment of each article resulting in a more thorough review.
Also, several systematic reviews were excluded from our analysis. However, we reviewed and cross-referenced their reference list against our literature search. We found no discrepancies, supporting the accuracy of our approach.
Our scoping review identified major knowledge gaps related to the safe delivery of ISS in the ED, which require further investigation. Patient safety during ISS is often evaluated in the context of identifying or mitigating LSTs with little focus on the potential impact and risks to actual ED patients. For all these reasons, our research team sought to address this issue in a mix-method study, which will aim at demonstrating the safety and impact of ISS.58
Data availability statement
No data are available.
Patient consent for publication
The authors would like to thank Chantal Beauregard for the valuable help on conducting a rigorous library search. The authors would like to thank the Société Française de Médecine d’Urgence (SFMU) for supporting this research project through a research award (JT). We would also like to thank the Fondation du CHU de Québec-Université Laval and the Emergency Physicians from the Hôpital de l’Enfant-Jésus, who also supported this work.
Contributors JT designed the study with the help of VB and under the supervision of MÉ. MÉ obtained research funding. JT, WL, GM and EJ assessed each study and performed data collection. JT drafted the manuscript, and VB, WL, GM, EJ, ER-D, AP and MÉ contributed substantially to its revision. MÉ accepts full responsability for the work and/or the conduct of the study, had access to the data and controlled the decision to publish.
Funding This work was supported by the Fondation du CHU de Québec-Université Laval, grant number 3967. JT received a Fellowship award from the Société française de médecine d’urgence.
Disclaimer The sponsors were in no way involved in the design of this study protocol.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.