Article Text
PDF
Abstract
Introduction Robust assessment is a crucial component in Advanced Cardiac Life Support (ACLS) training to determine whether participants have achieved learning objectives with little or no variation in their overall outcomes. This study aimed to evaluate resuscitation performance by real-time logs. We hypothesised that instructors may not be able to evaluate time-sensitive parameters, namely, chest compression fraction, time to initiating chest compression and time to initiating defibrillation accurately in a subjective manner.
Methods Video records and formal checklist-based test results of Megacode scenarios for the ACLS certification examination at several hospitals in Taipei were examined. For the study interest, three time-sensitive parameters were measured via video review assisted by a mobile phone application, and were used for evaluation. We evaluated if the pass/fail results made by instructors via checklists were correlated with these parameters.
Results A total of 185 Megacode scenarios were eligible for the final analysis. Among the three parameters, good chest compression fraction was statistically significant with a higher OR of passing (OR=3.65; 95% CI 1.36 to 9.91; p=0.01). In 112 participants with one parameter that did not meet the criteria, 25 were graded as fail, making the specificity 22.3% (95% CI 15.0% to 31.2%).
Conclusions Visual observation of cardiopulmonary resuscitation performance is not accurate when evaluating time-sensitive parameters. Objective results should be offered for training outcome evaluation, and also for feedback to participants.
- ACLS
- IOS
- android
- codetracer
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
The first study regarding the use of an app to assist in evaluation of resuscitation performance in an Advanced Cardiac Life Support course.
In this study we did not collect official feedback from instructors regarding the app.
Results by multiple instructors would gain generalisability but lose internal validation as well.
Introduction
The Advanced Cardiac Life Support (ACLS) protocol published by the American Heart Association (AHA) is the standard of care for patients with cardiac arrest.1 Studies have revealed that good ACLS protocol adherence is a determinant of return of spontaneous circulation (ROSC).2 3 While the presence of ACLS providers may increase the likelihood of ROSC and 1-year survival rates,4 resuscitation system errors are associated with decreased survival.5 These findings indicate the advantages of the ACLS protocol in critical situations, and emphasise the importance of the ACLS training course.
Robust assessment is a crucial component in ACLS training to determine whether participants have achieved learning objectives with little variation in their overall outcomes. Apart from high-quality chest compression (push hard and push fast), parameters of resuscitation efficiency such as early chest compression, early defibrillation and decreased hands-off time are also vital parts of the 2015 ACLS protocol.1 6 However, because of limited time and equipment in ACLS courses, real-time monitoring and recording of scenarios is difficult. The performance of the case scenarios is evaluated mostly by the instructors’ subjective judgement, of which the quality of cardiopulmonary resuscitation (CPR) is often overestimated.7 There is no standard regarding correct responses, and since no objective parameters can be applied, participants may also be confused about their performance during debriefing, and even after the courses are complete.
In previous studies, we found that the mobile app CodeTracer can record resuscitation performance and may be a useful tool in ACLS courses.8 9 Several studies have also been conducted to evaluate the performance of resuscitations using different methods, such as video evaluation and feedback devices.10–12 However, no study has been focused on the app-assisted objective evaluation of CPR performance.
This study assumed that instructors may not be able to evaluate time-sensitive parameters, namely, chest compression fraction, time to initiating chest compression and time to initiating defibrillation, efficiently in a subjective manner. Thus, we attempted to evaluate resuscitation performance using a simple mobile phone application. Real-time recording of CPR logs can be achieved, and objective parameters can be calculated simultaneously after scenarios are completed. We further compared these results with instructors’ judgement to see the differences. This alternative way to evaluate time-sensitive resuscitation performance may further assist in the improvement of resuscitation training.
Methods
Study design
The study was a retrospective observational study aimed at the improvement of performance evaluation in ACLS courses. The research question is whether visual evaluation of CPR parameters by instructors does accurately reflect time-sensitive parameters including time to chest compression, time to defibrillation and chest compression fraction.
For the purpose of the study, only scenarios with initial shockable rhythms (ie, ventricular fibrillation and pulseless ventricular tachycardia) were used for analyses. Based on the current guidelines,1 6 13 the chest compression fraction should be no less than 60% during cardiac arrest scenarios. Time to initiating CPR and time to defibrillation should be as soon as possible once victims are identified. In this study, we defined good time to initiating CPR to be less than 30 s, and good time to defibrillation to be less than 60 s.
Cardiac arrest scenario
Scenarios are held by certificated instructors in ACLS training in Mackay Memorial Hospital, Taipei, Taiwan. Participants undergo the simulations in groups of five to six. For each scenario, one of the participants is assigned to act as a team leader supported by the other members of the group. The team members follow only the instructions of the team leaders, and do not act by themselves. Most of the cardiac arrest interventions (chest compression, airway management, rhythm interpretation and defibrillation) are performed in real time, while intravenous cannulation and drug administration are given as verbal instructions because of the limitations of manikins. Scenarios were conducted in real time so instructors would neither give suggestions nor interrupt the progress in the middle of the scenario. All episodes were videotaped by a camera over the head side of the manikin in order to observe the conditions and interactions.
Performance evaluation by instructors
A standard AHA Megacode testing checklist was used for the evaluations. Instructors gave test results (pass or fail) based on overall performance. Specifically, parameters regarding CPR performance were observed over several items including ‘ensures high-quality CPR at all times’, ‘recognises shockable rhythms’, ‘clear before analyse and shock’, and ‘immediately resumes CPR after shocks’. All checklist items were required to be evaluated as appropriate in order to get a final ‘pass’ result.
Performance evaluation by the mobile phone application
A free, easy-to-use iOS/Android app (CodeTracer) was developed by Wistron Corporation, Taiwan for the real-time recording of CPR performance.8 The user interface is illustrated in figure 1. Several interventions which are performed during ACLS scenarios are set up as buttons: start/hold chest compression and ventilation (start/pause CPR), rhythm check (rhythm), defibrillation (Defib), cardioversion (shock), epinephrine administration (Epinephrine) and administering medications (OtherDrugs). Rhythm check or defibrillation/shock cannot be recorded until CPR is paused (figure 2). When the scenario begins, users could record every intervention the team performs by pressing the buttons. When the scenario ends, CodeTracer automatically computes parameters including chest compression fraction, time to initiating CPR, and time to initiating defibrillation and also generates a graphic log for later discussion (figure 3). Chest compression fraction refers to the percentage of time in which chest compressions are performed during resuscitation. In this study, we retrospectively collected videos as described above and evaluated again by the app. Two authors (Y-CS and M-YH) were responsible for data collection by reviewing the videotaped simulations. We then evaluated if the test results by subjective observations were correlated with these time-sensitive parameters. Instructors who evaluated the Megacode simulations with checklists did not participate in the evaluation by the app. Since authors in this study were also not involved as instructors in the training courses, the evaluations by the app would be independent from the previous pass/fail results.
User interface of CodeTracer. CPR, cardiopulmonary resuscitation; ROSC, return of spontaneous circulation.
User interface of CodeTracer during cardiopulmonary resuscitation (CPR). ROSC, return of spontaneous circulation.
Cardiopulmonary resuscitation (CPR) log of CodeTracer. IV, intravenous; PEA, pulseless electrical activity; VT, ventricular tachycardia.
Statistical analysis
Continuous variables were compared with t-test, and categorical variables with the χ2 test. Ninety-five per cent CIs and p values were reported. Values of p less than 0.05 were considered statistically significant. All analyses were performed using Statistical Analysis Software for Windows, V.9.4 (SAS Institute, Cary, North Carolina, USA).
Logistic regressions were applied to determine if the results of the subjective observations of pass or fail were influenced by the time-sensitive parameters (ie, chest compression fraction, time to initiating CPR and time to initiating defibrillation). We further computed the sensitivity and specificity of pass/fail results from subjective observations by taking the three time-sensitive parameters as the gold standard.
Patient and public involvement statement
Participants and the public sector were not directly involved in the design and conduct of this study.
Results
A total of 185 episodes of cardiac arrest scenarios were observed and evaluated during the period from 1 July 2015 to 30 September 2016 in a total of 10 ACLS courses. All participants were registered nurses with basic life support certificates. Instructors finished official AHA Megacode checklists, and graded the results as pass or fail. Among these episodes, 156 (84.3%) were graded as pass and 29 (15.7%) as fail. In the app evaluation, parameters including chest compression fraction, time to initiating chest compression and time to initiating defibrillation were recorded by CodeTracer by reviewing the videotaped simulations. The three time-sensitive parameters were better among participants with pass results, and are summarised in table 1. Distribution plots were used to show the trends (figures 4 and 5).
Distribution plot for time to initiating chest compression and time to initiating defibrillation.
Distribution plot for chest compression fraction.
Time-sensitive parameters of the two groups
Logistic regression was used to evaluate the adjusted ORs of the time-sensitive parameters on pass results. Of the three parameters, good chest compression fraction was statistically significant with a higher OR of a pass result (OR=3.65; 95% CI 1.36 to 9.91; p=0.01) (table 2).
Adjusted ORs for pass results
We then evaluated the sensitivity and specificity of subjective observation results based on the three parameters. In 73 participants with all three parameters considered good, 69 were graded as pass, making the sensitivity 94.5% (95% CI 86.6% to 98.5%). In 112 participants with one parameter that did not meet the criteria, 25 were graded as fail, making the specificity 22.3% (95% CI 15.0% to 31.2%).
Discussion
To our knowledge, this is the first study on use of a mobile phone application to assist in the evaluation of resuscitation performance in an ACLS course. We found that better performance on time-sensitive parameters among the pass group indicated instructors are able to ‘sense’ subjectively if teams are doing good to some extent. For teams with a higher percentage (>60%) of chest compression fraction, there was a higher possibility of pass results (OR=3.65; 95% CI 1.36 to 9.91). Furthermore, the high sensitivity (94.5%) of instructors’ judgement indicates that there is a low false-negative rate. If instructors graded the results as ‘fail’, the performance of the three parameters was poor.
However, low specificity (22.3%) reflects a high false-positive rate. This means that even if the instructors recognise the performance is acceptable, the team may still be doing poorly. In other words, instructors generally have difficulty identifying bad team performance based on subjective observation. This is correlated with our research question that visual evaluation of CPR performance may be suboptimal, especially for time-sensitive parameters.
Although in a formal ACLS training course, parameters associated with guideline adherence such as early defibrillation, early chest compression and low percentage of no-flow time are commonly taught,14 the efficiency of performance is difficult to evaluate. Without objective measures, it is difficult for instructors to offer feedback during debriefing, and participants may not be confident of their performance. This may contribute to low retention rates of skills, which is commonly observed after ACLS training.15–17 Moreover, subjective evaluation may allow participants to pass the Megacode tests and receive a certification even they do not perform well with respect to vital parameters, which may be associated with poor resuscitation performance in real situations.
Several studies have shown the promising future of resuscitation evaluation by digital devices. In Tobase et al,10 immediate-feedback mobile devices are found to verify the students’ performance with greater objectivity and precision in the basic life support course. In Lloyd et al,12 a diagnostic feedback patch was developed to measure both the compression depth and force during neonatal CPR and give audiovisual feedback. Lowe et al 11 review the key opportunities in resuscitation through video-assisted care performance analysis found that video may improve individual and team performance. Application-assisted or device-assisted teaching can also have potential benefits for ACLS courses. First, standards regarding objective parameters can be used. A scoring system can also be developed by automated calculation, and the pass or fail in the Megacode test will be more objective. Second, participants will understand their deficits during practice, and can make efforts to improve their performance based on objective outcomes. Third, resuscitation logs can be saved as each participant’s own records, and retention of the skills can be easily compared and evaluated.
We have to address several limitations of this study. First, since this study interest was the visual evaluation of time-sensitive parameters during resuscitation, other core parts which may alter pass/fail results in the Megacode tests were not considered. Second, it may be a little difficult for instructors who are not familiar with CodeTracer to record the resuscitation logs and observe team leader performance at the same time. In this study we did not collect official feedback from instructors regarding the app. In our experience, instructors are happy to have objective results right after the simulation. These results can be used for evaluation and for feedback during debriefing.8 9CodeTracer is a free app which can be used for any resuscitation training course regardless of the training devices. Products such as the Resusci Anne QCPR which can automatically record the CPR performance can be good alternative choices. Third, the checklist evaluations were done by multiple instructors in the ACLS training courses. Results by multiple instructors would gain generalisability but lose internal validation as well. Although all instructors are board certified, we believe that inter-rater reliability would be an issue and might introduce bias in our study. The low specificity of the checklist results could also be attributed to suboptimal assessment. Further emphasis on these important time-sensitive parameters for instructors should be addressed.
Conclusions
Visual observation of CPR performance is not accurate when evaluating time-sensitive parameters such as chest compression fraction, time to initiating chest compression and time to initiating defibrillation. Objective results should be used for training outcome evaluation and to provide feedback for participants.
Acknowledgments
The authors thank ChenYang Hsu for the in-depth discussion on study design and statistical methods (National Taipei University of Nursing and Health Sciences).
References
Footnotes
Contributors M-YH analyzed and interpreted the data. L-CK interpreted the data and contributed to manuscript development. S-WH supervised the data and contributed to manuscript writing. Y-KL supervised the study and interpreted the data. Y-CS analyzed the data, and was a major contributor in writing the manuscript. All authors read and approved the final manuscript.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Patient consent Not required.
Ethics approval This study was initiated after approval from the Institutional Review Board of Taipei Mackay Memorial Hospital, Taiwan.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement Extra data can be accessed via the Dryad data repository at http://datadryad.org/ with the doi:10.5061/dryad.b5s87dt.