Article Text
PDF
Abstract
Objective To evaluate the subjective and objective resuscitation performance of emergency medical technicians (EMTs) using mechanical cardiopulmonary resuscitation (MCPR) devices.
Design and setting This was a cross-sectional simulation-based study where participants installed the MCPR device on a training manikin.
Participants We assessed EMT-Intermediates (EMT-Is) and EMT-Paramedics (EMT-Ps) of the Emergency Medical Services (Ambulance) Division of the Taipei City Fire Department.
Primary and secondary outcome measures The primary outcome was the gap between self-perceived (subjective) and actual (objective) no-flow time during resuscitation, which we hypothesised as statistically insignificant. The secondary outcome was the association between resuscitation performance and personal attributes like knowledge, attitude and self-confidence.
Results Among 210 participants between 21 and 45 years old, only six were female. There were 144 EMT-Is and 66 EMT-Ps. During a simulated resuscitation lasting between four and a half and 5 min, EMTs had longer actual no-flow time compared with self-perceived no-flow time (subjective, 38 s; objective, 57.5 s; p value<0.001). This discrepancy could cause a 6.5% drop of the chest compression fraction in a resuscitation period of 5 min. Among the EMT personal factors, self-confidence was negatively associated with objective MCPR deployment performance (adjusted OR (aOR) 0.66, 95% CI 0.45 to 0.97, p=0.033) and objective teamwork performance (aOR 0.57, 95% CI 0.34 to 0.97, p=0.037) for EMT-Ps, whereas knowledge was positively associated with objective MCPR deployment performance (aOR 2.15, 95% CI 1.31 to 3.52, p=0.002) and objective teamwork performance (aOR 1.77, 95% CI 1.02 to 3.08, p=0.043) for EMT-Is. Moreover, regarding the self-evaluation of no-flow time, both self-satisfaction and self-abasement were associated with objectively poor teamwork performance.
Conclusions EMTs’ subjective and objective performance was inconsistent during the MCPR simulation. Self-confidence and knowledge were personal factors associated with MCPR deployment and teamwork performance. Both self-satisfaction and self-abasement were detrimental to teamwork during resuscitation.
- accident & emergency medicine
- medical education & training
- education & training (see medical education & training)
Data availability statement
Data are available upon reasonable request.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
- accident & emergency medicine
- medical education & training
- education & training (see medical education & training)
Strengths and limitations of this study
This study compared the subjective and objective out-of-hospital cardiac arrest resuscitation performance of emergency medical technicians (EMTs) in a quantitative manner.
The simulation process involved a mechanical cardiopulmonary resuscitation that is used in the prehospital setting worldwide, indicating that the results might be applied to other regions.
The study had a large sample size of 210 EMTs to test the hypothesis with adequate statistical power.
Using quantitative methods, the presence or absence of the ‘Dunning–Kruger effect’ in prehospital care providers could also be investigated.
The study was limited by a non-randomised design and a fixed crew size of three EMTs in the simulation setting, which may inhere a risk of selection bias and limited generalisability to other crew sizes.
Introduction
Survival of patients following out-of-hospital cardiac arrest (OHCA) depends on many factors, including bystander cardiopulmonary resuscitation (CPR), public access defibrillation and well-organised emergency medical services (EMS).1–3 Among them, emergency medical technicians (EMTs) play a pivotal role because they are usually the first medical personnel resuscitating the victim. EMTs perform CPR under many challenging circumstances, for example, on a quickly moving ambulance, and high-quality manual CPR is difficult and may cause occupational injuries.4–7 Thus, mechanical CPR (MCPR) has been widely applied to improve resuscitation quality and EMT safety.8–10
Although studies have not shown that MCPR in the prehospital setting did improve patient outcomes compared with manual CPR,11–15 current guidelines do support the use of MCPR devices under some circumstances, such as ambulance transport.16 17 To achieve the effective application of MCPR, it is important to eliminate the interruptions of chest compression before the commencement of mechanical compression.8 18 19 As a result, the EMS authorities of Taipei City started to introduce MCPR before ambulance transport, without changing the initial manual CPR procedures, for every OHCA resuscitation since 2018. In the densely populated metropolitan area of Taipei City, MCPR benefits patients and EMTs during ambulance transport and in circumstances when patients need to be carried down staircases. Given that MCPR devices—LUCAS or AutoPulse—are now incorporated into the standard protocol for OHCA resuscitation in Taipei City, sufficient training for the rapid and correct deployment of MCPR devices is crucial.
For medical professionals, a strong belief in their capabilities is helpful for successful resuscitation techniques.20 21 However, studies have also revealed the detrimental effects of overconfidence in both medical diagnosis and management.22 23 Besides, overconfidence might be associated with less competence, first reported in the psychology study as the Dunning–Kruger effect.24 This phenomenon was observed in certain areas25–28 but has never been studied in healthcare providers like EMTs.
Because we had observed, during the quality control processes at the Taipei City Fire Department, that EMTs did not usually accurately perceive the duration of chest compression interruptions (no-flow time) during deployment of MCPR devices, we designed a study to test the gap between self-perceived (subjective) and actual (objective) no-flow time during resuscitation, and the association between resuscitation performance and personal attributes like knowledge, attitude and self-confidence. By evaluating and comparing EMTs’ subjective and objective MCPR performance, we investigated whether the Dunning–Kruger effect is identifiable in EMTs performing resuscitation.
Methods
Study setting and participants
This cross-sectional study investigated EMT-Intermediates (EMT-Is) and EMT-Paramedics (EMT-Ps) of the EMS (Ambulance) Division of the Taipei City Fire Department. Taipei City is the capital of Taiwan, with approximately 2.5 million registered residents. The city’s EMS has a two-tiered response system consisting of a basic life support plus defibrillator (BLS-D) team and an advanced life support (ALS) team. The BLS-D team comprises more than 600 active EMT-Is, who have each completed at least 264 hours of training. They perform defibrillation and insert the supraglottic airway (SGA). The ALS team comprises more than 100 active EMT-Ps, specially trained for at least 1280 hours. The ALS team providers are authorised to perform endotracheal intubation and intravenous injections of resuscitation medications, like epinephrine, atropine and amiodarone, etc.
Each of the 41 BLS-D stations in Taipei City has two BLS-D ambulances staffed by two EMT-Is, although sometimes there is a third member (usually a volunteer EMT). Each of the four ALS stations in Taipei City has three ALS ambulances staffed by three EMT-Ps.
The Taipei City Fire Department has a single dispatch centre to process all incoming EMS calls. All dispatchers are required to complete at least 40 hours of training on priority dispatch. BLS-D is the universal response for all dispatch calls. For OHCA cases, additional ALS teams would be dispatched to the scene with BLS-D teams.
Recruitment and simulation process
Study participants were selected from 734 EMTs in 45 Taipei City Fire Department stations between 19 January and 16 April 2019. To qualify for participation, EMTs must have completed: (1) the re-education training course in the previous year, (2) the 2-hour high-performance CPR training course and (3) the 4-hour MCPR training course.
We visited about two or three stations every few days for 4 months. There was no randomisation process, and the impromptu visits depended on our daily duties. Selected stations were not notified before visitation to prevent them from preparing beforehand. We explained the purpose and process of the study after arriving at each station and picked EMTs who were on BLS-D/ALS duties at the time. After written informed consent was acquired, the three EMTs were randomly assigned a role as either team leader, compressor or airway manager. EMTs who participated in the study completed a questionnaire before the test scenario, stating their personal data, resuscitation-related knowledge, attitude and self-confidence. We picked and tested a few more EMT teams at the station if time permitted.
The MCPR device in the test scenario was either LUCAS or AutoPulse, depending on which was available at each station. The CPR training manikin was a Resusci Anne quality CPR (QCPR) device produced by Laerdal Medical, which recorded chest compressions’ rate, depth and interruptions. The entire simulation process was recorded by two video cameras placed at the foot and head ends of the manikin, respectively.
The test scenario started with the compressor and airway manager performing CPR on the manikin with a 30:2 compression–ventilation ratio. Besides performing bag-valve-mask (BVM) ventilation, the airway manager also set up the automated external defibrillator (AED) and inserted the SGA device—i-gel. Ventilation was changed to one breath every five to six seconds after successfully placing the SGA device. During the second AED heart rhythm analysis, the compressor and airway manager switched roles, and the compressor and airway manager started MCPR deployment. The team leader observed the entire process and offered advice, encouragement or a helping hand to the team members. The simulation process ended once the MCPR device was deployed.
After the test scenario, each EMT self-evaluated their performance and no-flow time during the first heart rhythm analysis, BVM ventilation, i-gel insertion, second heart rhythm analysis and MCPR deployment, respectively. These analyses were documented as subjective outcomes. An expert committee consisting of two experienced paramedics gave objective scores by examining the videos and measured actual no-flow time using timers.
Questionnaire and score sheet
The questionnaire (online supplemental appendix I) covered personal data such as EMT certification levels, years of service and prior resuscitation experience. The first three sections contained questions about knowledge, attitude and self-confidence. They were designed according to a previous study by Hsieh et al.29 Knowledge was quantified by the number of questions answered correctly in section 2 of the questionnaire, with a score ranging from 0 to 10. Attitude was quantified by questions 1–6 in section 3, with a score ranging from 6 to 30. Self-confidence was quantified by questions 7–12 in section 3, with a score ranging from 6 to 30. The fourth section was the self-evaluation of performance and no-flow time.
Supplemental material
The expert committee’s objective score sheet (online supplemental appendix II) contained sections on chest compression, airway management, MCPR deployment and overall performance—each included three to five scoring items. MCPR deployment performance was quantified by combining the scores ranging from 2 to 10 of items 1 and 2 in section 2. Teamwork performance was quantified by combining scores ranging from 4 to 20 of items 1–4 in section 3. The grading of airway management (BVM ventilation and i-gel operation), MCPR deployment and overall performance was done by two EMT-P instructors with at least 15 years of emergency management experience. The CPR quality gradings before MCPR deployment were automatically recorded by the QCPR device.
The questionnaire and the score sheet went through three Delphi rounds by five physicians, with all questions and scoring items achieving at least 80% of the content validity index.
Pilot test and sample size estimation
A pilot test with eight three-crew teams containing 12 EMT-Is and 12 EMT-Ps had been performed for questionnaire optimisation, scoring standard adjustment and sample size estimation. The study was estimated to require a sample size of 36 to achieve a power of 80% and a significance level of 5% (two-sided) to detect a mean difference of 4 s between subjective and objective no-flow time, assuming a SD of 8.3 s.
Statistical analysis
Continuous variables included no-flow time (in seconds) calculated by timers, questionnaire scores, age, years of service, number of OHCA dispatches, number of OHCA survivals and number of MCPR deployments. These variables were summarised in medians with IQR. Alternatively, categorical variables included the EMT (EMT-P or EMT-I) level, sex, appraisal of MCPR deployment and teamwork performance. Inferior performance was defined as scores lower than the 25th percentile, and superior performance was defined as scores higher than the 75th percentile. We chose to focus on the extremes of the upper and lower quartiles to evaluate factors contributing to the best and worse resuscitation performance. The categorical data were analysed with Pearson’s χ2 test. Inferential statistics were done using the Wilcoxon signed-rank (matched pairs) test, the two-sample Wilcoxon rank-sum (Mann–Whitney) test and simple and multivariable logistic regression analyses. Cohen’s Kappa statistics were used to measure inter-rater reliability. The linear regression model was applied to analyse the relationship between self-cognition and objective teamwork performance. Data were collected and analysed using R-3.5.3 and SPSS Statistics V.22.0 (IBM, USA).
Ethics approval
This study was approved by the Research Ethics Committee of the National Taiwan University (NTU-REC No. 201811HS013).
Patient and public involvement
Patients and the public were not involved in the study.
Results
Seventy groups (44 EMT-I groups, 20 EMT-P groups and 6 mixed groups of 2 EMT-Is and 1 EMT-P each) were included in this study, as shown in figure 1. Their baseline characteristics and self-confidence in MCPR deployment are shown in table 1. The subjective no-flow time was significantly longer than the objective no-flow time among all EMTs, as shown in table 2. This discrepancy also existed among EMT-P and EMT-I subgroups (table 2). EMT-Ps had better scores than EMT-Is in both subjective and objective MCPR deployment performance (table 2). Self-confidence was negatively associated with objective MCPR deployment performance among EMT-Ps, while knowledge was positively associated with objective MCPR deployment performance among EMT-Is (table 3). Self-confidence was negatively associated with objective teamwork performance among EMT-Ps, while knowledge was positively associated with objective teamwork performance among EMT-Is (table 4).
Overview of the recruitment process. CPR, cardiopulmonary resuscitation; EMT, emergency medical technician; EMT-I, EMT-Intermediate; EMT-P, EMT-Paramedic; MCPR, mechanical cardiopulmonary resuscitation.
Baseline characteristics and self-confidence in MCPR deployment
Subjective versus objective no-flow time and MCPR deployment performance
Factors† related to the MCPR deployment performance‡ among EMT-Ps and EMT-Is
Factors† related to the teamwork performance‡ among EMT-Ps and EMT-Is
The no-flow time during MCPR deployment negatively correlated with objective MCPR deployment performance (figure 2A) and objective teamwork performance (figure 2B). Additionally, self-satisfaction and self-abasement in evaluating no-flow time were associated with poor objective teamwork performance, but only the former showed statistical significance (figure 3). Unlike self-confidence, self-satisfaction was assessed based on objective no-flow time greater than subjective no-flow time, whereas self-abasement was assessed based on subjective no-flow time greater than objective no-flow time.
Correlation between no-flow time during MCPR deployment (range, 7–53 s) and resuscitation performance among 210 EMTs (70 groups). (A) Correlation between no-flow time and MCPR deployment score (items 1 and 2 in section 2 of the score sheet; range, 2–10), R2=0.58. (B) Correlation between no-flow time and teamwork score (items 1–4 in section 3 of the score sheet; range, 5–20), R2=0.12. EMT, emergency medical technician; MCPR, mechanical cardiopulmonary resuscitation.
Association between self-cognition (objective no-flow time minus subjective no-flow time during MCPR deployment, in seconds; range, −14 to 42) and objective teamwork performance (items 1–4 in section 3 of the score sheet; range, 5 to 20). Closed dots represent EMT-Ps, whereas open dots represent EMT-Is. EMT-I, emergency medical technician-intermediate; EMT-P, emergency medical technician-paramedic; MCPR, mechanical cardiopulmonary resuscitation.
Cohen’s Kappa coefficients of the objective gradings of MCPR device operation (online supplemental appendix II, section 2, item 1), correct MCPR device deployment (online supplemental appendix II, section 2, item 2) and overall performance (online supplemental appendix II, section 3) were 0.803, 0.822 and 0.736, respectively. All the p values were less than 0.001.
Discussion
There were three major findings in this study. First, the subjective and objective MCPR performance of EMTs was inconsistent. The Dunning–Kruger effect was identifiable in EMTs performing resuscitation in our study. Second, many EMT personal factors such as self-confidence and knowledge contributed to resuscitation performance, ignoring EMT certification levels. Third, EMTs who were neither self-satisfied nor self-abased in evaluating no-flow time had better objective teamwork performance.
Both EMT-Ps and EMT-Is lacked sufficient self-awareness of their no-flow time during resuscitation. Among all EMTs, the median no-flow time was 19.5 s longer than the self-perceived length, which would cause a 6.5% drop in chest compression fraction in a 5-min resuscitation period. The discrepancy between subjective and objective no-flow time has already been reported, and objective feedback was recommended to decrease the duration.30 We posit that recognition of interruption time is not accurate while EMTs concentrate on specific emergency medical techniques, particularly during urgent OHCA resuscitations. In these situations, adequate leadership and team-focused resuscitation should play a beneficial role.31 For example, team leader verbalisation makes EMTs aware of errors or encourages them to keep up the good work, reinforcing positive performance outcomes.32 33
During MCPR deployment, EMT-Ps had subjective and objective median no-flow time of 8 and 13.5 s, respectively. These durations were both shorter than that of EMT-Is (10 and 17 s, respectively). The discrepancy between subjective and objective no-flow time was also smaller in the EMT-P group. This indicates that EMT-Ps are more capable of recognising and reducing unnecessary disruptions during resuscitation, which is expected because EMT-Ps are better trained. However, the five-and-a-half-second gap in the EMT-P group indicates that even well-trained EMTs have unrecognised shortcomings in their resuscitation process.
For EMTs, MCPR deployment confidence was positively associated with the number of completed deployments, as shown in table 1. This association has also been observed in other resuscitation techniques, such as endotracheal intubation.21 Operator confidence is likely to affect resuscitation performance. Davis et al.34 have revealed a positive association between self-confidence and ultrasound accuracy by novice emergency physicians. In contrast, we found a negative association between self-confidence and MCPR deployment performance among EMT-Ps. We speculate that operator overconfidence plays a detrimental role because excessive complacency leads to detail neglect during resuscitation.
For EMT-Is, knowledge was positively associated with MCPR deployment and teamwork performance. EMT-Is may be unfamiliar with OHCA resuscitation because OHCA cases comprise a small fraction of their usual emergency medical dispatches. We believe that their resuscitation performance and outcomes will improve by enhancing their knowledge base. As for EMT-Ps, the negative association between self-confidence and performance indicates that highly trained EMTs may be careless. We believe that these shortcomings can be improved with a debrief after resuscitation. The debriefing process can enhance objective perception about resuscitation performance, balancing self-satisfaction or self-abasement.
The no-flow time negatively correlated with objective MCPR deployment performance and objective teamwork performance. The latter played a minor role, with a more moderate slope of regression line. However, the negative correlation between no-flow time and objective teamwork performance indicates the merit of teamwork-focused resuscitation in favour of personal ability. Planning, leadership and communication are the key elements of successful coordination during resuscitation.35 36 Other studies also indicated that simulation-based training could improve quality of care and treatment outcomes of resuscitation.37 38 Therefore, EMS authorities should develop team-oriented and simulation-based training models for EMT training and re-education courses. Additionally, instructor-led, low-dose but high-frequency training may be beneficial by reinforcing resuscitation skills retention.39 Moreover, our study first observed the Dunning–Kruger effect in EMTs’ resuscitation performance. This finding is important for the training of EMTs because even just keeping in mind that the effect exists can help the trainers and trainees stay humble, accept criticism and keep learning new knowledge and skills.
This study has the following limitations. First, the participant recruitment process lasted nearly 4 months, and the impromptu visits and selections were not randomised, which may inhere a risk of selection bias. Although we visited stations without prior notification, the EMTs who were tested later in the recruitment period might still have been aware and able to prepare; they might have been informed of the details of our simulation from their colleagues. Regardless, the results of participants who tested earlier showed no significant difference to those who tested later. Second, we discovered that almost all stations used half-length manikins as their regular training materials. Due to the inaccurate anatomical structure around the axillary region, the MCPR devices (LUCAS or AutoPulse) could not be positioned correctly on most half-length manikins. This would affect the scores of MCPR deployment among many EMTs, yet those effects were minor enough to be ignored. Third, each BLS-D station was equipped with either LUCAS or AutoPulse, which were fundamentally different devices with different deployment techniques. However, we purposely did not control for the brand of MCPR device so that we could test EMTs with devices they were familiar with. Fourth, in our simulation study, we applied a fixed crew size of three EMTs, so the results might not be applied to EMS teams with a different size of resuscitation crew.40 Finally, despite applying the Delphi method to test the questionnaire, there were still limitations regarding the external validity of participant attributes such as attitude and self-confidence.
Conclusions
EMT’s subjective resuscitation performance and no-flow time did not match the objective results. Self-confidence was negatively associated with EMT-Ps’ MCPR deployment and teamwork performance. In contrast, knowledge was positively associated with EMT-Is’ performance, which indicated that the training courses for different EMT levels should focus on various aspects. The negative association between objective teamwork performance and self-satisfaction or abasement highlighted the importance of self-awareness in emergency resuscitation procedures. Finally, the Dunning–Kruger effect first observed in EMT’s resuscitation performance may provide insight into the adequate training strategy for EMTs.
Data availability statement
Data are available upon reasonable request.
Ethics approval
This study involves human participants. This study was approved by the Research Ethics Committee of the National Taiwan University (NTU-REC No. 201811HS013). Participants gave informed consent to participate in the study before taking part.
Acknowledgments
The authors acknowledge the excellent performance of the emergency medical technicians and the quality assurance of the Emergency Medical Services (Ambulance) Division of the Taipei City Fire Department and their commitment and accomplishments, which significantly improve prehospital care. We thank all enrollees for agreeing to participate in this study. We thank Dr Sheng-Wen Hou, Dr Ming-Ju Hsieh and paramedic Wei-Chi Chu for participating in the expert committee. We thank Dr Chih-Wei Yang for his help in designing the score sheet. We thank Dr Chin-Hao Chang and Su-Mei Wang of the National Taiwan University Hospital Statistical Consulting Unit for their valuable opinions on the data presentation of this article. We thank Yu-Yun Wu for her assistance with the statistical analysis.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
W-SY and PY are joint first authors.
Twitter @pyen93, @DrBrian_Tw
W-SY and PY contributed equally.
Contributors W-SY conducted the study and performed the statistical analysis. PY drafted and revised the manuscript. W-SY and W-CChiang revised the manuscript. W-SY, W-CChie, MH-MM and W-CChiang conceived the study design. W-SY, PY, Y-CW and Y-CC supervised the data collection. All authors contributed substantially to the preparation and revision of this article. W-SY and PY contributed equally as the first author to this article. W-CChiang takes responsibility for this article as a whole.
Funding This study was supported by the Ministry of Science and Technology of Taiwan (Grant No: MOST 108-2314-B-002-130-MY3 and MOST 109-2314-B-002-154-MY2).
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.