You are here

  • Home
  • Archive
  • Volume 11, Issue 8
  • eHBB: a randomised controlled trial of virtual reality or video for neonatal resuscitation refresher training in healthcare workers in resource-scarce settings

Article Text

Article menu
XML
Medical education and training
Original research
eHBB: a randomised controlled trial of virtual reality or video for neonatal resuscitation refresher training in healthcare workers in resource-scarce settings
  1. Rachel Umoren1,
  2. Sherri Bucher2,
  3. Daniel S Hippe3,
  4. Beatrice Nkolika Ezenwa4,
  5. Iretiola Bamikeolu Fajolu4,
  6. Felicitas M Okwako5,
  7. John Feltner1,
  8. Mary Nafula5,
  9. Annet Musale5,
  10. Olubukola A Olawuyi4,
  11. Christianah O Adeboboye4,
  12. Ime Asangansi6,
  13. Chris Paton7,
  14. Saptarshi Purkayastha8,
  15. Chinyere Veronica Ezeaka4,
  16. Fabian Esamai5
  1. 1Department of Pediatrics, University of Washington, Seattle, Washington, USA
  2. 2Department of Pediatrics, Indiana University School of Medicine, Indianapolis, Indiana, USA
  3. 3Department of Radiology, University of Washington, Seattle, Washington, USA
  4. 4Department of Paediatrics, University of Lagos College of Medicine, Lagos, Nigeria
  5. 5Alupe University College, Busia, Kenya
  6. 6eHealth4everyone, Abuja, Nigeria
  7. 7Centre for Tropical Medicine, Nuffield Department of Clinical Medicine, Oxford, UK
  8. 8Department of BioHealth Informatics, Indiana University-Purdue University at Indianapolis, Indianapolis, Indiana, USA
  1. Correspondence to Dr Rachel Umoren; rumoren{at}uw.edu

Abstract

Objective To assess the impact of mobile virtual reality (VR) simulations using electronic Helping Babies Breathe (eHBB) or video for the maintenance of neonatal resuscitation skills in healthcare workers in resource-scarce settings.

Design Randomised controlled trial with 6-month follow-up (2018–2020).

Setting Secondary and tertiary healthcare facilities.

Participants 274 nurses and midwives assigned to labour and delivery, operating room and newborn care units were recruited from 20 healthcare facilities in Nigeria and Kenya and randomised to one of three groups: VR (eHBB+digital guide), video (video+digital guide) or control (digital guide only) groups before an in-person HBB course.

Intervention(s) eHBB VR simulation or neonatal resuscitation video.

Main outcome(s) Healthcare worker neonatal resuscitation skills using standardised checklists in a simulated setting at 1 month, 3 months and 6 months.

Results Neonatal resuscitation skills pass rates were similar among the groups at 6-month follow-up for bag-and-mask ventilation (BMV) skills check (VR 28%, video 25%, control 22%, p=0.71), objective structured clinical examination (OSCE) A (VR 76%, video 76%, control 72%, p=0.78) and OSCE B (VR 62%, video 60%, control 49%, p=0.18). Relative to the immediate postcourse assessments, there was greater retention of BMV skills at 6 months in the VR group (−15% VR, p=0.10; −21% video, p<0.01, –27% control, p=0.001). OSCE B pass rates in the VR group were numerically higher at 3 months (+4%, p=0.64) and 6 months (+3%, p=0.74) and lower in the video (−21% at 3 months, p<0.001; −14% at 6 months, p=0.066) and control groups (−7% at 3 months, p=0.43; −14% at 6 months, p=0.10). On follow-up survey, 95% (n=65) of respondents in the VR group and 98% (n=82) in the video group would use their assigned intervention again.

Conclusion eHBB VR training was highly acceptable to healthcare workers in low-income to middle-income countries and may provide additional support for neonatal resuscitation skills retention compared with other digital interventions.

  • paediatrics
  • information technology
  • education & training (see medical education & training)
  • neonatal intensive & critical care
  • medical education & training
  • neonatology

Data availability statement

Data are available on reasonable request. Deidentified data are available on request from Dr. Rachel Umoren at nestprog@uw.edu.

https://creativecommons.org/licenses/by/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution 4.0 Unported (CC BY 4.0) license, which permits others to copy, redistribute, remix, transform and build upon this work for any purpose, provided the original work is properly cited, a link to the licence is given, and indication of whether changes were made. See: https://creativecommons.org/licenses/by/4.0/.

https://doi.org/10.1136/bmjopen-2020-048506

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Strengths and limitations of this study

    • This study was a multicentre, randomised controlled trial of mobile virtual reality or video to support neonatal resuscitation skills retention in nurses and midwives who provide neonatal resuscitation in two low-income to middle-income countries.

    • The study used an evidence-based Helping Babies Breathe second edition curriculum designed for neonatal resuscitation training in resource-scarce settings.

    • Healthcare workers accessed the digital interventions on mobile phones in the 6 months following in-person Helping Babies Breathe training.

    • Healthcare workers were recruited from secondary and tertiary healthcare settings in urban and semi-urban resource-scarce settings, so the study findings may not apply to healthcare workers in high-resource, primary healthcare or rural settings.

    Introduction

    In 2019, there were 2.4 million deaths among infants under 28 days of age.1 2 These neonatal deaths now account for 47% of global under 5 years child mortality,2 and most cases are preventable.3–5 The majority of these deaths occur in low-income to middle-income countries (LMICs). Nigeria, with a neonatal mortality rate of 36 deaths per 1000 live births and Kenya at 21 deaths per 1000 live births in 2019,1 are at significant risk of failing to meet the United Nations Sustainable Development Goal 3 to reduce neonatal mortality to 12 per 1000 live births by 2030.3

    Intrapartum asphyxia or lack of breathing at birth is a leading cause of neonatal mortality.6 7 Training healthcare workers in neonatal resuscitation using the Helping Babies Breathe (HBB) curriculum builds competency and reduces newborn morbidity and mortality.8–13 However, neonatal resuscitation skills are quickly lost after trainings that use the traditional approach of small group facilitated classroom training.14–19 For this reason, periodic refresher training with ‘low-dose high frequency’ manikin-based simulations are recommended to support neonatal resuscitation skills retention.20 21 Unfortunately, access to manikin-based simulation is limited by trainer, space and equipment availability.22 23 Yet, the high penetration of smartphones and cellular network connectivity in urban and rural areas in LMICs, makes innovative simulation training feasible using mobile virtual reality (VR) simulations for healthcare workers (HCWs) who provide care in health facility and community-based settings.

    VR simulations are effective educational tools and can be engaged at the learner’s convenience, on their own smartphone, with game-based automated feedback that is ideal for episodic learning.24–26 However, little is known of their feasibility, acceptability or effectiveness for neonatal resuscitation skills retention in LMICs. We hypothesised that mobile VR simulation refresher training would address challenges to the quality of newborn resuscitation related to the maintenance of HCW knowledge and skills over time addressing the lack of standardised dissemination of updates to recommended practice and high rates of staff turnover. The objective of this study was to evaluate the impact of eHBB VR used with in-person neonatal resuscitation training on neonatal resuscitation educational indicators and performance outcomes, in comparison to other digital refresher training modalities.

    Methods

    Study setting

    The study was conducted in Lagos, Nigeria and Busia, Western Kenya. Twelve healthcare facilities (nine secondary and three tertiary) were located in Nigeria while eight facilities were located in Kenya. The healthcare facilities were located in urban and semi-urban areas and all have maternal and newborn services with newborn bed capacity ranging from 2 to 80 beds and delivery and neonatal unit staffing capacity from 7 to 124 nurses (see online supplemental file 1).

    Participants

    Study participants consisted of nurses and nurse-midwives assigned to labour and delivery, operating room and newborn care units. Site coordinators or research assistants requested contact numbers, units and wards of potential participants from head nurses at identified facilities. Research assistants contacted individuals to determine eligibility and obtained consent (see online supplemental file 2).

    Inclusion criteria

    Nurses and midwives who participate in deliveries and who provide neonatal resuscitation to inborn or outborn infants and provide study consent.

    Exclusion criteria

    Those who had attended a neonatal resuscitation training course in the 1 year preceding the study; individuals who did not provide neonatal resuscitation as part of their duties or would be unavailable or unwilling to participate in follow-up study activities throughout the 6-month postinitial training period.

    Randomisation

    Study IDs generated for each country site were randomly assigned via a computer-generated algorithm to the VR, video and control groups by a US-based study coordinator. Participants were enrolled and assigned a study ID before the HBB course by local study coordinators. Each participant received an Android study phone, preloaded with permission-based access linked to their study ID, via the mobile Helping Babies Survive powered by District Health Information System (DHIS2) app (mHBS/DHIS2), to the participant’s assigned digital intervention. The data analysis team was blinded to the study assignments.

    HBB course structure

    The HBB provider course (second edition)27 was taught by study HBB master trainers as 1 day, 8-hour long sessions from December 2018 to August 2019. A 30 min orientation was provided on use of the mHBS/DHIS2 app, including how to access the assigned digital intervention. All participants had access to a digitised HBB provider manual through the mHBS/DHIS2 app. The VR group in addition accessed the eHBB VR simulations which consisted of three interactive three-dimensional simulation scenarios representing care of a newborn requiring routine care, some resuscitation and prolonged resuscitation with positive pressure ventilation. The features of eHBB VR have been previously described and the application is available for free download.26 28 The neonatal resuscitation video used by the video group featured preparation for delivery and the resuscitation of a newborn requiring positive pressure ventilation.29 None of the interventions required internet for use. A total of 274 HCWs participated in the in-person HBB training.

    Precourse and postcourse assessments

    Standardised knowledge and skills assessments were conducted by trained research assistants. The HBB knowledge check (15 of 18 multiple-choice questions, ≥80% required to pass) and bag-and-mask ventilation skill check (BMV; 14 of 14 items required to pass) were conducted precourse and postcourse along with the objective structured clinical examination (OSCE) A checklist on preparation for delivery and initial steps of resuscitation (9 out of 12 items and 3 required items to pass). In addition, the postcourse assessment included the OSCE B checklist on prolonged newborn resuscitation (17 out of 23 items and 6 required items to pass). HBB checklists are available for free download from the American Academy of Pediatrics.30 A demographic survey was completed (figure 1).

    Figure 1
    Figure 1

    Study diagram. BMV, bag-and-mask ventilation; HBB, helping babies breathe; LDHF, low-dose high frequency; VR, virtual reality.

    Postcourse interventions and follow-up

    Following the course, participants were encouraged to use their assigned digital intervention weekly and to engage in standard bag-and-mask skills practice with a manikin at the HBB practice corner set up at their facility. Post-course assessments were repeated at 1, 3, and 6 months after the class. A follow-up survey was completed.

    Data collection

    Data were collected in person by study staff who had completed a HBB second edition master trainer course by experienced HBB master trainers. Staff used the mHBS/DHIS2 tracker app for offline data collection.26 The mHBS tracker app contained digitised HBB knowledge check, BMV skill check and OSCE A and OSCE B checklist and was used by the participants to report their HBB corner practice. The mHBS trainer app separately tracked educational interventions access and use. To standardise data collection and feedback to study participants, an enhanced neonatal simulator, called NeoNatalie Live (Laerdal Medical) was used for BMV. Compared with the low-fidelity NeoNatalie simulators used for HBB training (including the HBB practice corners in this study), NeoNatalie Live manikin can be programmed to simulate key physiological parameters, such as variable rates of lung stiffness and heart rate and provides auditory and visual cues, in the form of ‘crying’ and increased heart rate when the end-user provides BMV.31 In addition, brief automated feedback for the end-user is provided using a Bluetooth-connected tablet device at the end of the assessment as ‘well done’ or ‘needs improvement’ based on bag and mask performance. The use of the NeoNatalie Live manikin software enabled the correlation of observer collected metrics with manikin collected data. The automated feedback provided by Neonatalie Live was the only feedback provided following each assessment.31

    Patients or public involvement

    Patients or the public were not involved in the design, or conduct, or reporting, or dissemination plans of our research.

    Sample size calculations

    We hypothesised that there will be at least a 20% difference in the proportion of subjects who pass OSCE B at the 6-month evaluation between the VR group or video group and the control group. A sample size of 83 subjects per group would provide 80% power to detect a difference in pass rates between groups if the true pass rates were 85% and 65%, respectively, based on a two-sided α=0.05. The required total sample size for the three groups (VR, video and control) was 249. We recruited 274 participants total to allow for 10% dropout over the 6-month follow-up period.

    Data analysis

    An intention-to-treat analysis was performed, where participants were grouped according to their randomly allocated experimental group (VR, video or control) regardless of their actual exposure. Fisher’s exact test was used to test for any differences in pass rates among the three groups for each of the study evaluations: BMV skills assessment, and standardised simulations of routine care and initial resuscitation (OSCE A) and prolonged resuscitation (OSCE B). Post hoc pairwise comparisons and comparisons between demographic groups were also performed using Fisher’s exact test. Within-group comparisons of evaluation results between timepoints were performed using the sign test. Participant exposure to the interventions (time in the mHBS trainer app) and self-reported clinical activity during the follow-up period were compared between experimental groups using the Kruskal-Wallis test and Wilcoxon rank-sum test. All statistical calculations were conducted with the statistical computing language R (V.4.0.0; R Foundation for Statistical Computing, Vienna, Austria). Throughout, two-sided tests were used, with statistical significance defined as p<0.05.

    Results

    Recruitment, training and follow-up were conducted concurrently at participating sites from December 2018 to August 2019 with follow-up continuing until February 2020. Of the 394 nurses and nurse-midwives identified who attended deliveries at the participating sites, 274 consented to participate in the study. Of the 274 participants, 265 (97%) completed a 6-month assessment, with a similar dropout rate in each group (p=0.52 for the difference between groups, figure 2).

    Figure 2
    Figure 2

    Consolidated Standards of Reporting Trials diagram. eHBB, electronic Helping Babies Breathe.

    Most participants were female (91%), nurse-midwives or midwives (51%), who worked in the labour and delivery ward (72%). Nearly all owned a smartphone (table 1). Neonatal resuscitation knowledge and skills assessments were conducted immediately after the in-person course. There were no differences in knowledge check scores (VR 18 (17–18), video 18 (17–18), control 18 (17–18), p 0.76) or pass rates on the BMV (VR 46% (n=83), video 46% (n=84), control 52% (n=79), p=0.72), OSCE A (VR 76% (n=91), 78% (n=95), 72% (n=88), p=0.63) and OSCE B (VR 59% (n=91), video 73% (n=95), control 62% (n=88), p=0.13) assessments across groups on the immediate postcourse assessments.

    View this table:
    Table 1

    Demographics of study participants

    Neonatal resuscitation knowledge and skills on follow-up assessments

    Neonatal resuscitation skills assessments were conducted at 1, 3 and 6 months after the in-person course. Differences in pass rates on the BMV, OSCE A and OSCE B checklists across groups on the 6-month postcourse assessments were not statistically significant (table 2).

    View this table:
    Table 2

    Comparison of 6-month postcourse pass rates between groups

    To determine whether pass rates were impacted by years of experience, age, profession, ward or prior HBB training, BMV skills, OSCE A and OSCE B pass rates were compared between groups, one at a time. Participants with <5 years of experience performed better on the OSCE A and OSCE B immediate postcourse assessments (p=0.022 and p=0.034, respectively). Nurse-midwives performed better on BMV skills (p<0.001), OSCE A (p<0.001) and OSCE B immediate postcourse assessments (p=0.011) compared with nurses, although pass rates were similar for all three tests at 6 months (p=0.14–0.89). Ward assignment to newborn unit, neonatal intensive care unit (NICU) or postnatal ward was also associated with greater immediate postcourse pass rate on BMV skills (p<0.001), OSCE A (p=0.001) and OSCE B assessments (p<0.001). At the 6-month follow-up assessment, there were no significant differences in BMV skills, OSCE A or OSCE B pass rates by country site, years of experience, age, ward and HBB training >1 year prior to the study (table 3).

    View this table:
    Table 3

    Pass rates compared between demographic groups

    Neonatal resuscitation performance changes over time

    There was a decline in performance on neonatal resuscitation skills assessments at 1 month across all groups with a variable degree of recovery of skills by the 3-month and 6-month assessments (figure 3).

    Figure 3
    Figure 3

    Pass rates of bag-and-mask ventilation (BMV) skills check, objective structured clinical examination (OSCE) A and OSCE B assessments over time. Pass rates of BMV skills check, OSCE A and OSCE B assessments over time. Immediate postcourse, 1 month, 3 months and 6 months assessments indicated by solid circles. *Statistically significant changes within each experimental group (virtual reality (VR), video and control) from the immediate postcourse assessment. †Significant changes from the 1-month assessment.

    BMV skill pass rates

    BMV skills showed a decline at the 1-month assessment and remained significantly lower than the immediate postcourse baseline in all groups at 3 months (−23% VR, p=0.001, –25% video, p<0.001, –31% control, p<0.001) and in the video and control groups at 6 months (−15% VR, p=0.10, –21% video, p<0.01, –27% control, p=0.001).

    OSCE A pass rates

    While pass rates decreased on the OSCE A assessments across all groups at 1 month, the groups improved over time and OSCE A pass rates were close to the immediate postcourse baseline at 6 months (−1% VR, −1% video, 0% control, p=0.83), with the VR group demonstrating an earlier recovery of skills (−2% VR, −9% video, −7% control, p=0.52 at 3 months). At 6 months, the VR group showed good performance on questions: prepares the area for ventilation and checks function of bag, mask and suction device (VR 92% (n=87), video 89% (n=92), control 84% (n=86), p=0.25), recognises baby is crying and breathing well (VR 100% (n=87), video 99% (n=92), control 95% (n=86), p=0.07) and communicates with mother (VR 94% (n=87), video 86% (n=92), control 86% (n=86), p=0.12), although these differences were not statistically significant.

    OSCE B pass rates

    OSCE B pass rates were higher than the immediate postcourse baseline at 3 and 6 months in the VR group (+4% at 3 months, p=0.64; +3% at 6 months, p=0.74) and lower in the video (−21% at 3 months, p<0.001; −14% at 6 months, p=0.07) and control groups (−7% at 3 months p=0.43; −14% at 6 months, p=0.10). Across groups, the performance was sustained on some items of the OSCE B skills checklist that are necessary to improve ventilation such as reapply mask and reposition head, while other recommended steps such as clear mouth and nose of secretions, open the mouth and squeeze bag harder, showed a greater decline in performance (figure 4). On post hoc analysis of OSCE B assessments at the 6-month follow-up, there was a statistically significant difference between the VR and control groups on the frequency of performing the steps: opens mouth slightly (54% vs 37%, p=0.03) and squeezes bag harder (75% vs 59%, p=0.04) and providing the target ventilation rate of 30–50 breaths per minute (86% VR vs 73% control, p=0.04). Differences in performance between the video and control groups were not statistically significant on these metrics: opens mouth slightly (47% vs 37%, p=0.23), squeezes bag harder (73% vs 59%, p=0.06) and ventilation rate of 30–50 breaths per minute (85% video vs 73% control, p=0.06).

    Figure 4
    Figure 4

    Objective structured clinical examination (OSCE) B skills performance on critical steps of ventilation. VR, virtual reality.

    Participant exposure to interventions and clinical activities

    Participants were assigned access to study interventions through the mobile Helping Babies Survive (mHBS) app and were encouraged to self-report HBB corner practice and deliveries assisted during the follow-up period. The median user time spent in the mHBS trainer app was 103 (85–126) min. This software reported metric reflected the time spent accessing the educational interventions in all groups. There was no difference between groups in number of minutes spent using the mHBS trainer 103 (85–126) over the 6-month follow-up period (VR 101 (81–120), video 108 (87–133), control 102 (87–126), p=0.36). There was no significant difference in self-reported clinical activities with median number of deliveries assisted (VR 25 (5–64), video 25 (12–75), control 28 (7–108), p=0.51) and median deliveries requiring BMV (VR 9 (3–20), video 10 (4–24), control 9 (4–26), p=0.67). The median HBB corner practice days were also similar across groups (VR 16 (6–42), video 20 (7–38), control 16 (7–51), p=0.86)).

    Participant feedback

    Participant feedback indicated overall positive impressions of the VR and video refresher training interventions. On a 6-month follow-up survey with a Likert scale of 1–5 with 1 being strongly disagree and 5 being strongly agree, VR group participants agreed/strongly agreed that eHBB VR was easy to use (92%, n=82), realistic (90%, n=81) and provided valuable clinical practice (92%, n=81) and feedback (90%, n=68). A majority of the video group participants also agreed/strongly agreed that the video was easy to use (88%, n=83), realistic (96%, n=81) and valuable for clinical practice (85%, n=81). If given the opportunity, 95% (n=65) of VR and 98% (n=82) of video respondents would use their assigned intervention again.

    Discussion

    This is the first randomised controlled trial that assesses the impact of mobile VR training for neonatal resuscitation skills retention in HCWs in a resource-scarce setting after standard in-person HBB training. Mobile VR training was highly feasible and acceptable to HCWs in a LMIC setting. Previous reports on neonatal resuscitation training using the HBB curriculum have demonstrated a decline in skills within months of training which may interfere with transfer of skills to clinical practice.14–16 32 33 In this study of digital interventions for neonatal resuscitation skills retention, we found that digital interventions such as VR and video used for refresher training supported the retention of neonatal resuscitation skills in HCWs in Nigeria and Kenya. While the 6-month postcourse performance was similar across groups, when compared with immediate postcourse performance, the decreases in BMV skills pass rates over time were significant in the video and control groups but not in the VR group. Also, contrary to the expected decline in performance over time, OSCE B pass rates were higher at 3 and 6 months than immediately postcourse in the VR group, suggesting that the VR training may provide additional support for the skills needed for prolonged neonatal resuscitation.

    Both VR and video have been described for training in HCWs.24 34–40 Video is a familiar medium but has been long considered a passive learning modality which should be combined with an active learning modality such as manikin-based practice or simulation.22 41 Virtual simulation, an active learning modality, is thought to support learning through repetition, user engagement and identity formation.42 43 Although the overall performance of the video and VR groups was similar at the 6-month assessment, the VR group demonstrated an increased performance over time on some neonatal resuscitation skills that have been demonstrated gaps in educational simulation-based settings such as the steps to improve BMV.44 In settings where in-person refresher training is costly and potentially challenging, and may be even more difficult within the context of COVID-19 concerns, digital and telehealth interventions may adequately support the retention of neonatal resuscitation knowledge and skills.45

    Little is known about the feasibility and acceptability of VR as a novel educational modality for training HCWs in resource-scarce settings. Previous descriptions of mobile VR use with school-age children in LMICs suggest that mobile VR simulations can be used to demonstrate real-world phenomena, illustrate abstract concepts and motivate learners.46 After using VR, students asked questions that reflected a deeper level of curiosity, engagement and reflection on lesson topics. They also took ownership of the programme by recharging mobile devices and creating their cardboard viewers.46 Digital interventions such as VR may provide engaging, individualised and incentivised practice opportunities.39 A survey of HCWs’ perspectives on simulation-based training in Nigeria showed a lack of awareness of VR training, but willingness to use VR simulations if they were available.47 Computer-based simulations have been used in high-resource settings as an adjunct to in-person neonatal and paediatric resuscitation training.22 VR may support the transfer of knowledge to practice through interactive learning, problem-solving and standardised feedback.48 49 Connections that emerge between the participant’s offline and in-game identity, and the actual interactions with virtual newborns and mothers within the VR simulation, may modify attitudes and behaviours that relate to clinical practice.50–53 The HCWs in our study responded positively to mobile VR training.

    Recently, Erdsal et al54 described recommendations to improve the implementation of training programmes like HBB by establishing a system for training HCWs and conducting low-dose high frequency practice that is tailored to needs, incentivised and self-reflective. This practice should emphasise both cognitive and psychomotor skills important for successful neonatal resuscitation. For practising healthcare providers, the preparation of delivery and initial steps of resuscitation covered by OSCE A checklist are frequently performed in clinical practice, as approximately 10% of all babies born require some resuscitation.22 A number of digital innovations have been developed, over the past decade, to support HBB education and training.26 55 56 Based on our findings, after initial training, basic neonatal resuscitation skills may be supported by a range of digital training including VR, video and even digital guide only. However, prolonged resuscitation (represented by the OSCE B scenario) occurs in only 1% of deliveries, so the performance of these skills is less common in clinical practice, particularly in low-volume healthcare facilities. Simulation practice is important for skills retention in HCWs at these facilities and low-dose high frequency practice at a facility-based HBB corner is recommended.57 Pass rates on bag and mask skills were higher at 6 months in the VR group. Improvements in BMV performance over time were specifically seen in the critical skills needed to improve ventilation in a baby who is not responding. Resuscitation actions to improve ventilation like opening the mouth of the baby are notably hard to reinforce on manikin-based training because the manikin’s mouth is designed to be always open.58

    This study had several limitations. We only recruited participants from secondary and tertiary healthcare facilities urban and semi-urban resource-scarce settings. The study findings may not apply to HCWs who work in high-resource settings, at primary healthcare facilities or in rural settings. Although a majority of the study participants reported owning a mobile phone, to ensure uniform access to the study interventions, all participants were provided a study phone to enable access to the digital resources. VR applications are compatible with a wide range of low-cost mobile phones, but not all phones can run VR applications. Participants were asked to access the digital interventions weekly, but the average frequency of access to the application was monthly and may have impacted study findings. The optimal frequency of access is unknown and is an opportunity for future study.

    Conclusion

    Digital interventions supported the retention of neonatal resuscitation knowledge and skills for HCWs in Nigeria and Kenya. eHBB VR training was highly feasible and acceptable to HCWs in LMICs. eHBB VR may provide additional support for neonatal resuscitation skills retention when compared with other digital interventions.

    Data availability statement

    Data are available on reasonable request. Deidentified data are available on request from Dr. Rachel Umoren at nestprog@uw.edu.

    Ethics statements

    Patient consent for publication

    Ethics approval

    The study was approved by the University of Washington Institutional Review Board (IRB) approval number STUDY00005297, the Indiana University IRB approval number 1807371465, the Moi University, Health Research Ethics Committee approval number 0003109 and the University of Lagos College of Medicine Health Research Ethics Committee approval number CMUL/HREC/09/18/445.

    Acknowledgments

    We would like to acknowledge the senior administrative and nursing staff at healthcare facilities that participated in the study. We acknowledge the input of Dr Susan Niermeyer, Dr Michael Visik and Dr David Bolnick in developing the HBB 2nd edition video. We acknowledge the members of the University of Oxford LIFE Project who assisted in development of the eHBB VR application: Professor Michael English, Dr Niall Winters, Dr Hilary Edgcombe, Mr Jakob Rossner, Ms Naomi Muinga (KEMRI-Wellcome Trust, Kenya) and the staff at National Hospital Abuja who assisted with eHBB VR testing: Dr Amsa Mairami, Dr Fatima Mairami, Dr Adekunle Otuneye, Dr Lamidi Audu and Dr Mariya Mukhtar-Yola. Research assistants were provided with standardised Helping Babies Breathe 2nd Edition Facilitator training by Mr Sammy O. Barasa and Mr Geoffrey Mwai, of the Kenya Helping Babies Survive Master Trainer Corps. We would also like to acknowledge the study team members and collaborators who provided administrative, technical and other support for the study: Ms Oyin Akinrinola, Mr Dillon Afenir, Mr Bhavani Agnikula Kshatriya, Dr Benjamin Al-Haddad, Mr Iman Asangansi, Mr Prem Avanigadda, Mr Ariyo Ayokunle, Dr Brian Bresnahan, Mr Uchechukwu Chinedu, Mr Emeka Chukwu, Mr Chima Chukwudi, Ms Bailey Clopp, Mr Domnan Diretnan, Mr Aniekan Ebito, Ms Sakina Ginary, Ms Chioma Ginikanwa, Mr Oystein Gomo, Mr Alex McGee, Mr Manasseh Mmadu, Ms Ajayi Motunrayo, Mr Alvin Ogbonna, Mr Kevin Otieno, Ms Shruti Patel, Ms Amanda Stiffler, Dr Amy Thompson; in memorium, Dr Charles Spiekerman and Ms Elisabeth Meyers.

    References

    1. ↵
      1. UNICEF
      . Neonatal mortality: UNICEF, 2020. Available: https://data.unicef.org/topic/child-survival/neonatal-mortality/
    2. ↵
      1. UNCF
      . United Nations Inter-agency Group for Child Mortality estimation (UN IGME) levels & trends in child mortality: Report 2020, Estimates developed by the United Nations Inter-agency Group for Child Mortality Estimation. New York: UNCF, 2020.
    3. ↵
      1. Chou D,
      2. Daelmans B,
      3. Jolivet RR, et al
      . Ending preventable maternal and newborn mortality and stillbirths. BMJ 2015;351:h4255.doi:10.1136/bmj.h4255pmid:http://www.ncbi.nlm.nih.gov/pubmed/26371222
      OpenUrlFREE Full Text
    4. ↵
      1. Sharma G,
      2. Mathai M,
      3. Dickson KE, et al
      . Quality care during labour and birth: a multi-country analysis of health system bottlenecks and potential solutions. BMC Pregnancy Childbirth 2015;15 Suppl 2:S2.doi:10.1186/1471-2393-15-S2-S2pmid:http://www.ncbi.nlm.nih.gov/pubmed/26390886
      OpenUrlPubMed
    5. ↵
      1. Enweronu-Laryea C,
      2. Dickson KE,
      3. Moxon SG, et al
      . Basic newborn care and neonatal resuscitation: a multi-country analysis of health system bottlenecks and potential solutions. BMC Pregnancy Childbirth 2015;15 Suppl 2:S4.doi:10.1186/1471-2393-15-S2-S4pmid:http://www.ncbi.nlm.nih.gov/pubmed/26391000
      OpenUrlPubMed
    6. ↵
      1. Lawn JE,
      2. Blencowe H,
      3. Oza S, et al
      . Every newborn: progress, priorities, and potential beyond survival. Lancet 2014;384:189–205.doi:10.1016/S0140-6736(14)60496-7pmid:http://www.ncbi.nlm.nih.gov/pubmed/24853593
      OpenUrlCrossRefPubMedWeb of Science
    7. ↵
      1. WHO
      . Standards for improving the quality of care for small and sick newborns in health facilities 2020.
    8. ↵
      1. Hedstrom A,
      2. Perez K,
      3. Umoren R, et al
      . Recent progress in global newborn health: thinking beyond acute to strategic care? J Perinatol 2019;39:1031–41.doi:10.1038/s41372-019-0384-zpmid:http://www.ncbi.nlm.nih.gov/pubmed/31182774
      OpenUrlPubMed
    9. ↵
      1. Hoban R,
      2. Bucher S,
      3. Neuman I, et al
      . 'Helping babies breathe' training in sub-Saharan Africa: educational impact and learner impressions. J Trop Pediatr 2013;59:180–6.doi:10.1093/tropej/fms077pmid:http://www.ncbi.nlm.nih.gov/pubmed/23335632
      OpenUrlCrossRefPubMedWeb of Science
    10. ↵
      1. Kc A,
      2. Wrammert J,
      3. Clark RB, et al
      . Reducing perinatal mortality in Nepal using helping babies breathe. Pediatrics 2016;137.doi:10.1542/peds.2015-0117pmid:http://www.ncbi.nlm.nih.gov/pubmed/27225317
      OpenUrlPubMed
    11. ↵
      1. Singhal N,
      2. Lockyer J,
      3. Fidler H, et al
      . Helping babies breathe: global neonatal resuscitation program development and formative educational evaluation. Resuscitation 2012;83:90–6.doi:10.1016/j.resuscitation.2011.07.010pmid:http://www.ncbi.nlm.nih.gov/pubmed/21763669
      OpenUrlCrossRefPubMedWeb of Science
    12. ↵
      1. Msemo G,
      2. Massawe A,
      3. Mmbando D, et al
      . Newborn mortality and fresh stillbirth rates in Tanzania after helping babies breathe training. Pediatrics 2013;131:e353–60.doi:10.1542/peds.2012-1795pmid:http://www.ncbi.nlm.nih.gov/pubmed/23339223
      OpenUrlCrossRefPubMedWeb of Science
    13. ↵
      1. Versantvoort JMD,
      2. Kleinhout MY,
      3. Ockhuijsen HDL, et al
      . Helping babies breathe and its effects on intrapartum-related stillbirths and neonatal mortality in low-resource settings: a systematic review. Arch Dis Child 2020;105:127–33.doi:10.1136/archdischild-2018-316319pmid:http://www.ncbi.nlm.nih.gov/pubmed/31278145
      OpenUrlAbstract/FREE Full Text
    14. ↵
      1. Eblovi D,
      2. Kelly P,
      3. Afua G, et al
      . Retention and use of newborn resuscitation skills following a series of helping babies breathe trainings for midwives in rural Ghana. Glob Health Action 2017;10:1387985.doi:10.1080/16549716.2017.1387985pmid:http://www.ncbi.nlm.nih.gov/pubmed/29058568
      OpenUrlCrossRefPubMed
    15. ↵
      1. Musafili A,
      2. Essén B,
      3. Baribwira C, et al
      . Evaluating helping babies breathe: training for healthcare workers at hospitals in Rwanda. Acta Paediatr 2013;102:e34–8.doi:10.1111/apa.12034pmid:http://www.ncbi.nlm.nih.gov/pubmed/23113836
      OpenUrlCrossRefPubMed
    16. ↵
      1. Arlington L,
      2. Kairuki AK,
      3. Isangula KG, et al
      . Implementation of "helping babies breathe": a 3-year experience in Tanzania. Pediatrics 2017;139. doi:doi:10.1542/peds.2016-2132. [Epub ahead of print: 17 04 2017].pmid:http://www.ncbi.nlm.nih.gov/pubmed/28557724
      OpenUrlPubMed
    17. ↵
      1. Goudar SS,
      2. Somannavar MS,
      3. Clark R, et al
      . Stillbirth and newborn mortality in India after helping babies breathe training. Pediatrics 2013;131:e344–52.doi:10.1542/peds.2012-2112pmid:http://www.ncbi.nlm.nih.gov/pubmed/23339215
      OpenUrlCrossRefPubMed
    18. ↵
      1. Bookman L,
      2. Engmann C,
      3. Srofenyoh E, et al
      . Educational impact of a hospital-based neonatal resuscitation program in Ghana. Resuscitation 2010;81:1180–2.doi:10.1016/j.resuscitation.2010.04.034pmid:http://www.ncbi.nlm.nih.gov/pubmed/20599314
      OpenUrlPubMed
    19. ↵
      1. Reisman J,
      2. Arlington L,
      3. Jensen L, et al
      . Newborn resuscitation training in resource-limited settings: a systematic literature review. Pediatrics 2016;138:e20154490.doi:10.1542/peds.2015-4490pmid:http://www.ncbi.nlm.nih.gov/pubmed/27388500
      OpenUrlCrossRefPubMed
    20. ↵
      1. Bang A,
      2. Patel A,
      3. Bellad R, et al
      . Helping babies breathe (HBB) training: what happens to knowledge and skills over time? BMC Pregnancy Childbirth 2016;16:364.doi:10.1186/s12884-016-1141-3pmid:http://www.ncbi.nlm.nih.gov/pubmed/27875999
      OpenUrlCrossRefPubMed
    21. ↵
      1. Atukunda IT,
      2. Conecker GA
      . Effect of a low-dose, high-frequency training approach on stillbirths and early neonatal deaths: a before-and-after study in 12 districts of Uganda. The Lancet Global Health 2017;5:S12.doi:10.1016/S2214-109X(17)30119-5
      OpenUrl
    22. ↵
      1. Sawyer T,
      2. Umoren RA,
      3. Gray MM
      . Neonatal resuscitation: advances in training and practice. Adv Med Educ Pract 2017;8:11–19.doi:10.2147/AMEP.S109099pmid:http://www.ncbi.nlm.nih.gov/pubmed/28096704
      OpenUrlPubMed
    23. ↵
      1. Weinberg S,
      2. Jones D,
      3. Worku B, et al
      . Helping babies survive training programs: evaluating a teaching cascade in Ethiopia. Ethiop J Health Sci 2019;29:669–73.doi:10.4314/ejhs.v29i6.3pmid:http://www.ncbi.nlm.nih.gov/pubmed/31741637
      OpenUrlPubMed
    24. ↵
      1. Umoren R,
      2. Stadler DJ,
      3. Gasior SL, et al
      . Global collaboration and team-building through 3D virtual environments. Innovations in Global Medical and Health Education 2014;2014.doi:10.5339/igmhe.2014.1
    25. ↵
      1. SITZMANN T,
      2. Sitzmann T
      . A meta-analytic examination of the instructional effectiveness of computer-based simulation games. Pers Psychol 2011;64:489–528.doi:10.1111/j.1744-6570.2011.01190.x
      OpenUrl
    26. ↵
      1. Bucher SL,
      2. Cardellichio P,
      3. Muinga N, et al
      . Digital health innovations, tools, and resources to support helping babies survive programs. Pediatrics 2020;146:S165–82.doi:10.1542/peds.2020-016915Ipmid:http://www.ncbi.nlm.nih.gov/pubmed/33004639
      OpenUrlCrossRefPubMed
    27. ↵
      1. Kamath-Rayne BD,
      2. Thukral A,
      3. Visick MK, et al
      . Helping babies breathe, second edition: a model for strengthening educational programs to increase global newborn survival. Glob Health Sci Pract 2018;6:538–51.doi:10.9745/GHSP-D-18-00147pmid:http://www.ncbi.nlm.nih.gov/pubmed/30287531
      OpenUrlAbstract/FREE Full Text
    28. ↵
      1. Umoren R
      . eHBB VR and eHBB mobile applications on Google play store 2020.
    29. ↵
      1. Umoren R
      . Helping babies breathe 2nd edition neonatal resuscitation video, 2019. Available: https://nestprogram.wistia.com/medias/89c4riojae
    30. ↵
      1. Pediatrics AAo
      . Helping babies survive: American Academy of pediatrics, 2021. Available: https://www.aap.org/en-us/advocacy-and-policy/aap-health-initiatives/helping-babies-survive/Pages/Helping-Babies-Breathe-Edition.aspx
    31. ↵
      1. Patterson JK,
      2. Girnary S,
      3. North K, et al
      . Innovations in cardiorespiratory monitoring to improve resuscitation with helping babies breathe. Pediatrics 2020;146:S155–64.doi:10.1542/peds.2020-016915Hpmid:http://www.ncbi.nlm.nih.gov/pubmed/33004638
      OpenUrlCrossRefPubMed
    32. ↵
      1. Cordova E,
      2. Al-Rousan T,
      3. Castillo-Angeles M, et al
      . Effect of low-cost interventions on the retention of knowledge and skills following helping babies breathe training. Int J Gynaecol Obstet 2018;142:248–54.doi:10.1002/ijgo.12512pmid:http://www.ncbi.nlm.nih.gov/pubmed/29687893
      OpenUrlPubMed
    33. ↵
      1. Tabangin ME,
      2. Josyula S,
      3. Taylor KK, et al
      . Resuscitation skills after helping babies breathe training: a comparison of varying practice frequency and impact on retention of skills in different types of providers. Int Health 2018;10:163–71.doi:10.1093/inthealth/ihy017pmid:http://www.ncbi.nlm.nih.gov/pubmed/29618017
      OpenUrlCrossRefPubMed
    34. ↵
      1. Tsai P-F,
      2. Kitch S,
      3. Beck C, et al
      . Using an interactive video simulator to improve certified nursing assistants' dressing assistance and nursing home residents' dressing performance: a pilot study. Comput Inform Nurs 2018;36:183–92.doi:10.1097/CIN.0000000000000415pmid:http://www.ncbi.nlm.nih.gov/pubmed/29406394
      OpenUrlPubMed
    35. ↵
      1. Curtis HA,
      2. Trang K,
      3. Chason KW, et al
      . Video-Based learning vs traditional Lecture for instructing emergency medicine residents in disaster medicine principles of mass triage, decontamination, and personal protective equipment. Prehosp Disaster Med 2018;33:7–12.doi:10.1017/S1049023X1700718X
      OpenUrlPubMed
    36. ↵
      1. Coyne E,
      2. Frommolt V,
      3. Rands H, et al
      . Simulation videos presented in a blended learning platform to improve Australian nursing students’ knowledge of family assessment. Nurse Educ Today 2018;66:96–102.doi:10.1016/j.nedt.2018.04.012
      OpenUrlPubMed
    37. ↵
      1. Umoren RA,
      2. Gray MM,
      3. Schooley N, et al
      . Effect of video-based telemedicine on transport management of simulated newborns. Air Med J 2018;37:317–20.doi:10.1016/j.amj.2018.05.007
      OpenUrlPubMed
    38. ↵
      1. Sweigart LI,
      2. Umoren RA,
      3. Scott PJ, et al
      . Virtual TeamSTEPPS(®) Simulations Produce Teamwork Attitude Changes Among Health Professions Students. J Nurs Educ 2016;55:31–5.doi:10.3928/01484834-20151214-08pmid:http://www.ncbi.nlm.nih.gov/pubmed/26812380
      OpenUrlPubMed
    39. ↵
      1. Sheik-Ali S,
      2. Edgcombe H,
      3. Paton C
      . Next-Generation virtual and augmented reality in surgical education: a narrative review. Surg Technol Int 2019;33.
    40. ↵
      1. Muinga N,
      2. Paton C
      . Virtual reality for medical and nursing training in low-and middle-income countries 2019.
    41. ↵
      1. Komasawa N,
      2. Atagi K,
      3. Minami T
      . Combination of video and manikin hybrid simulation for rapid response emergency airway management training in medical students. J Clin Anesth 2018;46:75–6.doi:10.1016/j.jclinane.2018.02.003pmid:http://www.ncbi.nlm.nih.gov/pubmed/29414625
      OpenUrlPubMed
    42. ↵
      1. Umoren R,
      2. Rybas N
      . Who am I as a healthcare provider? integrating an awareness of selfhood and society into virtual learning. Pennsylvania: IGI Global, 2017: 166.
    43. ↵
      1. Gray MM,
      2. Thomas AA,
      3. Burns B
      . Evacuation of vulnerable and critical patients: multimodal simulation for nurse-led patient evacuation. Simul Healthc 2020 (published Online First: 2020/05/07).
    44. ↵
      1. Gray MM,
      2. Umoren RA,
      3. Josephsen J, et al
      . Gaps in NRP provider performance on standardized simulations: a report from the NRP eSim™ study. Pediatric academic societies conference; 2019; Baltimore, MD 2019.
    45. ↵
      1. Jones-Bamman C,
      2. Niermeyer S,
      3. McConnell K, et al
      . Teaching helping babies breathe via telehealth: a new application in rural Guatemala. Biomed Hub 2019;4:1–6.doi:10.1159/000502934pmid:http://www.ncbi.nlm.nih.gov/pubmed/31993430
      OpenUrlCrossRefPubMed
    46. ↵
      1. Vishwanath A,
      2. Kam M,
      3. Kumar N
      . Examining low-cost virtual reality for learning in low-resource environments. Proceedings of the 2017 Conference on Designing Interactive Systems; 2017. ACM 2017:1277–81.
    47. ↵
      1. Umoren R,
      2. Ezeaka VC,
      3. Fajolu IB, et al
      . Perspectives on simulation-based training from paediatric healthcare providers in Nigeria: a national survey. BMJ Open 2020;10:e034029.doi:10.1136/bmjopen-2019-034029
    48. ↵
      1. Sweigart LI,
      2. Umoren RA,
      3. Scott PJ, et al
      . Virtual TeamSTEPPS ®Simulations Produce Teamwork Attitude Changes Among Health Professions Students. J Nurs Educ 2016;55:31–5.doi:10.3928/01484834-20151214-08
      OpenUrlPubMed
    49. ↵
      1. Joynes C
      . Distance learning for health: what works-A global review of accredited post-qualification training programmes for health workers in low and middle income countries. London Int Dev Cent 2011;1.
    50. ↵
      1. Waggoner Z
      . My avatar, my self: identity in video role-playing games. Jefferson: McFarland, 2009.
    51. ↵
      1. Parsons TD,
      2. Rizzo AA
      . Affective outcomes of virtual reality exposure therapy for anxiety and specific phobias: a meta-analysis. J Behav Ther Exp Psychiatry 2008;39:250–61.doi:10.1016/j.jbtep.2007.07.007
      OpenUrlCrossRefPubMed
    52. ↵
      1. Seymour NE
      . Vr to or: a review of the evidence that virtual reality simulation improves operating room performance. World J Surg 2008;32:182–8.doi:10.1007/s00268-007-9307-9
      OpenUrlCrossRefPubMedWeb of Science
    53. ↵
      1. Robb A,
      2. White C,
      3. Cordar A, et al
      . A comparison of speaking up behavior during conflict with real and virtual humans. Comput Human Behav 2015;52:12–21.doi:10.1016/j.chb.2015.05.043
      OpenUrl
    54. ↵
      1. Ersdal HL,
      2. Singhal N,
      3. Msemo G, et al
      . Successful implementation of helping babies survive and helping mothers survive programs—An Utstein formula for newborn and maternal survival. PLoS One 2017;12:e0178073.doi:10.1371/journal.pone.0178073
    55. ↵
      1. Merali HS,
      2. Chan NH-M,
      3. Mistry N, et al
      . Designing and evaluating a novel mobile application for helping babies breathe skills retention in Uganda: comparative study protocol. BMJ Paediatr Open 2019;3:e000561.doi:10.1136/bmjpo-2019-000561
    56. ↵
      1. Somannavar MS,
      2. Goudar SS,
      3. Revankar AP, et al
      . Evaluating time between birth to Cry or bag and mask ventilation using mobile delivery room timers in India: the NICHD global network's helping babies breathe trial. BMC Pediatr 2015;15:93.doi:10.1186/s12887-015-0408-6pmid:http://www.ncbi.nlm.nih.gov/pubmed/26245688
      OpenUrlPubMed
    57. ↵
      1. Lopreiato JO,
      2. Sawyer T
      . Simulation-Based medical education in pediatrics. Acad Pediatr 2015;15:134–42.doi:10.1016/j.acap.2014.10.010pmid:http://www.ncbi.nlm.nih.gov/pubmed/25748973
      OpenUrlPubMed
    58. ↵
      1. Sawyer T,
      2. Strandjord TP,
      3. Johnson K, et al
      . Neonatal airway simulators, how good are they? A comparative study of physical and functional fidelity. J Perinatol 2016;36:151–6.doi:10.1038/jp.2015.161pmid:http://www.ncbi.nlm.nih.gov/pubmed/26583944
      OpenUrlPubMed

    Supplementary materials

    • Supplementary Data

      This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

    Footnotes

    • RU and SB are joint first authors.

    • CVE and FE are joint senior authors.

    • Twitter @nestsim1

    • RU and SB contributed equally.

    • CVE and FE contributed equally.

    • Contributors RU and SB conceptualised and designed the study, drafted the initial manuscript, designed the data collection tools and reviewed and revised the manuscript. CP and SP designed the data collection tools and reviewed and revised the manuscript. BNE, IBF, FMO, IA, MN, OAO, COA and AM collected data, and reviewed and revised the manuscript. JF and DSH carried out the initial analyses and reviewed and revised the manuscript. CVE and FE conceptualised and designed the study, collected data and critically reviewed the manuscript for important intellectual content. All authors participated in the revision of the manuscript, approved the final version and agreed to be accountable for all aspects of the work.

    • Funding This work was supported by the Bill & Melinda Gates Foundation (grant number OPP1169873). Under the grant conditions of the Foundation, a Creative Commons Attribution 4.0 Generic License has already been assigned to the Author Accepted Manuscript version that might arise from this submission.

    • Competing interests RU and CP developed the eHBB VR application. SB and SP developed the mHBS/DHIS2 application. The other coauthors have no conflicts of interest relevant to this article to disclose.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.