Article Text
PDF
Abstract
Objectives The aim of this study was to evaluate the implementation of the Practical Obstetric Multi-Professional Training (PROMPT) simulation using the Kirkpatrick’s framework. We explored participants’ acquisition of knowledge and skills, its impact on clinical outcomes and organisational change to integrate the PROMPT programme as a credentialing tool. We also aimed to assess participants’ perception of usefulness of PROMPT in their clinical practice.
Study design Mixed methods approach with a pre-test/post-test design.
Setting Healthcare network providing obstetric care in Victoria, Australia.
Participants Medical and midwifery staff attending PROMPT between 2013 and 2015 (n=508); clinical outcomes were evaluated in two cohorts: 2011–2012 (n=15 361 births) and 2014–2015 (n=12 388 births).
Intervention Attendance of the PROMPT programme, a simulation programme taught in multidisciplinary teams to facilitate teaching emergency obstetric skills.
Main outcome measure Clinical outcomes compared before and after embedding PROMPT in educational practice.
Secondary outcome measure Assessment of knowledge gained by participants through a qualitative analysis and description of process of embedding PROMPT in educational practice.
Results There was a change in the management of postpartum haemorrhage by early recognition and intervention. The key learning themes described by participants were being prepared with a prior understanding of procedures and equipment, communication, leadership and learning in a safe, supportive environment. Participants reported a positive learning experience and increase in confidence in managing emergency obstetric situations through the PROMPT programme, which was perceived as a realistic demonstration of the emergencies.
Conclusion Participants reported an improvement of both clinical and non-technical skills highlighting principles of teamwork, communication, leadership and prioritisation in an emergency situation. An improvement was observed in management of postpartum haemorrhage, but no significant change was noted in clinical outcomes over a 2-year period after PROMPT. However, the skills acquired by medical and midwifery staff justify embedding PROMPT in educational programmes.
- simulation
- interprofessional learning
- interprofessional education
- evaluation
- patient outcome
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Strengths and limitations of this study
This is one of the few mixed methods studies on evaluation of training programmes using multiple levels of Kirkpatrick’s assessment capturing participant reaction, knowledge acquisition, organisational change and patient outcome.
This is an outcome-based evaluation using the high levels of the Kirkpatrick’s framework (evaluating impact on the health service and patient outcome) providing evidence of training effectiveness.
The participants’ behaviour (under direct observation or by using videos) could not be studied in either simulation or a clinical setting.
Introduction
Interprofessional team-based, simulated training programmes are becoming increasingly popular to improve the performance of clinical workforce in emergency responses and its resultant clinical outcomes. The provision of high-quality birth suite care is no exception. Staff training in technical clinical skills is put to test in complex obstetric situations that require time critical management. Team members must be instantly engaged to achieve synergism in managing acute obstetric emergencies. Hence, acquisition of non-technical skills (NTS), such as effective communication and teamwork, are as important as mastering ‘hands-on’ clinical skills.1
Interprofessional education (IPE) focuses on ‘staff members working together to learn with, from and about each other’.2 IPE programmes help individuals to develop an understanding of other professional roles within the multidisciplinary team. Such an understanding is thought important for safe and effective clinical practice as a team.3 In order to maintain a level of confidence in managing these difficult clinical emergencies, regular up-skilling sessions are necessary.
PRactical Obstetric Multi-Professional Training (PROMPT) is a multiprofessional training package designed to expose participants (obstetricians, midwives, paediatricians and anaesthetists) to obstetric emergencies in a real-time environment.4–8 This simulation-based programme aims to recreate clinical problems either ‘in-situ’ in a birth unit or in a simulation centre and presents them to participants as realistically as possible. The scenarios can be designed specifically for the level of the participants and the available facilities. Evaluation of these programmes is necessary to assess if their objectives are met. Programmes can be evaluated using various frameworks, one of them being the six-level modification of Kirkpatrick’s framework.2 9 10 The various levels assess participant’s satisfaction, change in attitude or identification of what was learnt, if these skills changed participant behaviour in a clinical setting or ultimately affected clinical organisational change and patient outcomes (see table 1).
Modified Kirkpatrick’s framework (adapted from Barr’s six-level classification)
We introduced the Victorian state version of the PROMPT programme to our maternity service in 2013.11 In this study, we aimed to evaluate the impact of PROMPT in our health service by assessing the various levels in the six-level framework. Specifically, we wished to identify the ‘key learning points’ acquired, ‘how’ useful this workshop style teaching was rated and whether there was any evidence of change in patient outcomes. We also describe the process of ‘embedding’ this programme in educational up-skilling of staff.
Methods
Study design
The study follows a mixed methods design with quantitative analysis of patient outcome data and for participant rating of the intervention. The data regarding the key learning messages was extracted using qualitative content analysis identifying key themes.
Setting and participants
Monash Health maternity service provides birthing care for over 9000 women annually at three separate hospital sites, each with different levels of acuity, all within metropolitan Melbourne, Victoria. The three sites share common clinical practice guidelines, policies and procedures. Monash Health implemented the PROMPT programme in its current format across all its sites since 2013. Midwifery educators and dedicated senior obstetric medical staff run the programme. The PROMPT sessions are conducted 10 times per year at each site at monthly intervals. Midwifery and medical staff (junior and senior) are required to participate at least every 2 years. All medical and midwifery staff who had attended the PROMPT session at least once between 2013 and 2015 were invited to participate in the study.
PROMPT programme scenarios
The half-day programme consists of short, interactive lectures and scenario-based drills. Each drill is followed by a debrief covering clinical skills and NTS. The clinical scenarios include eclampsia, shoulder dystocia, neonatal resuscitation and postpartum haemorrhage (PPH). These are topics that were already covered in the prereading material provided to the participants.
Questionnaires
The evaluation of the PROMPT workshop followed a pretest and a post-test research design using paper-based questionnaires. The questionnaires were drafted, revised and agreed on by the PROMPT committee (represented by both medical and midwifery educators) to establish content validity. The questionnaires had been pilot tested in a home birth-based simulation programme (in a different participant group that included home birth midwives) at Monash Health, and results were published in a peer-reviewed journal.12 Each questionnaire had 26 items where participants’ responses are recorded using a 5-point Likert scale. The pretest evaluated levels of knowledge and confidence managing the obstetric emergencies covered. They are also asked about participants’ professional background and experience in these clinical emergencies. At the end of the workshop, the post-test evaluated the satisfaction and learning acquired from the programme. Participants were also asked to reflect on the essential learning points attained that were thought to be transferable to their (individual or team based) clinical practice using free text.
Textual data were analysed independently and inductively using content analysis undertaken by two researchers independently (AK and SamS) to produce key themes.13 The results were discussed, and after establishing consensus, all data were recoded. Some categories overlapped. but items were counted only once. Discrepancies were negotiated enabling final attribution of text within categories.
Clinical outcome measures
A retrospective cohort study examined all documented cases of the three major obstetric emergencies covered during the drills (eclampsia, shoulder dystocia combined with neonatal resuscitation and PPH). Clinical outcomes were evaluated in two cohorts: before the implementation of PROMPT in 2011–2012 (n=15 361 births) and after the implementation of PROMPT in 2014–2015 (n=12 388 births). Patient outcomes were evaluated using the following measures. For shoulder dystocia, we measured the use of external and internal manoeuvres, time between delivery of the head and the body, completion of documentation, major maternal perineal trauma (third and fourth degree tears) and neonatal outcomes including brachial plexus injury, clavicle or humerus fracture, Apgar score <7 at 5 min, umbilical cord lactates >8 mg/dL, admission to newborn services and perinatal death. For PPH, we classified women into two groups according to the estimated volume of blood loss (1000–1499 mL or ≥1500 mL) reporting rates of blood transfusion, transfer to the operating theatre, intravenous fluid resuscitation and use of a (Bakri) balloon tamponade.
Data were extracted from an electronic database of birthing outcomes, the Birthing Outcomes System (BOS), which records outcomes for all births ≥20 weeks of gestation and is routinely entered by midwifery staff.14 Where necessary, BOS data were supplemented by individual case record review.
Statistical analysis
Data were analysed with Prism for Mac V.7.0a (GraphPad software, San Diego, California, USA). Continuous data were expressed as medians and IQR because of skewed distributions. To compare the two cohorts, we used a Mann-Whitney U test for quantitative data and a Fisher’s exact test for contingency testing. A P value <0.05 was considered statistically significant.
Results
Participation
Since 2013, we have run 70 PROMPT sessions across our three sites with a total of 508 participants. Approximately one-third (n=178, 35%) of participants were medical staff (junior and senior). The remaining were midwifery staff (n=287, 56%), medical or midwifery students (n=34, 7%) or special care nursery staff (n=9, 2%). By 2015, 76% midwives and 90% senior medical staff had participated at least once in PROMPT.
Satisfaction with the simulation activity (level 1 Kirkpatrick’s framework)
Figure 1 summarises the participant knowledge, confidence and prior experience in managing obstetric emergencies. Staff confidence in management of eclampsia was lower than that for the other obstetric emergencies. The confidence and knowledge concerning neonatal resuscitation was higher for midwives than the medical staff (figure 1). In general, the workshops were rated highly by both medical and midwifery participants (median Likert score of 5 (maximum) for both groups) in regards to clinical usefulness of material covered and debriefing experience.
Bar diagrams showing level of knowledge (Q1), confidence (Q2) or prior experience (Q3) of medical staff (top panel) and midwifery staff (bottom panel) in dealing with eclampsia, shoulder dystocia, neonatal resuscitation (NLS) and postpartum haemorrhage (PPH). Numbers 1–5 on the y-axis denotes Likert scale rating, where 5 is the highest rating.
Knowledge acquired from the workshop (level 2b Kirkpatrick’s framework)
Four hundred and thirty comments made by 237 participants were available for content analysis (table 2). The key themes related to improved communication between staff members (n=87), developing knowledge of equipment and procedures (78 responses), learning leadership and followership (73 responses), being in a supportive learning environment (63 responses), the realism of the simulation (48 responses), understanding the roles of staff from another profession (46 responses) and prioritisation of tasks (33 responses).
Learning acquired from the PROMPT programme
Communication
Clear communication established directly with the members of the team (by using the individual’s name) and with others who assist in the process, for example, with switchboard calling an emergency code. Where appropriate, specific terminology should be used. The loop of communication should be closed by obtaining a response from the recipient to ensure accountability of the individual undertaking the task. The communication was seen to be even more crucial at certain times like the handover of a task to another member of the team.
Situational awareness
Identification and knowledge of equipment, its location in the birth unit, organisation of the equipment and knowing if it was in working order was recognised as relevant for the staff using it in an emergency with time constraints. A prior familiarity of content and a practice of using the postpartum and eclampsia kit were found to be essential. Necessary gear found missing at the time of workshop or kept in the wrong location, delayed the management and caused stress to the team.
Similarly staff members were keen to have a prior awareness of protocols and procedures, more so in complex situations like eclampsia and neonatal resuscitation and where clinical manoeuvres were needed like shoulder dystocia. The organisational pathways needed like calling an emergency code and methods to access operating theatres in an emergency were also emphasised.
Learning leadership and followership
The importance of leading an emergency team presented as an unexpectedly prominent theme. The key characteristics of a leader were to maintain a ‘helicopter view’ at all times and be clear and assertive with instructions to participants. it was considered important to establish and announce who the leader of the team was (either by the leader or another participant) and appoint one if already not designated. The leader could change during the emergency scenario depending on individual capability and who was available. Handover from one leader to another needed clear communication. In such situations, the new leader should initially ‘step back’ and assimilate the situation prior to taking over.
The rest of the team should patiently wait for instructions, offer to help (based on their individual scope of practice) and contribute to the team management by playing their designated role. If the instructions were unclear or participants were unable to perform the allocated task, they should speak up and close the communication loop.
Supportive learning environment
The PROMPT workshop was acknowledged to improve participants’ confidence and learning of clinical knowledge and skills through individual opportunity to practice and team feedback. In a simulated setting, the technical skills and procedures could be ‘unpacked’ into small steps during the feedback session. The combination of learning emergency skills in a simulated environment was seen as a step towards improving women safety.
Realism in simulation
The participants perceived the workshop to be similar to a real emergency through the role of an actor, scenario design, experiencing stress, working within timelines and location in a birth unit. The scenarios were based on rare emergencies and followed an unpredictable course resulting in participants feeling anxious and voiced the need to ‘stay calm’.
Role of interprofessional staff
Participants displayed a preference to revert to their natural/usual clinical roles when managing a clinical emergency as this was based on their strengths and scope of practice. The participants wanted to have a clear, well-defined role allocation that was ‘task specific’. Both medical and midwifery staff members were keen to share learning in the interprofessional setting and wanted to understand roles of the other discipline in the team. Both teams referred to learning teamwork and task sharing.
Prioritisation
Participants demonstrated a need to organise the tasks systematically and to get help early. They emphasised on timely escalation of tasks due to their awareness of their limitations in managing emergency situations and their scope of practice.
Organisational change to ‘embed’ the PROMPT programme (level 4a Kirkpatrick’s framework)
We describe the process of embedding the PROMPT programme using Kurt Lewin’s three-phase model.15
Step 1: unfreeze
The key issue of poor communication (occasionally leading to conflict) among medical and midwifery staff was recognised through incident reporting as a component of a risk management process. This was seen to delay mobilising resources required in an emergency setting, hence compromising optimum patient safety. In a time critical situation, where effective teamwork is the key, a need to create change was recognised by medical and midwifery leaders at the institution. The need to change was communicated both to the healthcare network executive group and to the clinical staff delivering patient care. This coincided with introduction of the PROMPT programme in the state of Victoria resulting in strategic inclusion of the team-training programme for medical and midwifery staff.
Step 2: change
The change described here is embedding the PROMPT programme as a component of routine educational practice. The principles learnt through the programme focused on communication, leadership and situational awareness, similar to the vision shared by the institution. The benefits of attending the programme were communicated to the staff and feedback encouraged from participants. Problems that hindered attendance (like rostering issues and managing patient workload on teaching days) were dealt with promptly. Funding was obtained from the institution by reporting benefits of change in attitude of interprofessional staff and observed improvement in performance, although this was not formally evaluated. This funding further facilitated the operational management of the programme, as dedicated clinical staff members could be employed to sustain the quality of teaching.
Step 3: refreeze
The observed improvement in attitudes of the interprofessional staff and effort to meet higher standards of clinical practice was encouraged. Leadership and operational support required to run the programme was improved (by increasing the numbers of faculty members facilitating the programme), and ongoing training support was provided to them. A process of providing team-based feedback was developed (using the PROMPT guidelines), and the role of learning through PROMPT was formalised, which lent itself to its use as credentialing tool. A mandatory requirement of 2 yearly attendance was set up for all medical and midwifery staff.
Clinical outcomes (level 4b Kirkpatrick’s framework)
In 2011–2012, there were 15 361 births, and in 2014–2015, there were 12 388 births at Monash Health.
Eclampsia
Across the 4 years, four women had an eclamptic seizure, two in 2011–2012 (0.13/1000) and two in 2014–2015 (0.16/1000). All four women were managed as per local protocol.
Shoulder dystocia
Table 3 summarises the incidence and outcomes related to shoulder dystocia. The rate of shoulder dystocia in 2011–2012 (n=268; 1.7%) was significantly lower than in 2014–2015 (n=290; 2.3%, P=0.001). No neonatal deaths were recorded in either group. The interval between delivery of head and body was shorter in the recent cohort (2.0 min (IQR 1–2) vs 2.0 min (IQR 1–3), P=0.04). In the cohort after implementation of PROMPT, we observed lower incidences of brachial plexus injury, humerus or clavicle fractures, low Apgar scores and nursery admissions, although these differences were not statistically significant (table 3). There was a significant decrease in the completion of the required shoulder dystocia emergency ‘management sheet’ (24% vs 17%; P=0.04).
Shoulder dystocia
Postpartum haemorrhage
For women with a PPH of 1000–1499 mL, there was no significant change in the number of cases between cohorts (n=561 (3.7%) vs n=511 (4.1%)), and no significant differences were observed in maternal outcomes or management strategies. For women with a PPH of >1500 mL, a significant difference was seen in the number of patients transferred to theatre after vaginal birth (30% vs 38%; P=0.049) and the use of Bakri balloons (6% vs 12%; P=0.02), which were introduced in 2011 (tables 4 and 5).
Postpartum haemorrhage 1000–1499 mL
Postpartum haemorrhage >1500 mL
Discussion
Main findings
Through a formal evaluation of PROMPT and a review of clinical outcomes, we have observed that this multidisciplinary training has a positive effect on managing of obstetric emergencies within our service. Consistent with mandatory workforce training requirements, participation of both medical and midwifery staff was excellent across all of our sites such that PROMPT has become an embedded component of ongoing professional development. In this paper, we have evaluated our PROMPT programme using the various levels of Kirkpatrick’s framework, observing encouraging results. All levels examined showed positive effects after implementation of this structured training. In addition, the evaluation allowed us to identify areas for future improvement such as record keeping of therapeutic measures.
Participants found PROMPT an effective approach for the acquisition of new skills and knowledge. Medical and midwifery staff members reported an increase in confidence and had high satisfaction scores on learning as a team (level 1 Kirkpatrick’s framework).
Our next level of assessment focused on key ‘take-home’ learnings acquired by the participants (level 2 Kirkpatrick’s framework). Communication and situational awareness were considered important NTS learnings by the majority of participants and is a finding consistent with other studies.5 6 The themes on ‘leadership’ and ‘following the leader’ are thought critical for safe team-based management, both in simulated and real emergencies.6 16 Poor performance in leadership despite good communication can also occur, hence, making leadership an independent learning goal of the workshop.17 Developing improved ‘situational awareness’ with knowledge of equipment use and efficient use of team members is an often-reported learning outcome of the PROMPT programme.5
Our final analysis reviewed the birthing outcome and safety data (Kirkpatrick’s level 4b). In the recent cohort, we observed a significantly increased incidence of shoulder dystocia. This could be related to an increased awareness of this condition but could also reflect the increasing numbers of obese pregnant women delivering at our centres. We observed a small but statistically significant difference in the interval between the delivery of head and body, the clinical relevance of which is debatable. These could be assessed individually using case reviews and learning gaps addressed through clinical case review meetings. The significant increase in the number of patients transferred to theatre for control of massive postpartum bleeding (PPH >1500 mL), and the increase in the use of (Bakri) balloon tamponade may reflect a greater awareness of the benefits of early and aggressive control of excessive bleeding following our PROMPT implementation. This was also noted in a recent randomised control trial where the units that participated in simulation based team training had a higher incidence of blood transfusion and surgical treatment of PPH.8
Strengths and limitations
The current study is one of few mixed methods studies attempting to draw a link between perceived learning, clinical practice and outcomes by using various levels of Kirkpatrick’s framework. As far as we are aware, only a few studies have evaluated simulation-based intervention through multiple ‘lenses’ of assessment, as reported in a recent review on obstetric emergencies.18 Most researchers have limited evaluations to either level 1 or 2 with some studies demonstrating a change in team behaviour and retention of skills.19 Studies looking at clinical outcome are scant.8 11 18 20 Our evaluation includes participant satisfaction with the scenario and debrief (level 1), and learning skills and knowledge acquired by the two major interprofessional groups (level 2). We demonstrate the process from introduction of the intervention and its ‘embedding’ in curricular training and ‘credentialing’ (level 4a). The PROMPT programme has been successfully integrated with teaching programmes globally; however, the description of the programme with change management principles is worthy of sharing. Above all, we have also compared the birthing outcome before and after the intervention was introduced into practice (Level 4b).
However, due to challenges related to study design that entails direct observation of participants in a ‘natural’ setting, we were unable to assess a change in observed clinical behaviour/teamwork that may have helped to directly connect workshop learning with clinical practice, which may be done using clinical checklists.21
Patient care and clinical outcomes are rarely reported as evidence of effectiveness of educational programmes.22 Most likely this is because programmes need to be embedded prior to evaluation and coverage of a sufficient proportion of the workforce needs to be achieved before improved care and outcomes would be expected. This can take many years.23–25 An evaluation of the PROMPT programme elsewhere demonstrated a significant decrease in brachial plexus injury, incidence of pH less than 7 and a reduction of hypoxic ischaemic encephalopathy by 50% when assessed over a 7-year interval.20 We were unable to explore detailed outcome data prior to 2010 as previously documented notes had missing clinically relevant details, hence precluding us from assessing 5 years before and after PROMPT that may have provided a better reflection of birthing outcomes. However, this may not have changed the result as a similar study failed to show a significant reduction in the composite obstetric outcome in units where multiprofessional simulation training was introduced.8
A major strength of this evaluation is that it allowed insights into service delivery and identification of potential deficiencies. For instance, we observed a reduction in the completion of shoulder dystocia management forms. In addition, our current record form lack certain outcome measures that would be of interest to evaluate clinical management, such as fluid volume usage during PPH.
Interpretation
Participants indicated that communication, situational awareness and leadership skills are key factors for managing emergencies as a successful team. The next level of evaluation planned will be to check the team performance in a real obstetric emergency setting to determine if the transfer of learning has occurred. This can be achieved by integration of level 3 assessment (behaviour) into our training development strategy by direct observation of performance in a simulated and/or clinical setting. Apart from more proactive management noted in postpartum haemorrhage, no significant difference was noted in clinical outcome. This may be due to existence of previously run simulation programmes, which focused on individual skills but not on effective teamwork. Although participants recognise the importance of teamwork and communication in their learning, this was not transferable to a change in clinical outcome.
This evaluation has already resulted in changing the organisational practice at our institution (Level 4a).26 An annual attendance of PROMPT is encouraged for all staff, and a 2 yearly attendance is a mandatory requirement for staff working on the birth unit. It is used for credentialing the staff members with remediation plans for participants unable to meet the expected standards of performance for both technical skills and NTS. Our goal will be to continue to strengthen this process and to formalise it further, linking it with professional development.
Conclusions
The study highlights the need for teaching teamwork, communication and leadership skills in managing obstetric emergencies through a high-fidelity simulation programme. The impact on clinical outcomes seems limited, yet we identified some differences related to management of shoulder dystocia and postpartum haemorrhage that could have made a difference in certain individual cases. Improved participant confidence with up-skilling of both procedural skills and NTS has a potential to change clinical practice and outcomes, hence, validating the incorporation of these IPE simulation strategies in clinical care.
Acknowledgments
We would like to thank the birth unit staff across sites at Monash Health.
References
Footnotes
Contributors AK conceived and designed the study, analysed the data, wrote the first draft of the manuscript and finalised the submission. SamS performed the study, analysed the data, edited the manuscript and approved the final submission. DL, JM, PN, MS and SalS performed the study and approved the final submission. EMW conceived and designed the study, contributed to analysis tools and edited and finalised the submission. DN analysed the data and contributed to analysis tools. PD is the senior author, conceived, designed, performed and supervised the study, analysed data and approved the final submission.
Funding The PROMPT programme at Monash Health was supported by funding of the Victorian Managed Insurance Authority (VMIA) when it was introduced.
Disclaimer Victorian Managed Insurance Authority had no role in study design, data collection and analysis or manuscript preparation.
Competing interests The authors MS, PN, DL, SalS and JM are involved in the delivery of PROMPT at their hospital but have no financial involvement.
Patient consent Not required.
Ethics approval The study was approved by Monash Human Research Ethics Committee as a quality assurance project.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement There are no additional unpublished data from the study.