Article Text

Original research
Simulation-based training for increasing health service board members’ effectiveness: a cluster randomised controlled trial
  1. Nicholas Faulkner1,
  2. Breanna Wright1,
  3. Alyse Lennox1,
  4. Marie Bismark2,
  5. Jane Boag3,
  6. Sophie Boffa4,
  7. Bruce Waxman5,
  8. Janet Watson-Kruse6,
  9. Geoff Paine1,
  10. Peter Bragge1
  1. 1BehaviourWorks Australia, Sustainable Development Institute, Monash University, Clayton, Victoria, Australia
  2. 2Melbourne School of Population and Global Health, University of Melbourne, Melbourne, Victoria, Australia
  3. 3Victorian Managed Insurance Authority, Melbourne, Victoria, Australia
  4. 4Safer Care Victoria, Victoria Department of Health and Human Services, Melbourne, Victoria, Australia
  5. 5Wonthaggi Hospital, Bass Coast Health, Wonthaggi, Victoria, Australia
  6. 6Independent researcher, Melbourne, Victoria, Australia
  1. Correspondence to Dr Nicholas Faulkner; nicholas.faulkner{at}monash.edu

Abstract

Objectives There is a paucity of research on how to improve the functioning of health service boards, despite their importance in influencing patient care. We examined the impact of simulation-based training on health service board members’ perceptions of their skills in communicating during board meetings and of board meeting processes.

Design Prospective, cluster randomised controlled trial.

Setting Health service boards in Victoria, Australia.

Participants Twelve boards were randomised, and pre- and post-intervention data were collected and analysed from 57 members of these boards.

Interventions Boards were randomly allocated to either a treatment condition in which they received a 2-hour simulation-based training session or to a wait list control condition.

Primary and secondary outcome measures Primary outcome variables were board members’ perceptions regarding: (1) their skill and confidence in communicating during board meetings and (2) processes at their board meetings. Measures were collected in the intervention group before and 3 months post-training and compared with a wait list control group.

Results Skills and confidence in communicating during board meetings was higher after training (control marginal mean=5.11, intervention marginal mean=5.42, mean difference=0.31, 90% CI (−0.03 to 0.66), one-sided p=0.068, d=0.40). Board meeting processes were also improved after training (control marginal mean=4.97, intervention marginal mean=5.37, mean difference=0.40, 90% CI (0.14 to 0.65), one-sided p=0.005, d=0.54).

Conclusions Simulation-based training appeared to improve board members’ skills and confidence, and perceptions of board meeting processes. A larger scale trial is needed to examine possible impacts on patient outcomes.

Trial registration Open Science Framework: http://osf.io/jaxt6/; Pre-results.

  • education & training (see medical education & training)
  • health policy
  • organisation of health services
  • quality in health care
http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

Statistics from Altmetric.com

Strengths and limitations of this study

  • This study tests an intervention to improve the effectiveness of health service boards.

  • Extensive evidence review and stakeholder consultation informed the development of the simulation-based training intervention.

  • The study focused on self-assessed skills and confidence and board meeting processes.

  • The scope of the study did not include an assessment of patient outcomes.

Introduction

Research indicates that hospital boards have the ability to influence patient safety and quality of care.1–4 However, multiple inquiries and reviews have raised concerns about the skills and experience of board members and the effectiveness of board processes for achieving these goals.5 6 For example, a government review attributed a cluster of perinatal deaths in Victoria, Australia7 in part to suboptimal board processes. The review concluded that, ‘gaps in board skills, information and oversight are a key priority for strengthening governance of patient safety in hospitals’7

Although there is a growing body of research investigating the experiences and practices of health services board members,1 2 8 little is known about which interventions are effective for improving health service board members’ skills and board functioning. However, research has suggested that developing, evaluating and implementing interventions to overcome challenges faced by boards are needed to improve the governance of patient care.1

An emerging body of evidence suggests that one such challenge involves board members having effective but sometimes difficult conversations with executive directors and other board members.1 9–11 A qualitative study of health service board members and executives found that ‘monitoring progress’ and ‘holding staff to account’ were the among the key leverage points that health service board members could use to improve quality of patient care.1 Similarly, research on other types of boards has found that effectively questioning executives on their actions when necessary and working together with executives to obtain necessary information to monitor and improve performance are among the most important challenges facing supervisory boards.9 Finally, research has found that the effectiveness of UK National Health Service boards is compromised by issues relating to internal dynamics, including communication issues.11

One intervention that may improve the health service board members’ skills in having difficult conversations with other board members and executives is simulation-based training. Simulation-based training is an educational approach that places learners in realistic situations that provide an opportunity to practice and learn in a safe environment.12–14 It has increasingly been used to foster adult education in the medical field, and there is emerging evidence that simulations are more effective than traditional learning methods.12 15 Studies have found that exposing student health practitioners to simulations can foster the development of knowledge and skills,16 including communication skills.17 Debriefing and allowing participants to reflect on the simulation practice is integral to the success of this type of training.18 Simulation training enables guided practice that may develop the difficult skill of communicating within a health service board meeting to support the optimal delivery of quality healthcare. However, there is a paucity of research on the use of simulation training in health service board settings.

The current study hypothesised that providing simulation-based training on board communication would improve board members’ skills and confidence in communicating effectively during health service board meetings and improve the effectiveness of board meeting processes. A cluster randomised controlled trial design was chosen due to its suitability for assessing an educational and community-level intervention in a real-world setting.

Method

Setting

The study took place in Victoria, Australia. In the Victorian public health system, there are 85 independent health service boards with directors appointed by the Minister of Heath and governed by the Department of Health and Human Services. Victorian health service boards are responsible for the effective and efficient governance of their health service, including monitoring and improving quality and safety, and risk management.

Intervention development process

The intervention was developed through a structured four-step process. First, we convened an expert panel to steer the topic development. Second, we undertook an evidence and practice review to understand existing knowledge of effective strategies to optimise boardroom functioning and the lived experience of the health boardroom sector. Third, we facilitated a structured stakeholder dialogue to deliberate on the evidence and prioritise a feasible intervention that could be piloted. Fourth, we worked with workshop facilitators (JW-K and GP) to develop realistic, context-specific scenarios for the scenario-based training. Further details on these processes are provided in the trial protocol.19

Patient and public involvement

Although no patients or members of the public were directly involved in the design of or recruitment for the trial, one healthcare consumer consultant represented patient perspectives in the structured stakeholder dialogue described above.

Trial protocol and registration

We prospectively registered the trial before data collection (http://osf.io/jaxt6). We also published a study protocol that detailed the background, method and analytic approach prior to completing data collection.19 The background and methods section reported here is consistent with those reported in the published protocol.

Study design

The study used a cluster randomised design with a simulation-based training intervention group and a wait list-control group. Primary outcome measures were collected using a survey at baseline and at 3 months. The data collection period began in May 2018 and concluded in January 2019 (boards started in the trial on different dates to accommodate different board meeting dates and agendas). Health service boards were randomly allocated to intervention or control arms by central computer randomisation using randomly permuted blocks of four. Randomisation was stratified by region (metro/regional) to ensure similar numbers of regional and metropolitan boards were allocated to each arm of the trial.20 Consent from board chairs to participate in the study was gained before randomising their board to conditions. Due to the nature of the intervention, no blinding after assignment was used.

Participant eligibility

All members of the health service boards participating in the trial were eligible to participate. No exclusion criteria were applied. Health service boards were recruited jointly by the Victorian Department of Health and Human Services and the Victorian Managed Insurance Authority, who approached boards in both metropolitan and regional areas in Victoria, Australia.

Sample size calculations

Sample size was determined prospectively and primarily by time and budget constraints. We anticipated including 12 boards in the trial, each of which we estimated would have five board members who would provide both pre-data and post-data (ie, approximately 60 individuals across 12 boards). We prospectively decided to use a p<0.1 significance level due to the limited sample size. We also prospectively decided to use one-sided significance tests because, for practical purposes, the resulting course of action (ie, discontinue training) would have been the same if the simulation training had a negative effect or had no effect.21–24

We estimated that this sample size would provide 80% power to detect a 0.39 standardised mean difference between the intervention and control conditions in time 2 outcomes, controlling for time 1 outcomes.19 25 This meant that the trial was powered to detect an effect size smaller than that found in existing research examining the effect of other forms of training on physicians’ communication skills26 and smaller than the average effect of patient simulation training in nursing education, as identified in a meta-analysis.12

Intervention

The intervention involved immersive, simulation-based training of health service boards to increase their confidence in asking targeted questions and obtaining satisfactory responses. Approximately 1 week prior to each workshop, members of the workshop team reviewed publicly available materials about board members to understand the board composition. They also spoke with the Board Chair for 20–30 min to learn about the current function of the board and identify any specific issues faced by the board that could inform the workshop. During this conversation, the board chair was given an opportunity to specify which of the three possible ‘starting points’—asking difficult questions, dealing with pushback or refocusing the conversation—would be the most suitable starting point for their board. This was used to establish the best scenarios for the workshop. This conversation prior to the workshop also enabled the Board Chair to clarify any questions about the workshop. The Health Service Chief Executive Officer (CEO) was also contacted prior to the workshop to ensure that they were aware of the trial and field any questions they may have. Each CEO was offered an opportunity of a briefing by the workshop team after the workshop to briefly demonstrate the workshop simulation activity. This did not involve divulging any information from the workshop itself, which was confidential to participants to enable a ‘safe space’ in which to discuss their communication challenges and practice strategies to address these.

Training sessions ran for 2 hours and involved all board members. Each participant had the opportunity to engage in a short (approximately 5–8 min) scenario-based simulation exercise with a facilitator and trained actor. Participants could choose between prepared scenarios or a scenario they would like to explore. Simulations were set up by the facilitator and conducted as improvisations between the participant and the actor. Each simulation was observed by all other participants in the session and was divided into three parts: part 1 was the initial simulation/challenge, part 2 was facilitated reflections and feedback and part 3 was a repeat simulation where the participant could try new strategies. This methodology was consistent with the agenda-led, outcome-based analysis technique that is commonly used in medical education to structure simulation and other forms of training.27 28

Data collection

Participants were sent emails at both time points inviting them to complete the survey. Additionally, participants in the simulation training received a face-to-face reminder and were given an opportunity to complete the time 1 survey online immediately before the training occurred.

Outcomes

Primary outcomes were perceived skills and confidence in communicating in health service board meetings (eg, ‘I am confident in my ability to get the information I need in Board meetings’, ‘Even when other board members disagree with me, it’s easy to express my opinions’) and perceptions of board meeting processes (eg, ‘All directors make robust contributions to discussions’, ‘There is adequate time in Board Meetings to address all agenda items thoroughly’). Secondary outcomes were self-reported perceptions of the relevance and utility of the training. These measures were only included in the time 2 survey in the intervention arm. They included both Likert response questions (eg, ‘The training was relevant and useful to my role as a board member’, ‘The training has helped the board better achieve its objectives’) and open-ended questions designed to examine participants’ qualitative reflections on the training (eg, ‘What are your reflections on the training you received?’). Likert response items used a scale ranging from 1=‘strongly disagree’ to 6=‘strongly agree’. Complete wording of all questions and response scales can be found in the published protocol.19

Analysis methods

Survey items with Likert response categories were subjected to exploratory factor analysis (using principal axis factoring and promax rotation) to identify a plausible factor structure. As per the protocol, the items reflecting ‘board meeting processes’ and ‘skills and confidence’ were entered into separate analyses, as were the Likert items measuring secondary outcomes. For all analyses, the number of factors was determined using Cattel’s scree test criterion.29 30

Generalised estimating equations (GEE) were used to estimate the impact of the simulation training on primary quantitative outcomes while accounting for the nesting of individual participants within boards. These analyses were adjusted for remoteness area (metro, regional) because it was used as a balancing variable in the stratified randomisation.31 As such, the independent variables included in these analyses were: time 1 scores; experimental condition (0=control, 1=treatment) and remoteness area (0=metro, 1=regional and remote). The dependent variables were time 2 scores. GEEs usually use a Huber-White sandwich estimator that requires a large number of clustering units (more than 30–50 boards) to generate accurate estimates of standard errors.32 33 We used a one-step jack-knife estimator to minimise this potential limitation.34–36 We calculated Cohen’s d effect size measures using techniques appropriate for trials using a two independent groups, pretest and post-test design.37 GEE analyses were conducted in R (V.3.5.0)38 using the geepack package (V.1.2).39

For qualitative open-ended questions, we used thematic analysis to analyse responses.40

Results

Participants

As shown in figure 1, seven boards were randomised to the control wait list and five to the intervention arm. We continued to invite boards until we had recruited 12 boards into the trial. A total of 57 participants provided responses at both the pretime and post-time points and were included in the primary analyses. Demographic characteristics of participants were similar in both the intervention and control conditions. In the control condition, participants had mean age of 60.0 (SD=8.1, min=43, max=73), 15 (42.9%) were men, and there were three metro and four regional boards. In the intervention condition, participants had a mean age of 58.5 (SD=9.2, min=41, max=75), 10 (45.5%) were men, and there were two metro and three regional boards.

Figure 1

Flow diagram showing progression through the trial.

Computing multi-item scales

Results indicated that single-factor solutions were appropriate for each of the ‘skills and confidence’, ‘board meeting processes’ and ‘perceptions of training’ measures (loadings for all items were greater than or equal to 0.48). Scree plots are shown in online supplemental figures S1 to S5. We averaged the relevant items to compute internally consistent scales measuring productive board meeting processes (αtime 1=0.87, αtime 2=0.89), skills and confidence (αtime 1=0.91, αtime 2=0.93) and perceptions of the training (α=0.96). The intraclass correlation coefficients for the primary outcomes were 0.11 and 0.07 for productive board meeting processes and skills and confidence, respectively, at time 2. The correlations between time 1 and time 2 measures were r=0.41 for board meeting processes and r=0.48 for skills and confidence. Unadjusted descriptive statistics at both time points are shown in online supplemental table S1.

Effect of the training on primary outcomes

Results indicated that the intervention significantly improved board members’ communication skills and confidence (b=0.31, 90% CI (−0.03 to 0.66), p=0.068) and board meeting processes (b=0.40, 90% CI (0.14 to 0.65), p=0.005) (see table 1 and figure 2). Standardised effect size measures indicated that the intervention had a ‘small–medium’ effect on perceived skills and confidence in communicating (d=0.40, 90% CI (−0.04 to 0.84)) and a ‘medium’ effect on board meeting processes (d=0.54, 90% CI (0.19 to 0.88)).

Table 1

Generalised estimating equation results

Figure 2

Marginal means of time 2 outcomes, controlling for region and time 1 scores. ***p<0.01, *p<0.1. Error bars show 90% CIs. Dots show predicted values.

Perceptions of the training

The median score on a series of statements about the utility of the training was 5.14 (on a scale with a possible range of 1–6), suggesting a generally positive response. Overall, qualitative responses to the training session were also positive.

It was very powerful training and each person received some feedback that was pertinent to each individual.

However, some comments indicated that they would have preferred if the training was pitched differently.

I don’t think it was pitched to a relatively mature board and I found it frustrating on that basis.

A few participants suggested that the impacts of the training went beyond improved communication, highlighting an improvement in board dynamics.

In addition to assisting with communication between board members and the CEO, it provided an opportunity to get to know other board members at a deeper level.

The majority of intervention participants stated that the training session altered their behaviour in board meetings. Participants reported feeling more comfortable and confident to speak up and ask questions in the board meetings.

You think about using better word choices before speaking or phrasing questions differently to either get your point across or to find out more information

Participants also described being more mindful about the ways in which they frame questions and the styles of questions that they use. The training also helped to ensure that communication occurs in a respectful manner.

It has helped me to be more mindful about more tactful and effective ways to seek answers to my questions.

The ability to express oneself in such a way to maintain respect for others was strongly put forward.

While approximately half of the participants did not think any aspects of the training session needed to be changed, a number of participants suggested increasing the length of the session. This included providing more time for role playing, tuition about communication skills and summing up the session. A couple of participants also suggested that a follow-up session would be useful.

I thought that the training was excellent, a little more time for the training and follow-up would be helpful to reinforce new learnings.

Discussion

Research and inquiries after major health service failures have identified health service boards as an important influence on the quality of care in hospitals.1–7 The current study tested the impact of simulation training as a technique for improving how health service boards function and improving the skills and confidence of health service board members. This focus makes it unique in the published literature: research on health service board members has typically focused on understanding their practices rather than trialling interventions to measure their impact on other outcomes.

Our findings provide initial evidence that simulation-based training may improve board meeting processes and increase board members’ skills and confidence in communicating during board meetings. Qualitative findings indicate that the intervention may have improved board meeting processes by helping board members better understand other people on their board and by increasing their ability to seek information in a tactful, respectful and effective manner.

By conventional effect size standards, quantitative results show that the training had a medium-sized effect on board meeting processes and a small-to-medium-sized effect on communication skills and confidence. These effects are broadly consistent with, although slightly smaller than, those observed in meta-analyses examining the effects of simulation training in other contexts, which have typically found medium to large effect sizes when comparing simulation training to no training.41–44 They also indicate that other interventions beyond training may also be needed to further improve board skills and functioning.

There are several possible reasons that may have contributed to the effect size being slightly lower in this study than in studies of simulation training in other contexts. One possibility is that many of the board members included in the study were already highly confident, and therefore had less room to improve than the medical students included in many previous studies on simulation training. Consistent with this possibility, meta-analyses have shown that the effects of simulation training tend to be larger for graduate students than some types of clinicians.12 A second possibility is that the use of self-assessed outcomes may have reduced the effect size, as participants may not have been able to recognise gaps in their own knowledge and skills prior to completing the training. This phenomenon is called ‘response shift’ in educational literature.45 Accordingly, reviews of simulation training have found it tends to have a larger effect on ‘performance-based’ outcomes than on ‘self-assessed’ outcomes.12 A third possibility is that the intervention was brief in nature. A greater effect size may have been observed with a longer training period or the ability to reinforce skills across more than one session.

In addition to improving skills and board processes, simulation training appears to be generally appreciated and considered useful by health service board members who participate in it. This finding is consistent with studies that have examined participant satisfaction with simulation training in other contexts, which have typically found that simulation training results in higher student satisfaction than traditional forms of instruction.43 44 46

These findings have implications for theory and policy. From a theoretical perspective, our results indicate that simulation training is effective for changing communication skills and behaviours in health service board settings. These findings extend knowledge on simulation training by showing that simulation training can be effective not only in clinical settings46 47 but also in health service board settings. For policy, our findings suggest that using simulation training to provide board members with a safe place to practice and learn can improve their skills and confidence in communicating in board meetings. This is particularly important given that there is limited research on how to improve these skills, but a growing body of research indicating that they are critical for maximising patient safety and quality of care.1 9 The findings of this study will be used to inform the future development and large-scale implementation of simulation-training for health service board members in Victoria, Australia.

This study is not without limitations. First, the study did not include measures of objective data about board meeting processes (eg, agendas, meeting transcripts) and relied instead on self-report measures. As such, there remains a possibility that different findings may be obtained if objective measures were used. Second, the study did not include measures of patient outcomes. Future larger scale research is needed to examine the extent to which improvements in board processes result in improvements to patient safety. Third, for practical and ethical reasons, our sample only included boards who were willing to receive the training. As such, we were only able to estimate the effect of the intervention on hospital boards that were willing and interested in participating. The effect may have been different if boards were forced to receive the training, but our trial could not examine this possibility. Forth, the timeframe used here only allowed us to estimate the effects of the training at 3 months. Further research is needed to examine how long improvements last.

In sum, this study indicates that the functioning of health services boards can be improved through simulation-based training. In doing so, it provides some of the first evidence from a randomised controlled trial about what works for improving the effectiveness of health service boards. Our findings provide a platform for larger trials of the intervention to a wider group of boards and further evaluation of effects on patient outcomes.

Acknowledgments

We gratefully acknowledge Kristy Spillman’s assistance in recruiting boards for this trial, and the valuable contributions of participants in the stakeholder dialogue that helped to inform the intervention design.

References

Supplementary materials

  • Supplementary Data

    This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

Footnotes

  • Twitter @nickjfaulkner, @mbismark, @BraggePeter

  • Contributors NF, BWr, PB and AL conceived and developed the study protocol. BWr, PB, MB, JB, SB, BWa, JW-K and GP provided expertise with design of the intervention. BWr, AL, PB and NF coorganised data collection. NF and AL analysed the results. NF, BWr and AL wrote the first draft of the manuscript. All authors provided feedback and approved the final manuscript. We also gratefully acknowledge Kristy Spillman’s assistance in recruiting boards for this trial.

  • Funding This work was funded by the Victorian Managed Insurance Agency (VMIA) VMIA2017. The contents of this article are the responsibility of the authors and do not reflect the views of VMIA.

  • Competing interests JB is employed as the Head of Risk Programs & Client Advisory by VMIA who are funding this research. NF, BWr, PB, AL and GP report receiving funding to their institution from VMIA for this research.

  • Patient consent for publication Not required.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Data availability statement No data are available. Data are not available for sharing due to the nature of the ethics approval granted for this study.

  • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.