Article Text
PDF
Abstract
Background Outcome benefits of using the WHO Surgical Safety Checklist rely on compliance with checklist administration.
Objective To evaluate engagement of operating room (OR) subteams (anaesthesia, surgery and nursing), and compliance with administering checklist domains (Sign In, Time Out and Sign Out) and checklist items, after introducing a wall-mounted paperless checklist with migration of process leadership (Sign In, Time Out and Sign Out led by anaesthesia, surgery and nursing, respectively).
Methods This was a pre-post observational study in which 261 checklist domains in 111 operations were observed 2 months after changing the checklist administration paradigm. Compliance with administration of the checklist domains and individual checklist items was recorded, as was the number of OR subteams engaged. Comparison was made with 2013 data from the same OR suite prior to the paradigm change.
Results Data are presented as 2013 versus the present study. The Sign In, Time Out and Sign Out domains were administered in 96% vs 98% (p=0.69), 99% vs 99% (p=1.00) and 22% vs 84% (p<0.001) of cases, respectively. The percentage of relevant checklist items administered in each domain was 60% vs 92%, 84 vs 93% and 80% vs 99%, respectively (p<0.001 for all comparisons). Two-subteam (or better) engagement at Sign In (surgeons usually absent) was 40% vs 94% of cases. Three-subteam (or all staff present) engagement at Time Out and Sign Out was 15% vs 92% and 9% vs 25% of cases, respectively (p<0.001 for all comparisons).
Conclusions Improvements in team engagement and compliance with administering checklist items followed introduction of migrated leadership of checklist administration and a wall-mounted checklist. This paradigm change was relatively simple and inexpensive.
- Checklists
- Safety culture
- Compliance
- Leadership
- Patient safety
Statistics from Altmetric.com
Introduction
The WHO Surgical Safety Checklist is a series of prompts that aim to improve communication and prevent errors or omissions in operating room (OR) patient care. It is administered in three ‘domains’: Sign In (when the patient arrives in the OR); Time Out (just prior to the first surgical incision) and Sign Out (prior to the patient leaving the OR). The Checklist or adaptations thereof have been widely adopted since the original report of its positive effects on patient outcomes.1 There has been ongoing research into how to optimise checklist benefits and there is increasing evidence that a positive influence on patient outcomes is more likely if there is good compliance with administration of checklist items.2–5
Our institution participated in the original WHO Surgical Safety Checklist study,1 and since then we have incorporated an adapted version (hereafter referred to as the ‘Checklist’) in standard OR practice under a paradigm in which the circulating nurse is responsible for initiating and administering all three domains from a paper copy of the Checklist. We have retained a close focus on monitoring and fostering both compliance with Checklist administration and engagement of OR team members in the process. Like other institutions who have audited their own experience, we have noted poor compliance with administration of the Sign Out domain and inconsistent administration of the various Checklist items listed within each of the three domains. In addition, we have reported difficulty in consistently achieving undistracted engagement of OR team members in the administration process. Indeed, despite periodic reminders and discussion about the importance of sound Checklist practice at staff forums, these Checklist practice metrics showed no general trend to improvement (and in some respects deteriorated) between audits published in 2011 and 2013.6 ,7 In this study, we report the effect on compliance and engagement of introducing a new Checklist administration paradigm in which a large poster version of the Checklist is mounted on the OR wall and responsibility for leading the three domains is migrated among the OR professional groups.
Methods
We previously reported superior engagement in Checklist administration at a local hospital where the Checklist was wall-mounted and each domain was led by a different OR professional group (anaesthesia, nursing and surgery) hereafter referred to as ‘subteams’ of the overall OR team. Specifically, anaesthesia led Sign In; surgery led Time Out and nursing led Sign Out.7 We believed this strategy could improve engagement in our own environment for several reasons. For example, we frequently observed a time-pressured anaesthetic subteam continuing activity such as application of monitoring equipment or intravenous cannula insertion while the circulating nurse administered the Sign In domain; and similarly, we often observed the surgical subteam continuing to arrange the sterile field while the circulating nurse administered the Time Out domain. It seemed logical that asking the anaesthetic and surgical subteams to lead these respective Checklist domains would effectively obligate at least one subteam member to cease other activity and engage properly. Large wall-mounted Checklist posters were a necessity for this process because gowned and gloved surgeons would have difficulty leading from a paper Checklist at Time Out, but we also anticipated that the wall-mounted format might confer other benefits. For example, in use of the paper-based Checklist we commonly witnessed administering nurses reciting it from memory which often resulted in omission of items and sometimes the administration of confabulated ‘items’ not actually on the Checklist. Since administration from large wall-mounted posters would render any deviation from the Checklist obvious to the entire OR team, it seemed plausible that this would discourage such deviations. On this basis, we set out to introduce the wall-mounted format with migrated leadership to our ORs and to evaluate the effect on Checklist compliance and OR subteam engagement.
The resulting prospective, observational study was approved by the University of Auckland Human Participants Ethics Committee (ref.: 013356). The setting was the same 13-room multidisciplinary adult OR suite observed in our previous studies of the former nurse-led paper-based Checklist administration paradigm.6 ,7 The new Checklist process included (1) large print, wall-mounted Checklist posters (one for each Checklist domain) and (2) allocation of leadership of each domain to an OR subteam central to the processes occurring at that phase of the procedure and therefore prone to being distracted and failing to engage. Specifically: the Sign In domain was allocated to the anaesthetic subteam who would be preparing the patient for induction of anaesthesia; Time Out was allocated to the surgical subteam who would be about to make the first incision and Sign Out was allocated to the nursing subteam who would have just completed the first swab and instrument count. In preparation for the change, there was consultation to establish buy-in from leadership of all affected OR subteams. A Checklist Revision Group visited at least one regular meeting of each OR subteam to explain the new process and answer questions. The wall-mounted posters were compiled, printed, evaluated for readability and piloted for 2 months in one OR consistently used by a single surgical team. The wall poster adaptations of the Checklist currently in use are given in online supplementary file 1. The posters were put up in all ORs a week prior to the roll out date so that staff became accustomed to them, and a poster board was established in the OR suite to further remind staff of the key changes and to count-down to the roll out which took place in October 2014. Checklist Revision Group staff were present for guidance in the operating suite during the first week following the change. The study took place in January 2015 after a 2-month ‘bedding in’ period.
Data collection
Data were collected over a 4-week study period. All procedures during the study period requiring the presence of all three OR subteams (anaesthesia, surgery and nursing) were eligible for observation. Data were collected by a single observer (PO) following a prestudy training and orientation period conducted by a senior researcher (SJM) in December 2014. Before starting data collection, we ensured inter-rater reliability between the observer and the senior researcher was suitable (>90% concordance). Any discrepancies during the training period were discussed, and interpretations of our definitions (see below) were clarified.
The observed cases formed a sample of convenience across the range of surgical specialties operating in the suite. The choice of cases observed was left to the observer who, to maximise data capture over the 1-month study period, moved between ORs opportunistically in order to be present for as many checklist domain administrations as possible. This meant that on some occasions not all Checklist domains were observed for each case. For each case in which at least one Checklist domain was observed we recorded surgical discipline, procedure name and case acuity (elective or acute). As in our previous studies, the observer's purpose for being in the OR was not explicitly clarified to the staff present in order to avoid the obvious potential for observation to modify the behaviour.
To ensure that the data could be compared with our previous studies the observer used the same methods and definitions to assess and record compliance with Checklist item administration, timing and subteam engagement during the use of the Checklist. Completion of an individual Checklist item was defined as verbal communication of that item during Checklist administration by either the administering person or another staff-member present. Any items not discussed during Checklist administration but communicated by staff at other times were not considered to have occurred as part of the Checklist and so were recorded as not completed.
As previously, we considered an individual to be engaged in the Checklist process if he or she was listening and/or contributing, with cessation of all other activities and conversations throughout. Since a major goal of the Checklist is to ensure communication of important information between the different professional subteams in the OR, our outcome focus was on the simultaneous engagement of those professional subteams. We required at least one individual from an OR subteam (anaesthesia, surgery and nursing) to be engaged (as defined above) for that subteam to be considered engaged. Thus, our evaluation of engagement during each checklist domain administration was a categorical measure with three possibilities: one, two or three subteams engaged. A fourth engagement outcome was those instances in which all staff present in the OR (ie, all members of all subteams) were engaged. To be clear, it was possible to have three-subteam engagement by virtue of one member of each subteam being engaged without having engagement of all staff present in the OR. We recorded the presence of each OR subteam to ensure that their absence did not confound this measure (eg, the surgical subteam was often not present at Sign In).
Adherence with correct timing of the Checklist domains was assessed. In our centre, policy stipulates that the Sign In domain has to be completed in the OR prior to any intervention or drug administration, Time Out has to be completed immediately prior to first incision and Sign Out has to be completed immediately following completion of the first swab and instrument count while the senior surgeon is still present in the OR.
For simplicity, data from our current study were compared with those from our most recent previous study conducted in the same OR suite.7 It is important to note that there was little difference in levels of Checklist compliance and engagement reported in our 20116 and 20137 studies despite intervening periodic reminders about good Checklist practice. Indeed, the most notable difference was a deterioration in subteam engagement at Sign In and Time Out in 2013 compared with 2011. It should also be noted that there were no other improvement initiatives targeted at Checklist practice in the period between 2013 and the present study. Based on this combination of prior stability in indices of Checklist performance and the lack of any reason to believe that performance would have improved after 2013 (indeed, anecdotal impressions suggested the opposite), we believe that the 2013 data are a reasonable point of comparison for our current data. Both earlier studies used the same methods and compliance assessment tool to record compliance, engagement and timing of Checklist domain administration as used in the present study.
Endpoints and analysis
There were two primary endpoints. The first was the rate of compliance with administration of the Checklist domains and individual Checklist items. We calculated ‘domain compliance’ as the percentage of all observed cases in which the domain was administered and ‘domain completion’ as the percentage of eligible Checklist items that were completed (for domain-compliant cases). Items considered not applicable for a case were excluded from the analysis of that case. The second primary outcome analysed by Checklist domain was the percentage of domain-compliant cases in which one, two or three OR subteams were engaged, and also the percentage of cases in which every staff member present in the OR was considered engaged. A secondary endpoint was the percentage of domain-compliant cases in which correct domain timing was observed.
We compared our primary and secondary endpoints for 2013 versus the present study using a χ2 test, Fisher's exact test or two-tailed unpaired t tests (table 2). For engagement, we compared the two data sets (2013 and 2015) with respect to the expected gold-standard level of engagement for each Checklist domain; thus, for Time Out and Sign Out when all three OR subteams are invariably present, the comparison was for three-subteam engagement. For Sign In where, in our institution, the surgical team is frequently absent, we combined two-subteam and three-subteam engagement into a single category for the purposes of this comparison. We excluded entire OR team engagement from our analysis (due to insufficient numbers, and because this category was dissimilar in nature to the one-subteam, two-subteam and three-subteam engagement categories). A p value of 0.05 was taken to indicate significance for all analyses. We did not correct our p values for multiple testing. Analyses were conducted using SPSS Statistics V.22.
Results
Two hundred and sixty-one checklist domains (85 Sign In, 100 Time Out and 76 Sign Out) in 111 operations were observed over the 1-month study period. A summary of the case mix is provided in table 1. Data describing domain compliance, domain completion and subteam engagement are presented in table 2 where comparison is made with our 2013 data from the same OR suite.
The distribution of cases, summarised by procedure type and case acuity
Domain compliance, domain completion and subteam engagement, by domain and audit year
Domain compliance
In comparison with the 2013 study (prior to the change in administration paradigm), there was no change in compliance with administration of the Sign In and Time Out domains, though there was a ceiling effect; compliance with these domains was very good in both the 2013 and 2015 data. In contrast, there was a substantial improvement in compliance with administration of the Sign Out domain in 2015 (22% vs 84% of cases in 2013 and 2015, respectively).
Domain completion
The 2015 data suggest improved consistency in checklist item administration in all domains under the new paradigm (table 2).
Team engagement
The new Checklist paradigm was associated with substantially increased and statistically significant improvements in subteam engagement in all three domains (table 2). Most notable was the marked improvement in three-subteam engagement during Time Out (15% of cases in 2013 and 84% in 2015, p<0.001). This improvement is even more robust if instances of engagement by all OR staff are included with the three-subteam engagement (15% of cases in 2013 and 92% in 2015).
Timing of checklist administration
The comparison of timing of Checklist administration (compliance with the institutional recommendations described in methods) was qualitatively similar between 2013 and 2015. The timing of all domains showed high compliance in both studies with an obvious ceiling effect [93% (2013) vs 100% (2015) of cases for Sign In, 98% (2013) vs 100% (2015) of cases for Time Out and 95% (2013) vs 98% (2015) of cases for Sign Out].
Discussion
A comparison of our 2013 and 2015 data demonstrates that implementation of the wall-mounted Checklist with migrated leadership in our operating suite was associated with substantial improvements in the following: compliance with administration of the Sign Out domain (22% of cases vs 84%, p<0.001); compliance with administration of the individual Checklist items within all domains (Sign In 59.9% of eligible items vs 91.9%; Time Out 84.4% vs 92.9%; Sign Out 80.0% vs 98.9%, p<0.001 in each case) and expected gold-standard subteam engagement within all domains (Sign In 40% of cases vs 83%; Time Out 15% vs 84%; Sign Out 9% vs 22%, p<0.001 for each). It is notable that during several hundred Checklist domain observations in our previous (2013) study, we never recorded proper engagement of every staff member present in the OR during Checklist administration.7 Although still far from universal in the present study, engagement of the entire OR team was seen in at least some cases (table 2).
The improvement in compliance with administration of Checklist items across all three domains (‘domain completion’) seemed most likely attributable to the wall-mounted Checklist format in the manner we anticipated (see Methods). Anecdotally, the directed gaze of the administering person often seemed to draw the attention of other OR staff to the Checklist, thereby enhancing their engagement in the process (rather like the use of visual aids in a lecture). The fact that everyone was looking at the Checklist appeared to discourage omission of prescribed items or confabulation of new ones. Again as anticipated, the improvements in subteam engagement at Sign In and Time Out seemed related to allocation of leadership of these domains to the subteams previously observed to be most prone to disengagement (anaesthetists and surgeons, respectively). We also formed the view that migrated leadership appeared to promote an understanding of the challenges of engaging colleagues in the process which encouraged members of the various subteams to be more compliant when it was another team's turn to lead.
In a related vein, the substantial improvement in compliance with administration of Sign Out was arguably the most interesting (and least anticipated) of the gains. In our 2011 study,6 Sign Out was administered in only 2% of cases and this poor result was attributed to uncertainty over exactly when it should take place. Unlike Sign In (done immediately after the patient entered the OR) and Time Out (done just prior to the first surgical incision), the Sign Out domain initiation was not clearly linked to any other discrete event. After the 2011 study, it was resolved to link Sign Out to completion of the first swab and instrument count as an aide memoire for initiation, but this resulted in only a modest improvement to administration in 22% of cases in 2013.7 An intuitively obvious explanation for the much greater improvement to 84% of cases in the present study is the fact that the Checklist was visible on the wall as a reminder. However, excellent nurse-led domain compliance with Sign In and Time Out had been recorded in both 2011 and 2013 studies where there were no Checklist posters on the wall so posters were clearly not obligatory to achieve domain compliance. We suspect that our paradigm change has unintentionally resolved a more complex problem with Sign Out administration. In informal discussions following the change, nurses consistently suggested that their improved compliance with Sign Out resulted mainly from the more supportive environment in which the process was taking place. Specifically, because the anaesthetists and surgeons were engaging in the Checklist process by initiating and leading their ‘own’ domains (Sign In and Time Out, respectively), the nurses felt empowered and motivated to proactively initiate the Sign Out domain they were entrusted with, whereas in the past it had ‘seemed more like an imposition on the flow of work’ and the nurses had often felt more comfortable simply omitting it.
Despite this improvement in compliance with administering Sign Out, there was still a disappointing level of three-subteam engagement in this domain. This was almost invariably because the surgeons continued with activities in the surgical field (such as suturing the wound) during nurse-led Sign Out and was similar to the problem when the nurses previously led Time Out; that is, the surgeons would usually acknowledge the process but not completely engage in it by ceasing all other activities. We concede that there is no hard evidence that multitasking during Checklist administration detracts from its efficacy in preventing errors, but it seems intuitively suboptimal and easily avoided since complete engagement in the Checklist process requires little time.
We believe the findings of this study are important because there is substantial evidence suggesting that the use of the WHO Surgical Safety Checklist (or similar variants) can improve postsurgical outcomes1–5 ,8 but does not always do so.9 One likely reason for failure to improve outcomes is inadequate engagement with the use of the checklist;10 and improvement in outcome also seems to be related to completeness of checklist administration.2–5 It follows that continued pursuit of effective strategies for improving compliance and engagement in checklist use is justified. The importance of this goal was stressed in a recent editorial by Haynes et al,11 which stated ‘no quality tool, particularly one that relies upon team communication and interaction, can be expected to make a difference if it is not actually used in any meaningful sense’. It is recognised that optimising the use of a surgical safety checklist is difficult and requires a multifaceted approach.12 The use of a wall-mounted Checklist as opposed to a paper Checklist and the sharing of leadership as opposed to vesting responsibility for leading all domains solely in the OR nursing subteam are two such ‘facets’ that may contribute to achieving optimisation. Others have alluded to possible positive effects on compliance through the use of these strategies,13–15 but we are unaware of any other studies in which their efficacy has been evaluated.
There are several limitations to our study which must be acknowledged. First, the longitudinal preintervention and postintervention design leaves open the possibility of influences other than our process change contributing to the observed improvements. However, we emphasise that there were no substantive gains in Checklist administration quality between our two previous studies published in 20116 and 20137 despite attempts to raise staff awareness of the need for improvement after the 2011 study. Moreover, there were no obvious confounding influences or other interventions instituted after the 2013 study that might have contributed to the improvements reported here.
Second, we used a single trained observer for gathering data in the ORs. This introduced the potential for idiosyncratic interpretation of observations and bias towards reporting a positive outcome. The observer underwent training to establish concordance in interpretation of OR events with a senior investigator, and it is self-evident from our data that the aspects of postintervention Checklist practice, which remained conspicuously suboptimal (eg, poor three-team engagement at Sign Out and a low incidence of all staff present in the OR being properly engaged), were recorded as such. We are therefore confident that the observations were sufficiently accurate and objective.
Third, this was a single-centre study, and the centre involved has a history of Checklist research. The generalisability of the results could therefore be questioned. Nevertheless, the new paradigm is a relatively simple inexpensive adjustment to practice and would, of itself, be readily generalisable. We encountered no significant difficulties in making the paradigm work in our OR suite, or in several others within our hospital complex where it has subsequently been introduced. Whether it would elicit the same improvements in compliance and engagement in another institution would depend on the prevailing standards of Checklist practice under existing protocols and the receptiveness of OR staff. Our own anecdotal experience was that receptiveness actually improved during the roll out as subteams given an active leadership role found that they actually enjoyed the process and found it useful.
Fourth, we have not demonstrated the durability of the measured improvements over time, but this will be the subject of future work.
In conclusion, we achieved substantial gains in compliance with Surgical Safety Checklist administration and in OR subteam engagement in the process by switching from paper-based to wall-mounted checklists, and by strategically migrating leadership of the three Checklist domains among the anaesthesia, surgical and nursing subteams. For any institutions struggling to optimise compliance and OR team engagement in a paper-based Checklist paradigm, the wall-mounted Checklist with migrated leadership is an option worth considering.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
- Data supplement 1 - Online supplement
Footnotes
Contributors SJM, TL and DAD were responsible for study design. AO was responsible for data collection and compilation. SJM provided clinical oversight of data collection. SJM, AFM and DAD provided general oversight of the study processes. JH conducted the statistical analyses. JH and SJM wrote the initial draft of the manuscript. All authors provided critical revision of the manuscript.
Competing interests AFM was the anaesthesia lead in the WHO Safe Surgery Saves Lives initiative and is Chair of the Board of the Health Quality and Safety Commission New Zealand.
Ethics approval The University of Auckland Human Participants Ethics Committee.
Provenance and peer review Not commissioned; internally peer reviewed.