Table 2

Studies of checklist compliance

ArticleStudy descriptionComparison populationMethod of measuring complianceCompliance measureChecklist complianceComments
Avansino et al19Study of intervention to improve compliance
Compliance measured over 12 months after checklist introduction
Monthly feedback of compliance provided to surgical team
Baseline at introduction of checklistAudit
Submitted paper checklists
Compliance defined as any aspect of checklist completedImproved compliance
Baseline compliance:
Overall 88%
Weekends 64%
Weekdays 90%
12-month compliance:
Overall 97%
Weekends 100%
Weekdays 94%
Feedback may increase compliance
Lax definition of compliance
Steady increase in compliance over 12 months
Gottumukkala et al31Study of intervention to improve compliance
All cases videotaped Compliance measured over 3 years through review of videos
Feedback provided to team with ‘corrective action’ to improve compliance
Baseline when recording system in place and evaluation startedAudit
13% of videos of checklists performed
Completion of each element of the checklist assessed by multiple ratersImproved compliance (time series analysis)
(crude values not available)
Decreased variability in performance over time
Feedback and incorporating stakeholder solutions to compliance failures may improve compliance
Possible selection bias and generalisability may be limited to interventional radiology but reliable measure of compliance
Steady increase in compliance
Khoshbin et al26Compliance measured 1 year after introduction of a surgical ‘time out’ and 2 years after introduction of a preoperative ‘huddle’NoneAudit
Direct observation of ‘huddles’ and ‘time outs’
Compliance defined as any part of ‘huddle’ or ‘time out’ completed
Completion measured for 4 elements of the huddle and 9 elements of the ‘time out’
‘Huddles’:
Compliance 64%
Mean completion 3.2/4
‘Time out’:
Compliance 99.1% Mean completion 6.2/9
Meaningful completion of checklist can be marred by the perception of the checklist as a ‘task’
Unblinded audit
Levy et al24Compliance measured over 7-week periodNoneAudit
Direct observation of cases
Compliance defined as completion of all preincision components of the surgical checklist
Completion measured for the 13 elements in the checklist
Hospital-reported compliance 100%
True compliance (full checklist completed) 0%
Time out compliance 97%
Completion
The average number of checklist items performed 4/13
4/142 had more than 50% of elements completed
Hospital-recorded compliance may not reflect implementation fidelity
Meaningful completion of checklist can be marred by the perception of the checklist as a ‘task’
Factors commonly completed were patient identification and procedure
Factors least completed were sterility confirmation, essential imaging, team member concerns and anticipated blood loss
Norton27Study of intervention to improve compliance
Universal Protocol introduced Education provided before implementation and feedback on effective use provided during audit
Baseline at introduction of checklistAudit
Direct observation and documentation by surgical team
Compliance defined as time out and site verification completionImproved compliance
Site initialled:
Observed 97%–100%. (increased from 88% first month)
Documentation 97%–100%
Time out compliance:
Observed 97%–100%
Documentation 97%–100%
Education and feedback may maintain compliance
Compliance does not always reflect meaningful completion
Limited to Universal Protocol
Norton and Rangel29Study of intervention to improve compliance
Two separate pilots used to develop and modify intervention
Ongoing education during implementation
Data from piloting of checklist before implementationAudit
Direct observation of cases
Compliance defined as ‘usage of the checklist’Improved compliance
First pilot, overall compliance 80%–90%
6 months after introduction
Sign in 90%–100%
Sign out 95%–100%
Addressing barriers identified through piloting the checklist and education may improve compliance
The Hawthorne effect may lead to improved compliance
Ongoing improvement in compliance
Ride et al25Compliance measured for 48 casesNoneAudit
Direct observation of cases
Measures include team member attendance
Compliance defined as verbal verification of all checklist items
Documentation of checklist completion
Correct use of checklist
Attendance for sign-in
surgeon 10%
Anaesthesiologist 95%
Verbal compliance:
Sign ins 95%
Time outs 69%
Sign outs 44%
Documentation:
Sign ins 79%
Time outs 71%
Sign outs 31%
Correct use:
Sign in and time out 46%
Sign out 23%
The sign out is poorly done and correct use of the checklist is not common
Correct use is not defined in this study
Challenges to compliance reported by authors included patient agitation and difficulty coordinating members of the care team
Montgomery et al30Study of intervention to improve compliance
Intervention to improve compliance using feedback provided to teams
Baseline prior to interventionAudit
Direct observation of surgical pause
Compliance defined as performance of the surgical pause
Completion measured for the 4 elements of the surgical pause
Compliance improved
Preintervention:
Compliance 100%
Completion of all 4 steps 51%
Postintervention:
Compliance 100%
Completion of all 4 steps 77%
Feedback may improve completion
Improvement of 4 elements:
Introduction 61%–87%
Identity and procedure 90%–100%
Checklist performed 98%–100%
Concerns voiced 71%–100%
Putnam et al28Study of intervention to improve compliance
Staged strategy to improve compliance
First stage: safety council and modification of checklist
Second stage: workshops stakeholder audits/feedback
Baseline prior to interventionsAudit
Direct observation of surgical pause
Compliance defined as completion of entire checklist
Completion measured for the 14 checkpoint elements.
Compliance improved
Baseline:
Compliance 0%
Completion 30%
After first stage:
Compliance 19%
Completion 76%
After second stage:
Compliance 61%
Completion 96%
Feedback, education and incorporating stakeholder solutions to compliance failures may improve compliance