Study outcome measures and main findings
| Lead author Reference Year | Outcome measures | Main findings | Level of evaluation |
|---|---|---|---|
| Aboumatar41 2012 | Primary outcome measures: preintervention and postintervention safety knowledge (19-item bespoke test), self-efficacy in safety skills (9-item bespoke survey), system-based thinking (using validated STS). Secondary outcome measures: Postintervention student satisfaction and safety intentions (2-item survey) | High participant satisfaction—intersession quality rated as excellent or very good by 92%. Significant improvement in composite systems thinking scores (61.15–67.56, p<0.001). Significant improvement in self-efficacy for all taught communication and safety skills (p<0.001). Significant improvement in safety knowledge scores pre–post (64% vs 83%, p<0.001). High self-reported safety behavioural intentions—85% reported they would speak up about safety concerns | 1, 2a, 2b, 3 |
| Ahmed20 2014 | Participants’ satisfaction postcourse. Patient safety knowledge (MCQs), skills (bespoke questionnaire) and safety attitudes (modified validated questionnaire) pre–post. Behavioural change via questionnaire and review of ‘quality improvement databases’ | High participant satisfaction. Significant improvement in 2 of 4 safety attitudes domains (ability to influence safety and behavioural intentions). Significant improvement in objective safety knowledge (51.1–57.6%, p<0.001). Trainees reported significantly more patient safety incidents in the 6 months following introduction of the intervention (p<0.001). 32 QI projects in various stages of implementation | 1, 2a, 2b, 3, 4a |
| Anderson18 2009 | Multimethod evaluation. Pre–post questionnaire assessing safety knowledge and perceptions of course (hopes, concerns and expectations). Additional postcourse satisfaction questionnaire and focus groups | Majority (>50%) satisfied with course; however, low scores on perceived preparation for the course. Postcourse medical student concerns emerged as being unfounded, and hopes and expectations in both the uni-professional and interprofessional groups were met. Focus group revealed consensus of added value in working interprofessionally. Significant improvement in students’ knowledge whether working uni-professionally or interprofessionally (p=0.001) | 1, 2a, 2b |
| Arora21 2012 | Participant satisfaction postcourse. Patient safety knowledge (MCQs) and safety attitudes (modified validated questionnaire) pre–post. Safety event identification and reporting 6 months postcourse via proforma | High participant satisfaction—overall satisfaction mean 4.63/5. Significant improvement in 2 of 4 safety attitude domains (attitudes to error analysis and improving safety, and ability to influence safety). Significant improvement in objective safety knowledge (45.3–70.6%, p<0.01) and subjective safety knowledge (p<0.01). Postcourse, participants recorded a higher number of observations associated with greater understanding, recognition and analysis of patient safety issues | 1, 2a, 2b, 3 |
| Cox28 2009 | Professional group differences in attitudes and skills on 6 subscales (human fallibility, disclosure of medical errors, teamwork/communication, event reporting, systems of care, curricular time spent with other professionals). Assessed by bespoke survey pre–post intervention | Significant professional group differences preintervention in all 6 subscales. Postintervention differences in 4 subscales were resolved with the exception of human fallibility (p<0.001) and curricular time spent together (p<0.001). Medical students scored significantly worse on all subscales apart from human fallibility | 2a, 2b |
| Cox31 2011 | Satisfaction via simple survey. Qualitative analysis of narratives using constant comparative method | High participant satisfaction—85% rated it as a positive learning experience. 44% self-reported improvement in safety attitudes. High participant engagement—78% of residents submitted a story and 87% attended at least 1 safety session. 79 narratives submitted by residents over 3 months. Majority of stories involved errors (86%) | 1, 2a |
| Dudas34 2011 | Participant satisfaction. Patient safety attitudes (modified items derived from Safety Attitudes Questionnaire) | High participant satisfaction—76% recommended that the session continue. Significant improvements in patient safety attitudes pre–post in 9 of 10 items (p<0.01) | 1, 2a |
| Gupta43 2014 | Participant satisfaction postcourse (survey). Self-assessment and knowledge assessment about quality and safety principles precourse using a bespoke tool | High participant satisfaction. Experiential components were felt to be of most value. Almost half (49%) of items in the knowledge assessment were answered correctly preintervention (but no postintervention comparison data were reported). 75% of participants had ongoing formal or informal roles in QI or patient safety within their current practice environment following the course (specific time postintervention unreported by authors) | 1, 2b, 4a |
| Hall32 2010 | Patient safety attitudes and self-reported safety skills (previously published tool). Comparison preintervention and 1 year postintervention and with historical control. Analysis of student-submitted reports compared with contemporaneous reports from patient safety reporting system (PSN) | At baseline, no differences in any patient safety attitudes or safety skills between intervention and control. At 1 year postcourse, intervention group expressed significantly higher comfort level in identifying the cause for an error postintervention (3.72 vs 3.27, p<0.05). No significant difference in PSN worthy reports or in blame tone between participants and PSN reporters. Significantly higher robustness of proposed solutions by participants compared with PSN reporters (3 vs 0, p<0.001) | 2a, 2b |
| Holland42 2010 | Curriculum evaluation. Objective knowledge assessed via MCQs and true/false items precourse and immediately postcourse. Reflection on learning assessment at year-end including knowledge, skills, abilities and beliefs items | High satisfaction with curriculum (mean 3.53/4). Residents perceived significant improvements in knowledge, skills, abilities, beliefs and commitment to improve quality of care (all p<0.001). Significant improvement in knowledge (19.50–23.00, p<0.05). 20 QI projects proposed, 50% at various stages of implementation | 1, 2a, 2b, 3, 4a |
| Jansma23 2010 | 11-item questionnaire exploring attitudes, intentions and behaviour towards reporting incidents (using vignettes and modified previously published tool). Assessed at baseline, immediately postcourse and 6 months postcourse | Attitudes towards incident reporting significantly improved (5 of 6 vignettes), p<0.001. Intentions towards incident reporting significantly improved between baseline and 6-month follow-up (p<0.05). No significant improvement in reporting behaviour | 2a, 3 |
| Jansma24 2010 | Satisfaction and patient safety behaviours (via semistructured interview) 3 months postintervention to assess whether action implemented and the barriers and promoters to action(s) | High participant satisfaction—mainly positive reaction by 67%. 91 action points formulated by 68 participants. 62 (90%) residents reported taking action at 3 months; 50 (55%) actions were carried out fully. Barriers to implementing actions mentioned more than twice as frequently as compared with promoters. Barriers mostly related to work pressures and rotations | 1, 3 |
| Jericho33 2010 | Attitudes towards adverse event reporting assessed preintervention and postintervention using a bespoke questionnaire (12 months). Quarterly adverse event reports submitted by residents | Significant improvement in attitudes towards reporting (no p value). Number of reports increased from 0 per quarter in the 2 years preintervention to 28 per quarter for the 7 quarters postintervention, with no sign of decay | 2a, 3 |
| Jha22 2013 | Acceptability of the intervention by participants postintervention. Preintervention and postintervention administration of the APSQ, assessing attitudes and knowledge.52 Follow-up at 6 weeks: repeat APSQ, in-depth interviews, and an online survey about success in implementing learning points | Response to patient involvement in teaching was largely positive. Mean attitude and knowledge scores on the APSQ increased postintervention compared with preintervention (no p values reported). Response rate to 6-week follow-up APSQ was poor (38%). Only 6 participants participated in follow-up in-depth interviews; 3 provided evidence of implementation of learning in practice | 1, 2a, 2b, 3 |
| Leung25 2010 | Patient safety attitudes and self-report knowledge (adapted previously published questionnaire) assessed precourse and 3 months postcourse | Participants supportive of inclusion of patient safety in curriculum and in professional examinations. Significant improvement in 8 of 15 items on patient safety attitudes. Significant improvements in all 5 items on self-reported patient safety knowledge; however mean scores still perceived as ‘fair’ or ‘poor’ | 2a, 2b |
| Miller27 2014 | Postintervention questions exploring perceptions of the intervention. Patient safety attitudes (16-item bespoke questionnaire) preintervention and postintervention | Overall positive feedback about the course content. 69% of medical students preferred taking the course individually (the remainder preferring a groupwork format). Significant improvement in all items of the survey (p<0.05) assessing patient safety attitudes among medical students | 1, 2a |
| Myung26 2012 | Participant satisfaction (method not described). Patient safety awareness (40-item bespoke questionnaire) pre–post | Student and faculty commented on repetition of some material and desire for more interactive educational methods. Significant improvement in patient safety awareness in 36 of 40 items (p<0.05) | 1, 2a |
| Paxton37 2010 | Patient safety knowledge assessed via MCQ precourse and postcourse, and again at between 1 and 12 months postcourse. Application of learning assessed on long-term follow-up. Control group compared precourse and 6 months postcourse | Significant improvement in knowledge score at short-term (29.3–73.7%, p<0.001) and long-term follow-up (49.1%, p<0.001). 57.1% said they had applied the information learned in practice. No significant difference in knowledge found in control group | 2b, 3 |
| Rodrigue30 2013 | Perceptions of experience with faculty development opportunities, performance and QI tools and training (bespoke survey). Resident participation in performance improvement, QI and patient safety programmes | Non-significant increase in number of residents that felt their training programme provided tools and training in QI. Postintervention, residents reported a non-significant increase (12.1%) in participation in departmental/institutional QI or safety projects, with faculty reporting a significant increase (38.2%, p=0.001) | 2a, 3 |
| Scott40 2011 | Satisfaction with reporting mechanism. Participant attitudes and motivation regarding reporting and intervention (bespoke survey). Percentage of all adverse event reports submitted by residents via electronic reporting system | 83% felt the system was burdensome. Monthly average number of adverse events reported by residents significantly increased by 5.5 times (6 (1.6%) to 33 (9%), p<0.001). Significant improvement in relative proportion of near-miss reports (0.3 (6%) to 9 (27%), p<0.001). Main motivators for reporting were patient wellness (87%) and financial incentive (64%) | 1, 2a, 3 |
| Shaw39 2012 | Programme satisfaction using 7-item survey postintervention and focus group to explore experiences. NPSG-knowledge improvement using MCQ test preintervention and postintervention. NPSG-compliant behaviours in a simulation scenario. Self-reported confidence in safety and quality (bespoke survey) | Spaced Education participants found cases authentic, engaging and memorable. Significantly higher proportion of Spaced Education interns responded positively to satisfaction and self-reported confidence items (4 of 7 items, p<0.05). Both online programmes significantly improved knowledge (p<0.001). No significant difference in knowledge in control group. Higher proportion of Spaced Education participants with improved NPSG-behaviours (mean 4.79/13 vs 4.17/13 in SQ group; significant for surgical participants: 5.67 Spaced Education group vs 2.33 SQ group, p<0.05) | 1, 2a, 2b, 3 |
| Slater19 2012 | Satisfaction questionnaire to evaluate online module and each workshop. Patient safety culture assessed using modified ‘Hospital Survey on Patient Safety Culture’ precourse and postcourse. Knowledge assessed using MCQs pre–post. Project outcomes using run charts. Interviews to explore experiences with TAPS | High rates of satisfaction for workshops (mean score 4.1/5), less so for online module (3.3). No change in safety culture scores for most dimensions apart from significant improvement in ‘communication/openness’ (p<0.01). Improved multiprofessional communication and teamwork reported via interview. Of the 5 participants who completed pre–post knowledge test, all but 1 improved score. 8 of 11 teams demonstrated improvements in patient safety practices/outcomes via run charts | 1, 2a, 2b, 3, 4a |
| Smith35 2012 | Satisfaction questionnaire to cohorts across the 2 years. Qualitative analysis of cases presented, interventions proposed and success of follow-through | High participant satisfaction—overall quality of QI conference mean 4.49/5. 46 interventions suggested; attempt to initiate 25 (54%) and of these 18 (72%) deemed successful: 8 led to objective permanent system-wide change and 10 resulted in subjective behavioural change | 1, 3, 4a |
| Stahl38 2011 | Participant satisfaction. Participant knowledge pre–post (24-item questionnaire based on previous studies). Participant behaviour postcourse (number of times observed and intervened in a patient safety risk) | Significantly greater satisfaction in intervention vs control group (75% vs 54%, p<0.05). Significantly greater improvement in patient safety knowledge in intervention vs control group (83% vs 75%, p<0.001). Significantly greater proportion of intervention group self-reported intervening to avoid error compared with control group (77% vs 61%, p<0.05) | 1, 2b, 3 |
| Tess44 2009 | Programme evaluation, survey of participant attitudes (bespoke survey), and participation in patient safety and QI work | High participant satisfaction including significant improvement in quantity of teaching, and overall value of clinical rotations postintervention. Significant postintervention improvement in 6 of 12 questions addressing attitudes about culture of safety and 3 of 11 items on residents’ perception of educational goals during the residency programme (all p<0.05). All participants completed an adverse event review. Significant improvement in engagement with departmental QI meeting (>66% postintervention vs 10%) | 1, 2a, 3, 4a |
| Wilson29 2012 | Course satisfaction. Evaluation based on class participation (30%), peer evaluation (15%) and group project paper and presentation (total 55%) | The attendance score for medical students was the lowest (8.59 of 10). Peer evaluation of all students was high; medical students were the ‘low outlier’ in 8 of 10 categories. Students rated assigned reading material as extremely helpful. Learners’ perceived that analysing the case studies in multidisciplinary groups gave more insight into understanding the problems and proposing solutions | 1 |
APSQ, Attitudes to Patient Safety Questionnaire; MCQ, multiple choice question; NPSG, National Patient Safety Goal; PSN, Patient Safety Network System; QI, quality improvement; SQ, safety questionnaire; STS, System Thinking Scale; TAPS, Training and Action for Patient Safety.