Article Text
PDF
Abstract
Objectives To evaluate the impact and feasibility of multisource feedback compared with traditional feedback for trauma team captains (TTCs).
Design A mixed-methods, non-randomised prospective study.
Setting A level one trauma centre in Ontario, Canada.
Participants Postgraduate medical residents in emergency medicine and general surgery participating as TTCs. Selection was based on a convenience sampling method.
Intervention Postgraduate medical residents participating as TTCs received either multisource feedback or standard feedback following trauma cases.
Main outcome measures TTCs completed questionnaires designed to measure the self-reported intention to change practice (catalytic effect), immediately following a trauma case and 3 weeks later. Secondary outcomes included measures of perceived benefit, acceptability, and feasibility from TTCs and other trauma team members.
Results Data were collected following 24 trauma team activations: TTCs from 12 activations received multisource feedback and 12 received standard feedback. The self-reported intention for practice change was not significantly different between groups initially (4.0 vs 4.0, p=0.57) and at 3 weeks (4.0 vs 3.0, p=0.25). Multisource feedback was perceived to be helpful and superior to the existing feedback process. Feasibility was identified as a challenge.
Conclusions The self-reported intention for practice change was no different for TTCs who received multisource feedback and those who received standard feedback. Multisource feedback was favourably received by trauma team members, and TTCs perceived multisource feedback as useful for their development.
- MEDICAL EDUCATION & TRAINING
- ACCIDENT & EMERGENCY MEDICINE
- TRAUMA MANAGEMENT
Data availability statement
Data are available upon reasonable request. Qualitative and quantitative data are available upon reasonable request.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
STRENGTHS AND LIMITATIONS OF THIS STUDY
This study captured multisource feedback data following trauma cases in the clinical setting.
Trauma team captains received feedback from four assessors, including a registered nurse, the supervising staff physician, another trauma team member as well as a self-assessment.
A small sample size at a single hospital may limit the generalisability of the findings.
We relied on a self-reported change in behaviour to capture the catalytic effect of multisource feedback instead of observed changes in behaviour or other potential measures of catalytic effect.
Background
The Royal College of Physicians and Surgeons of Canada has implemented a competency-based medical education (CBME) framework called Competence By Design. This model requires postgraduate medical trainees to progress through four stages with assessment based on a set of predetermined Entrustable Professional Activities (EPAs) at each stage. Emergency medicine and general surgery training programmes have three and four EPAs, respectively, related to the care of the trauma-injured patient.1 2 Within each of these EPAs, there are specific milestones that address the resident’s ability to communicate with allied healthcare providers, practise the principles of crisis resource management, support team members dealing with difficult cases, leadership qualities, etc.
The assessment of a resident trauma team captain (TTC) can be challenging. Physician supervisors are simultaneously focusing on the clinical management of the patient and may not have the cognitive capacity to make detailed observations and provide high-quality feedback. Using both physician and allied health (nurses, respiratory therapists, etc) perspectives for resident feedback may provide a more holistic assessment of TTC performance.3 4
Multisource feedback (MSF) is a workplace-based process whereby multiple team members provide feedback on both clinical and non-clinical skills to a specific individual. It has been integrated effectively into programmes of assessment within medical education5–12 and has proved to be effective in team-based work environments.8 10 In the context of trauma, MSF may improve the quality of feedback with multidisciplinary assessors bringing a unique focus and perspective to the assessment process.13 Specifically, MSF has been used to provide assessment and feedback on crisis resource management skills (leadership, communication, situational awareness, resource utilisation and problem-solving). This is a skillset that has been shown to be integral to high-quality trauma team function.14
Before MSF can be broadly recommended in the trauma context, its utility must be evaluated. In 2010, the International Conference on Medical Education outlined seven criteria for good assessment: validity or coherence, reproducibility or consistency, equivalence, feasibility, educational effect, catalytic effect and acceptability.15 16 The catalytic effect is the assessment process’s ability to create, enhance and support learning, and has been identified as a central pillar of formative assessment. As residency programmes lean on assessment as a driver of learning,17 the catalytic effect of any assessment assumes increasing importance. We sought to compare the catalytic effect of MSF with standard feedback in the assessment of postgraduate medical residents participating as TTCs. Second, we sought to describe trauma team members’ perception of MSF.
Methods
Setting and participants
This was a mixed-methods, non-randomised prospective study at a single academic, level one trauma centre in Ontario, Canada. The trauma team is activated approximately 400 times per year in cases of major trauma and is composed of a postgraduate medical resident TTC, a faculty trauma team leader (TTL), two registered nurses (RNs), a respiratory therapist and a medical resident in anaesthesiology, general surgery, or orthopaedic surgery training programmes. The TTC was in at least their third year of an emergency medicine (FRCPC or CCFP-EM) or general surgery (FRCPC) training programme. They were responsible for leading the resuscitation of trauma patients under the supervision of the TTL. The TTL was an attending physician from either critical care, emergency medicine or general surgery.
Data collection
Data were collected using convenience sampling. When research personnel were available to attend trauma activations (typically between 07:00 and 12:00, 7 days of the week), the TTC was assigned to the MSF group. If no research personnel were available, the TTC was assigned to the standard feedback group. TTCs could participate in both groups, but only once in each. Based on previous studies,18 we planned to collect data following 20–30 trauma activations (10–15 in either group).
The main researchers (LA and TC) were medical student and staff emergency medicine physician, respectively, at the hospital where this study took place. TC had an active clinical working relationship with the trauma team members through his role as emergency medicine clinician. TC also actively worked with the TTCs participating in the study through his role as trauma team leader. LA conducted the majority of the in-person questionnaires, with some being conducted by TC. All online questionnaires were conducted by LA.
MSF group
Following a trauma activation, TTCs in the MSF group completed a self-assessment form and received written feedback from the trauma team, RN, TTL and a third member of the team (a resident from anaesthesiology, general surgery or orthopaedic surgery) in person, using the MSF tool described below. TTCs completed the catalytic effect questionnaire 1 (Q1) immediately after reviewing the written feedback in person and an online questionnaire 2 (Q2) 21 days later.
Standard feedback group
TTCs in the standard feedback group received feedback according to standard practice. Standard practice at our institution encourages feedback from the TTL after each trauma case. Feedback can be verbal or written (either paper based or online). Sometimes feedback is not given at all. TTCs in the standard feedback group were contacted within 24 hours following a trauma activation. If TTCs in this group received feedback, they completed the catalytic effect Q1 within 24 hours of the trauma and Q2 21 days later. These questionnaires were completed online.
MSF tool
An MSF tool for each team member was developed for this study (online supplemental table A1) based on a literature review of MSF in the healthcare setting (ie, MSF/PAR tool, trauma feedback surveys, FACT tool, resuscitation assessment tools).5 19–21 It underwent an iterative design process with input from a group of healthcare professionals (a TTC, TTL, anaesthesiologist, RN) who were not directly involved with this study but did have trauma team experience. The tool included an overall entrustment score (online supplemental table A2),22 an assessment of specific domains (communication, interprofessionalism, collaboration/teamwork, leadership/professionalism and medical expert) and a section for feedback on the MSF process itself. The domains were partitioned among the team providing feedback (online supplemental table A3). The decision to partition the feedback domains was based on prior work suggesting that this may improve MSF’s efficiency and reduce the assessment burden placed on any single assessor group.13 For example, RNs provided feedback on communication and teamwork, whereas TTLs focused on leadership and medical expertise.
Supplemental material
Primary outcome: the catalytic effect of MSF
Two questionnaires, Q1 and Q2, sought to capture the catalytic effect of MSF (online supplemental table A4). They were developed by study authors (LA, TC) based on a literature review23–25 with input from the same group of healthcare professionals who assisted with the MSF tool development. TTCs completed Q1 on paper immediately after reviewing the written MSF. They completed Q2 21 days later online (Qualtrics, Provo, Utah, USA). We evaluated the self-reported intention to change practice using a 5-point Likert scale on Q1 and Q2 as a proxy measure for the catalytic effect.26
A post-hoc subgroup analysis was completed to compare the catalytic effect of any TTC who underwent both MSF and standard feedback with those who only received MSF or standard feedback.
Secondary outcomes: acceptability, feasibility and qualitative comments
The acceptability and feasibility of MSF were measured on the MSF tool distributed to team members and Q1 distributed to the TTCs. Each question item designed to measure the acceptability or feasibility was evaluated using a 5-point Likert scale.
In addition to the 5-point Likert scale items, we encouraged narrative comments from all team members regarding the use of MSF in the trauma context. This was both on the MSF tools distributed to all team members and on Q1 and Q2 distributed to the TTCs.
Patient and public involvement
It was not appropriate or possible to involve patients or the public in the design, or conduct, or reporting, or dissemination plans of our research.
Data analysis
Data were transcribed into Microsoft Excel (V.16) or Microsoft Word (V.16) and anonymised for further analysis. Quantitative Likert scale data were examined for normality using the Shapiro-Wilk test. Data deviated from normal distribution; therefore, the data were analysed in a non-parametric fashion using the Mann-Whitney U test for significance (significance met if p<0.05). Data variation is depicted in IQR. Statistics and graphics were completed using GraphPad Prism (V.9.0).
The narrative comments provided by the team about MSF in trauma were regarded as qualitative data and were thematically analysed.18 27–29 All narrative comments were coded independently by two authors (LA and TC). The preliminary codes were then cross-referenced to identify patterns across the two participant groups. The smallest unit of analysis (codes) was organised into subthemes and themes.29 Following nine trauma cases, authors (LA and TC) compared and discussed the preliminary themes. Following this, data were collected and analysed from three further cases to determine if saturation of themes was achieved following 12 cases. Thematic saturation of the qualitative data was achieved after 12 cases for both the MSF group and the routine feedback group.
Results
Data were collected from 19 TTCs following 24 trauma activations between July 2019 and April 2020. Twelve received MSF and 12 received standard feedback. Nineteen different TTCs participated, with five participating once in both groups (table 1). All 12 TTCs in the MSF group received feedback, whereas only 7 of the 12 TTCs (60%) in the standard feedback group reported receiving any feedback. Of the five TTCs who participated in both the MSF group and the standard feedback group, three received feedback in the standard group and two did not receive any feedback.
Demographics of TTCs
Catalytic effect of MSF
The self-reported intention for practice change was measured immediately on Q1 using a 5-point Likert score with the item ‘The feedback will change my practice/behaviour in future trauma cases’. There was no statistical difference (p=0.57) between standard feedback and MSF groups, with a median score of 4.0 (IQR 3.0–5.0, n=11) in the MSF group and 4.0 in the control group (IQR 4.0–5.0, n=7) (figure 1). The catalytic effect measured on Q2, administered 21 days following the initial feedback, was measured with five different items. Similarly, there was no statistical difference (p=0.25) with a median of 4.0 (IQR=3.0–4.0, n=54) in the MSF group and 3.0 (IQR=3.0–4.0, n=41) in the standard feedback group (figure 1).
Catalytic effect of MSF compared with standard feedback as measured by the respective items (IA–E) both immediately after receiving feedback (Q1) and 3 weeks later (Q2). No significant difference was found between MSF and standard feedback for any catalytic effect question. Data analysed using Mann-Whitney U test and presented as median Likert score (IQR). MSF, multisource feedback; ns, non-significance; TTC, trauma team captain.
Acceptability and feasibility of MSF
TTC and team member perceptions regarding the acceptability and feasibility of MSF in trauma are presented in figure 2. MSF was perceived to be acceptable. The item ‘MSF will improve the performance of the TTC’ received a median Likert score of 4.0 (IQR=4.0–5.0, n=11) from TTCs and 4.0 (IQR=4.0–5.0, n=34) from all other team members. The item ‘MSF provides better feedback than our current model’ received a median Likert score of 4.0 (IQR=4.0–5.0, n=9) from TTCs and 4.5 (IQR=4.0–5.0, n=28) from all other team members. There was no statistical difference found when comparing the median Likert scores of TTCs with other trauma team members.
Quantitative responses regarding acceptability and feasibility of multisource feedback (MSF) in trauma from both trauma team members versus TTCs. (A) Acceptability was measured by items A1 and A2. (B) Feasibility was measured by item F1. Data analysed using Mann-Whitney U test and presented as median Likert score (IQR). ns, non-significance; TTC, trauma team captain.
The item ‘MSF is a feasible method of giving feedback in trauma teams’ was designed to discern the TTCs’ and team members’ thoughts on the feasibility of MSF in trauma. The median Likert scale responses for TTCs and other team members, respectively, was 4.0 (IQR 3.0–4.0, n=9) and 4.0 (4.0–5.0, n=35) (figure 2). Thus, trauma team members generally regarded MSF as feasible, but gave higher scores for acceptability than feasibility.
Two other items related to the qualification of the team members to give the TTC feedback and the consistency of the feedback were included on Q1 given to TTCs. Within the group of TTCs who received MSF, they agreed that the ‘individuals giving them feedback were qualified to do so’ (Likert score of 4.0, IQR=4.0–5.0, n=11) and that the ‘feedback from different team members was consistent’ (Likert score of 4.0, IQR=4.0–5.0, n=11).
Narrative comments
Within the group who participated in MSF, either as a team member or TTC, analysis of the narrative comments on the use of MSF revealed three themes and six subthemes (table 2). All themes and subthemes were identified upon analysis of the first nine cases. An additional three cases in each group were analysed and no further themes or subthemes were identified (online supplemental table A5).
Themes and subthemes identified in the narrative comments on MSF from the TTCs and other team members
‘Utility’, ‘Logistics’ and ‘Applicability’ were the three themes identified by trauma team members (including the TTC) participating in the MSF process (table 2). ‘Utility’ was the most common theme comprising 25 out of 43 total comments. Subthemes of ‘helpful’ and ‘unknown’ were used to code the respective comments about the usefulness of MSF. Most of the comments found MSF to be helpful (20 out of 25), with some comments suggesting it may not be better than feedback from the TTL or may be burdensome in the new era of CBME (5 out of 25).
The second theme, ‘logistics’, comprised 13 of 43 total comments. This theme included comments about the ‘feasibility’ of MSF in trauma. Most comments suggested that the feasibility of administering MSF in a busy trauma case is difficult, especially when consulting services have work to resume in other areas of the hospital. ‘Applicability’ was the third theme with 5 out of 43 comments. Subthemes included comments about the ‘suitability’ of MSF to trauma, as well as its ‘validity’ and ‘generalisability’. There was dissent about the applicability of MSF to trauma. Some TTCs and team members stated that MSF is ‘well suited to trauma’. The qualitative comments gathered are specific to the helpfulness of MSF in trauma, as well as the feasibility of implementation and how well MSF applies to trauma.
Subgroup analysis
The five TTCs who participated twice all underwent feedback via MSF first, then subsequently underwent standard feedback. When they participated in the standard feedback group, only three of the five TTCs received feedback from the TTL.
For the catalytic effect, the results of this subgroup were similar to the overall group. On Q1 administered immediately, the item ‘The feedback will change my practice/behaviour in future trauma cases’ received a median score of 4.0 (IQR 3.5–5.0, n=5) in the MSF group and 4.0 in the control group (IQR 3.0–5.0, n=3). The five items on Q2, administered 21 days later, received a median of 4.0 (IQR=3.0–4.0, n=25) in the MSF group and 3.0 (IQR=2.0–4.0, n=15) in the standard feedback group.
Discussion
We found no difference in TTCs’ self-reported intention to change practice after MSF compared with traditional feedback. The majority of TTCs and team members did find MSF helpful in the trauma context. The lack of difference in measured intention to change practice in the current study is in contrast with previous work that has found a catalytic effect of MSF.5 24 30 31 This is surprising as MSF is a resource-intensive process, and qualitative comments from TTCs generally regarded MSF as being helpful. Reasons for this difference may include the following: (1) this study was not large enough to detect a difference (ie, type 2 error), (2) the MSF tool used was not valid, (3) the questionnaires did not adequately measure the catalytic effect, (4) the MSF process was new and not inherently part of the culture at our institution, and (5) there could be no difference between MSF and standard feedback in the trauma context.
Although not a study outcome, we did observe a difference in the durability of the catalytic effect between the two groups. TTCs in the MSF group had less of a decay in the catalytic effect between the initial measurement and at 3 weeks compared with TTCs in the standard feedback (figure 1). This finding of a more sustained catalytic effect in the MSF group would be an interesting avenue for future work.
MSF has several potential benefits in the trauma context. Most TTCs agreed that the interprofessional team members were qualified to provide feedback and the content of the feedback was consistent across team members. In our study, MSF was provided within 1–2 hours of the initial patient assessment/management. This has implications on how well the feedback is received; timely feedback that is aligned across multiple sources who are respected encourages acceptance and change.32 With respect to feedback content, MSF has been shown to be effective for providing feedback on both clinical and non-clinical skills such as critical resource management and communication skills.13 These skills are required for high-functioning teams such as a trauma team.14 33
Trauma team members commented on the utility and logistics of MSF. MSF was viewed favourably by TTCs receiving the feedback as well as the trauma team members providing it. It was perceived to be superior to the existing feedback process and helpful to TTC development. Our existing TTC feedback process involves verbal or online feedback from the TTL. Most participants perceived MSF to be superior and helpful for the TTC, although some felt that it adds limited or unknown value to the existing feedback process.
We identified the feasibility of MSF as a potential challenge to its implementation and use in trauma. Feasibility is a criterion for good assessment,15 and if MSF is to become standard practice, this aspect must be addressed explicitly. In this study, MSF was embedded into the trauma workflow; participants were asked to complete the MSF questionnaires after the initial resuscitation of the patient and before resuming other duties. Within this study, timely feedback was accomplished after every trauma targeted but may not be possible after many trauma cases, with multiple competing demands of the emergency department and hospital. Feasibility of MSF has been achieved in other studies; however, there is uncertainty if full implementation of MSF into a busy department or trauma bay is feasible without research personnel guiding the surveys.9 34 Furthermore, the use of paper-based questionnaires, which require a significant amount of time to ensure completion9 and lack of time to reflect on the feedback given,35 has previously been identified as a logistical challenge of MSF and may apply to this study.
Another topic identified in this study is assessment of burden and fatigue. Recently, CBME and EPAs for medical trainee feedback has been adopted in Canada.36 CBME poses a challenge for MSF due to resident and staff physicians feeling overwhelmed by the burden of increased assessment requirements.37 This was reflected in some study participants sharing that mandatory feedback is becoming more of a burden, and some participants want less formalisation of feedback.
Moving forward, MSF in trauma may further enhance practice change if it was better integrated into the trauma environment, allowing for more feedback to be collected from multiple stakeholders, over multiple trauma cases. MSF could potentially be integrated with CBME such that online EPAs are automatically triggered to staff physicians, RNs and senior resident colleagues when a TTC resident doctor has a trauma shift. Better integration of MSF for trauma into CBME would allow for MSF from multiple traumas over a series of weeks/months. TTCs could then be polled on their intention for practice change, a surrogate measurement for catalytic effect, once MSF from multiple traumas is collated.
This study has several limitations. First, we relied on a self-reported change in behaviour instead of observed changes in behaviour or other potential measures of catalytic effect. Second, TC is an attending trauma team leader and collected study data following a few of the trauma activations. LA collected much of the data and completed the data analysis to help mitigate bias from TC as a faculty member. Third, this study had a small sample size and was conducted at a single hospital. Fourth, we acknowledge that the findings may lack transferability to other contexts and other healthcare providers. However, the researchers have provided a rich description of the study context and procedures to aid with increasing the likelihood of transferability. Future research could explore transferability by using the same data collection procedures at a different trauma centre. The strengths of this study include capturing data from real, non-simulated trauma cases in a timely manner.
Future work should focus on: (1) how best to design and deliver MSF to ensure integration into current workflow and enhance feasibility; (2) the collection of MSF over multiple traumas with multiple assessors and (3) exploring qualitatively why we did not find a difference in catalytic effect. Better integration of MSF into the trauma workflow and collecting more assessments longitudinally may demonstrate positive changes in future TTC practice.
Conclusions
This pilot study found no difference in the self-reported catalytic effect between residents who received MSF and those who received traditional feedback. MSF was viewed to be acceptable and enhanced current feedback practices. Identified challenges relate to the feasibility and efficiency of MSF within the trauma context.
Data availability statement
Data are available upon reasonable request. Qualitative and quantitative data are available upon reasonable request.
Ethics statements
Patient consent for publication
Ethics approval
This study involves human participants and ethics approval was granted through the Queen’s University Health Sciences and Affiliated Teaching Hospitals Research Ethics Board (File #6026756). Participants gave informed consent to participate in the study before taking part.
Acknowledgments
We would like to thank Christina Gowlett, Zachary Warren, Dave Commerford, Riley Danford and Dr Chris Evans for their contribution to developing and editing the multisource feedback tool. We would also like to thank Vlad Latiu for help in data collection.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Contributors LA was involved in the study design, survey creation, data collection, data analysis, statistical analysis and manuscript writing. AKH was involved in the data analysis as well as the manuscript writing and revisions. HB was involved in the data analysis as well as the manuscript writing and revisions. TC was involved in study design, survey creation, data analysis and manuscript writing. TC accepts full responsibility for the conduct of the study, had access to the data, and controlled the decision to publish. LA affirms that the manuscript is an honest, accurate and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned (and, if relevant, registered) have been explained.
Funding LA was awarded the Summer Studentship from the Department of Emergency Medicine, Queen's University.
Competing interests AKH reports receiving payment for working with the Royal College of Physicians and Surgeons of Canada on the implementation and evaluation of Competency By Design in Canada. No other conflicts of interest to declare.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.