Article Text
PDF
Abstract
Objective The purpose of this study was to determine the impact of a 1-day evidence-based medicine (EBM) workshop on physician attitudes and behaviours around teaching and practicing EBM.
Design A mixed methods study using a before/after cohort.
Setting A medical school delivering continuing professional development to 1250 clinical faculty over a large geographic area in Canada.
Participants 105 physician clinical faculty members.
Intervention A 1-day workshop presented at 11 different sites over an 18-month period focusing on EBM skills for teaching and clinical practice.
Outcome measures (1) A quantitative survey administered immediately before and after the workshop, and 3–6 months later, to assess the hypothesis that comfort with teaching and practising EBM can be improved.
(2) A qualitative survey of the expectations for, and impact of the workshop on, participant behaviours and attitudes using a combination of pre, post and 3 to 6-month follow-up questionnaires, and telephone interviews completed 10–14 months after the workshop.
Results Physician comfort with their EBM clinical skills improved on average by 0.93 points on a 5-point Likert scale, and comfort with EBM teaching skills by 0.97 points (p values 0.001). Most of this improvement was sustained 3–6 months later. Three to fourteen months after the workshop, half of responding participants reported that they were using the Population Intervention Comparator Outcome (PICO) methodology of question framing for teaching, clinical practice or both.
Conclusions Comfort in teaching and practicing EBM can be improved by a 1-day workshop, with most of this improvement sustained 3–6 months later. PICO question framing can be learnt at a 1-day workshop, and is associated with a self-reported change in clinical and teaching practice 3–14 months later. This represents both level 2 (attitudes) and level 3 (behaviours) change using the Kirkpatrick model of evaluation.
- medical education and training
- primary care
- qualitative research
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Strengths and limitations of this study
The primary quantitative outcome measure was a change in comfort level, which has not been validated and may not reflect a clinically important outcome.
This was a before-after design and the participants who responded to surveys and interviews 3–14 months later might have differed from the non-responders.
A single, novice interviewer was used for the qualitative interviews.
Only 14 of 60 eligible participants agreed to be interviewed, and 51 of 105 completed reflective surveys.
Changes in behaviours were self-reported, and not objectively measured.
A strength was that conclusions were obtained from three sources (interviews, specific survey questions and free-text survey questions) and were consistent across the three sources.
Introduction
Although evidence-based medicine (EBM) faculty development workshops are common, much remains to be learnt about the lasting impact on physicians’ attitudes towards EBM and changes in practice or teaching behaviours following these interventions.1–7
Kirkpatrick’s model of evaluation8 distinguishes among four levels of outcomes after an educational intervention, ranging from the most basic, the learner reaction, to knowledge acquisition or changes in attitudes, changes in behaviours and finally changes in patient or organisational outcomes (see figure 1). Knowledge generally can be separated into conceptual, instrumental and persuasive knowledge.9
Kirkpatrick’s model of evaluation.
Much research on EBM faculty development has focused on Kirkpatrick level 2 outcomes (changes in knowledge or attitude).7 10–17
Most studies that examined level 3 or 4 outcomes (changes in behaviour or clinical outcomes) after an EBM workshop failed to find an important effect.1 7 10 12 13 16
Previous research has shown that a half-day workshop focusing on critical appraisal failed to lead to changes in attitudes, evidence-seeking behaviour, confidence or other areas of critical skills ability.13 More extended interventions had improved attitudes and knowledge of EBM skills,6 7 10–12 14 16 but had mixed impact on physicians’ ability to use resources at the point of care.7 12 14 A Cochrane review showed that question framing skills could be improved in the short to medium term, but had eroded at 1 year.2
Experienced teachers in the EBM field have suggested that workshops are more likely to lead to change if they are interactive, delivered in small groups, involve role play and simulation of real-life situations, use a mentorship model and have a high educator to learner ratio.18 One educational framework that has been suggested is the Four Component Instructional Design Model (4C/ID), which comprised employing learning tasks, delivering supportive information, delivering procedural information and then repetitive part-task practice to enhance transfer of the learnt skills to clinical practice.19
Methods
This study was developed in response to informal feedback from clinical faculty at the Northern Ontario School of Medicine (NOSM) in Canada. Some reported that they did not feel comfortable with their EBM skills, and wished to have faculty development to improve them.
Given that NOSM’s 1250 clinical faculty members are distributed among 90 teaching communities spread over 800 000 km2 in Northern Canada, delivering EBM faculty development is a significant challenge.
Intervention
A 1-day EBM workshop was delivered at 11 widely dispersed NOSM teaching sites to a total of 105 participating physicians between October 2014 and May 2016. Individual workshops consisted of 5–20 participants, and one EBM tutor acted as lead facilitator for all 11 workshops. If there were more than 10 registrants, a second facilitator was recruited. The lead facilitator and two of the other facilitators are clinicians and the other facilitator is an epidemiologist with a medical background. All are experienced EBM tutors. The workshops were offered at no or minimal cost to participants.
The target audience was physicians or surgeons in clinical practice, but some nurse practitioners, physiotherapists, medical learners and medical librarians also attended. Most of the participants were family physicians. Only practising physicians and surgeons participated in the assessment phase.
A needs assessment was performed, which included a literature review, discussion with EBM leaders within and beyond the institution, emailed surveys to local education groups, and telephone and personal discussions with community-based medical leaders. A proposal was then developed to deliver a 1-day travelling EBM workshop, and to assess its impact using a mixed methods study. Approval was obtained from the Laurentian University Research Ethics Board (File 2014-06-18). Medical communities in Northern Ontario were polled for interest in hosting an EBM workshop, and the facilitators went to interested communities to deliver a 1-day stand-alone session.
A semistandardised agenda was customised to meet local needs and interests, and was revised in response to ongoing feedback. Workshops lasted from 6 to 7.5 hours depending upon local request. All participants were asked to bring a smartphone, tablet or laptop, and internet access was provided.
Each workshop began with a presentation on the ‘5 A’s’ of EBM (ask, acquire, appraise, apply, assess), and a discussion of some of the benefits and limitations of EBM. Participants were introduced to the Population Intervention Comparator Outcome (PICO) method of question framing, and the groups worked through several clinical scenarios to practise applying this, focusing on using PICO to develop key search concepts, but also to define what outcomes were or were not important to participants and their patients.
Next, at all workshops the groups were introduced to a variety of databases, applications and other resources available to them. The groups explored these together in real time and discussed strategies for when and how to use each.
Although critical appraisal skills were not the main focus of the workshops, a semididactic presentation was given on common methods of manipulating the presentation of data to make results appear better than they are.20 An exercise in critical appraisal was available if participants were interested, and all groups were introduced to a tool for performing rapid critical appraisal.21 Depending on group interest, size and time, other presentations were delivered on cognitive error and/or basic statistics and methodology. Sixty to ninety minutes was spent on teaching EBM in clinical settings, using a combination of a didactic presentation and participant sharing of experiences, challenges and barriers.
Participants then applied these resources and skills to perform their own quick searches based on clinical scenarios supplied by facilitators and by participants. Application of these skills to real-time searches for clinical scenarios accounted for about one-third of the time in each workshop. After each search exercise, the group would meet to discuss barriers, successes and possible problem-solving strategies.
Assessment/instruments
A mixed methods approach was used to explore the research question that physicians’ comfort in teaching and practising EBM could be improved by a 1-day interactive workshop that focused on question framing and bedside information management. Using Cresswell and Plano Clark’s classification of mixed methods research,22 we applied a sequential design, beginning with quantitative data collected using survey methodology to examine changes in comfort levels associated with workshop participation. We then followed up with interviews and free-text survey responses to explore the strategies and mechanisms that lead to changes in physician’s comfort levels in more detail, and to explore changes in behaviours after the workshop. Participants were asked as a secondary question what they were hoping to learn from the workshop.
Quantitative instruments
To test our research hypothesis, participants answered four questions on a 5-point Likert scale immediately before and after the workshop. Three to six months later, they were contacted by email, and asked to complete a follow-up survey. One email reminder was sent about 2 weeks later. Statistical significance was measured using the Pearson χ2 test (SPSS V.20, IBM).
Part-way through the process, as a result of a trend noted in the open-ended responses, questions were added to the reflective survey about whether participants had been using PICO question framing before the workshop, and whether they had used PICO in the previous month for either clinical or teaching purposes. Permission for this change was obtained from the Laurentian University Research Ethics Board.
Qualitative instruments
In the preworkshop survey, participants were asked to give free-text answers to a question about what they were hoping to gain from the workshop and to answer a question about their prior EBM training. In the survey immediately after the workshop, in addition to the questions about comfort levels, participants were asked what other topics they would like to see in future workshops, and suggestions for other modalities to deliver EBM faculty development (for programme evaluation purposes). In the 3 to 6-month follow-up survey, they were also asked open-ended questions about what they were doing differently in practice and in teaching as a result of the workshop.
All participants attending any of the first four workshops (60 participants) were asked to participate in a telephone interview 10–14 months after the workshop through an email invitation, followed by a telephone request. Participants from the later workshops were not asked to participate in interviews due to a different funding structure for the last seven workshops. A postgraduate resident physician without prior experience with qualitative research completed the interviews. This approach was chosen to meet funding expectations that the project would introduce physicians-in-training to the research process. The interviewer prepared by completing readings on qualitative interviewing and by participating in pretesting of the interview questions using mock interviews. Interviews were conducted by phone at the respondents’ offices, with no other observers present or listening, and were audio-recorded and transcribed. No repeat interviews were performed, and transcriptions were not returned to the participants for comments or corrections. Data saturation was not considered, as the goal was to offer participation to all the eligible attendees. Participants received a stipend of $C108 to compensate for their time. Those being interviewed did not generally know the interviewer, and no field notes were made. The interviewer was not involved in the qualitative analysis. The questionnaire used for the interviews is found in online supplementary appendix 1. Some questions were included for local faculty development programme evaluation and were not analysed for the purpose of this paper.
The interview transcriptions were coded using thematic analysis by two experienced qualitative researchers. We identified participants’ perspectives on the themes of keeping up to date, clinical application and role modelling and teaching EBM to contextualise the quantitative data. Our analysis also allowed for emergent coding in addition to the themes identified prior to the study. The analysis was conducted using NVivo V.9 software (QSR International). Discrepancies in coding were resolved in meetings between the two qualitative researchers.
Free-text responses to the surveys were collected by the lead author and separated by themes.
Results
One hundred and five participants completed a preworkshop survey. Ninety-five completed the immediate postworkshop survey, and 51 completed a follow-up survey 3–6 months later.
Among those who completed the preworkshop survey, 72 (69%) reported that they had some prior EBM training.
Responses to the four survey questions before the workshop, immediately after and 3–6 months later are seen in figure 2. Questions are found in table 1.
For each question there was an improvement in comfort of 0.74–1.11 points on a 5-point Likert scale, and most of this increase was sustained 3–6 months later. All before/after workshops and before/3 to 6-month comparisons were significant at a p value of 0.001.
Instrument for measuring primary hypothesis
Free-text survey findings
Prior to the workshop, the most commonly reported goals were to learn more about how to do effective, quick, real-time searches, to learn how to teach EBM in the clinical setting and to discover more information resources. Only 6 of 105 participants had a goal that could be broadly classified as learning how to be better at critical appraisal or understanding methodology (‘understanding research journals better; feel more comfortable to read and understand studies; be more comfortable with concepts of EBM, eg, number needed to treat; how to effectively survey papers without bogging down in details; ability to rapidly scan/assess studies for importance/applicability; more familiar with analyzing studies’).
Three to twelve months after the workshop the most useful components were reported to have been:
teaching tips
PICO question framing
being introduced to new resources and apps
tips on quick search strategies.
Multiple respondents reported that they were using some of the resources introduced at the workshop in their clinical practices.
The usage of PICO methodology for question framing was examined in two ways in the surveys. After the first 20 follow-up surveys were received, it was noted that 6 of the 20 had reported without prompting that they were using PICO question framing for teaching or for clinical practice after the workshop. New questions were added to the follow-up survey, specifically asking whether participants had been using PICO before the workshop, and whether they had used it in the month prior to the follow-up survey. Twenty out of 30 who responded to the PICO questions reported that they had used PICO for teaching, clinical practice or both in the prior month, while only one had been using it before the workshop.
Question framing had not been a perceived need among participants prior to the workshop.
Interview findings
Of 60 participants who were asked, 14 agreed to be interviewed. Thematic saturation was reached within the 14 interviews.
Gaining access to the most relevant resources for clinical practice
Many participants stressed that they hoped that EBM training would help them to gain access to the most recent and relevant resources for their clinical work. Related reasons included learning how to better integrate EBM into clinical practice, to improve patient care and to find and interpret information more efficiently.
‘Well what I was hoping to do is I was hoping to actually learn how to access more information for giving me the best guidelines for evidence based medicine.’ (A 003-1)
Common resources used after the workshop
Although some participants stated they had not changed the way they access information as a result of the workshop, there were recurring statements from participants that they are now accessing a wider variety of resources.
‘…the workshop really highlighted the TRIP Database which I found is actually a really useful one that I didn’t really know about before.’ (D 003)
‘I have occasionally used TRIP and I’ve occasionally used the NNT site which I wasn’t using before.’ (B 002)
‘So I still tend to use UpToDate as my go-to because it’s easy to access, I can remember my password, it’s generally relevant, but that has not changed. […] If there’s other questions I need to do afterhours or to sort of flush (sic) out something, then I do use BMJ (Evidence Updates) that I use – so yes, I have kind of expanded my finesse at looking at different issues.’ (A 001)
Clinical application of EBM
Supporting patient-centred care
Some participants commented that they are now more cognisant of incorporating EBM with patient-centred care.
‘I'm probably talking about evidence based medicine with patients a bit more, directly with them to help educate them to make better decisions as well.’ (A 002)
‘It just reinforces me taking the time when I see that unique case that I’m not as familiar with, to make sure that I take the time to do it, bring the patient back in a timely fashion and then talk to them about it. It certainly doesn’t make me afraid to say I don’t know all the answers to it, but I’m going to find them out.’ (D 001)
Assessing one’s current practice
Others felt that the workshop was helpful in that it provided reassurance as to how they are currently using EBM in their practice and teaching approaches.
‘Well and the fact that it reassured me that I am presently practicing in my practice using evidence-based medicine as much as I can where that information is available, and happy to make sure that I look things up and do it in front of the patient if it’s necessary to.’ (D 001)
Mindfully incorporating EBM
While some participants stated that their perceptions of EBM did not change since the workshop, others commented that they are now more mindful of incorporating EBM into their practice and are eager to continue their learning.
‘I think I’m more cognizant of numbers and trying to integrate it more into my practice, but also to use it to educate patients to make the decisions for their healthcare.’ (A 002)
‘It probably consolidated the whole notion that there is clinical applicability and my job is to figure out how to apply it to the patient in front of me.’ (A 001)
‘I suppose on some level it’s like other areas of medicine that the more you learn the more you realize you don’t know. And so I have on that level been sufficiently engaged that I’ve signed up for the weeklong McMaster course, and so have some of my colleagues. So it’s helpful to stimulate my further thirst for a greater understanding, but also to just put in enough time to really consolidate my learning that way.’ (D 002)
Role modelling and teaching EBM
Strengthening teaching capacity of EBM
The most common reasons mentioned by participants for attending the EBM workshop were to learn and/or strengthen teaching approaches of EBM
‘…it’s one thing to use EBM, it’s another thing to teach it. So to get ideas on how to teach somebody how to do that because it’s sort of something that, you know, you learn by doing but if we can help people along. So that’s sort of what prompted me to go.’ (B 004)
Tips for teaching
‘Well, the sites that the fellows used and doing the PICO searches I found to be the most useful thing. And I was able to translate that to some of my students and get them doing PICO searches.’ (B 005)
‘I’m also having more conversations with learners about what tools they’re using for their searches and trying to broaden their choices as well.’ (D 002)
‘I think with the learners I push them to formulate the questions and do the data gathering and bring it back and we review it. So yeah, it’s definitely something that I encourage when I have learners and, you know, I would have them doing the PICO questions, for example, on things that I should —wouldn’t have done for myself necessarily, but because they’re there I like to try to find something for them to exercise that tool as often as possible.’ (D 003)
Evaluating published studies
Some participants mentioned the discussion around effectively appraising articles was also very useful for them in providing guidance on how to evaluate published studies.
‘ …there was a lot of good dialogue about how to evaluate if the study, the quality of the study, so, a lot of discussion around that, which I found very helpful.’ (C 002)
Population Intervention Comparator Outcome
Eight out of 14 of those interviewed stated that they were now using PICO for question framing, while only one reported that they were using it before the workshop. Some of the specific comments from interviews on the application of PICO included:
‘Yes I do [use PICO] and that’s totally new, I wasn’t using the PICO system before at all.’ (D 002)
‘…that's the way I often will frame a question on the TRIP Database. And it usually leads me to articles and then reviewing the articles will help direct my therapy of the patient.’ (C 001)
“That’s the other thing I learned from the workshop was how to use [PICO]. […] I think that’s a good system when you know how to use it. […] Like I didn’t – I never even heard of it before the workshop.’ (B 003)
‘I had a resident at that time so that spurred me to assign a lot more PICO questions to try and answer critical questions.’ (D 003)
Discussion
We set out to determine whether a 1-day travelling faculty development workshop focusing on EBM skills could lead to changes in physicians’ comfort levels or behaviours related to the practice or teaching of EBM.
Several findings emerged from this mixed methods study. Clinical faculty members reported in the preworkshop surveys that they wanted help with teaching practical EBM, and with real-time bedside searches, rather than with critical appraisal or statistical skills.
Participants’ comfort levels with both teaching and practising EBM skills did improve, and most of this improvement was sustained 3–12 months later. This represented a Kirkpatrick level 2 (attitude) change.
PICO methodology for question framing was a popular tool that most participants had not been familiar with before the workshop. More than half reported that they were using PICO 3–14 months after the workshop. This represented a Kirkpatrick level 3 (behaviour) change. This information was triangulated from three separate data sets, strengthening the conclusion that this is a useful and teachable skill. Participants had not perceived the need for faculty development in framing questions before the workshop. The medium to longer term use of PICO question framing was a new finding.
While previous studies of the impact of EBM workshops have been able to demonstrate changes in knowledge and attitudes,7 10–16 they have been less successful in demonstrating changes in behaviours.1 2 7 10 12 13 16 We have demonstrated that certain behaviours did change after a brief intervention, specifically participants reported that they were using more and different information resources after the workshop, and that they were using PICO methodology for question framing. There are several possible explanations for these findings. The focus of this workshop was more on instrumental knowledge such as question framing and real-time information management and resources, reinforced with part-task and whole-task exercises, rather than on conceptual use of knowledge such as critical appraisal, and these skills may be more amenable to uptake from a 1-day intervention. A study from 200712 failed to show that participants were using the resources that they had been taught, although a 2016 study14 found that participants were using more secondary resources. Improvements in resources and the technologies that support them may have led to our finding and that from Thor et al of improved uptake.
There are several important limitations to this study. It may be difficult to duplicate a specific workshop in another setting. As well, self-reported comfort with practising and teaching EBM is not a validated measure, and may not reflect a clinically important outcome. However, given that this study was instigated due to a reported lack of comfort with these skills among clinical faculty, it does appear that the workshops achieved this goal. The study design was an uncontrolled before/after cohort. About half of the attendees responded to the follow-up survey and one quarter of eligible participants agreed to be interviewed. Those who respond to a follow-up survey, and those who volunteer to do an interview a year later may not be representative of all those who attended the workshop. In particular, it is quite possible that participants who did not change behaviour after an educational intervention would be less likely to complete a follow-up survey or agree to be interviewed. Changes in behaviours and attitudes were self-reported, and may not represent true changes in practice. The interviewer had no previous experience with qualitative research or interviewing.
The most important strength of the study was that findings were consistent and were able to be triangulated using the mixed methodology.
Conclusions
Comfort among clinical faculty in practising and teaching EBM can be improved with a 1-day workshop, and most of this improvement is sustained at 3–6 months.
The three areas that participants identified wanting help in were in improving EBM bedside teaching skills, learning new resources for searching and for keeping up to date, and improving bedside searching skills, There was little interest expressed in furthering critical appraisal or statistical skills.
Three to twelve months after a 1-day EBM workshop, half of respondents reported using PICO question framing for clinical work, teaching or both.
Acknowledgments
We wish to acknowledge the contributions of John Hogenbirk in statistical analysis, the library staff at NOSM in performing literature searches, Dr Clare Cook in reviewing the draft article and Dr Priya Aronnilakkara in conducting the interviews. Thanks to Drs Mark Dube, Hassan Gorji and Sam Stone for acting as workshop facilitators. We also wish to thank all the participants who contributed their time and effort to completing the surveys and interviews.
References
Footnotes
Contributors DA led the conception and design, developed and delivered workshops, participated in data gathering, analysis and interpretation and drafted the paper. JA contributed to the design, analysis and interpretation of the work and revisions of drafts. MM and LB performed the qualitative analysis, wrote most of the qualitative component of the paper and contributed to revisions of drafts. JG and CC contributed to the conception and design of the study, and contributed to draft revisions. All authors gave final approval of the paper and agree to be accountable for the work.
Funding This project was funded by the Northern Ontario Academic Medicine Association Clinical Innovations Fund (Grants No. 14-02 and 15-02) and by the Postgraduate Medical Education Portfolio of the Northern Ontario School of Medicine.
Competing interests None declared. The researchers who performed the interviews and interview coding did not participate in workshop development or delivery, thus mitigating the risk of intellectual conflict of interest.
Ethics approval Laurentian University Research Ethics Board.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement Interested parties may contact the lead author for information regarding any additional unpublished data.