Objectives In a bid to promote high-quality postgraduate education and training and support the General Medical Council’s (GMC) implementation plan for trainer recognition, the Wales Deanery developed the Educational Supervision Agreement (EdSA). This is a three-way agreement between Educational Supervisors, Local Education Providers and the Wales Deanery which clarifies roles, responsibilities and expectations for all. This paper reports on the formative evaluation of the EdSA after 1 year.
Design Evaluation of pan-Wales EdSA roll-out (2013–2015) employed a mixed-methods approach: questionnaires (n=191), interviews (n=11) with educational supervisors and discussion with key stakeholders (GMC, All-Wales Trainer Recognition Group, Clinical Directors). Numerical data were analysed in SPSS V.20; open comments underwent thematic content analysis.
Participants The study involved Educational Supervisors working in different specialties across Wales, UK.
Results At the point of data collection, survey respondents represented 14% of signed agreements. Respondents believed the Agreement professionalises the Educational Supervisor role (85%, n=159 agreed), increases the accountability of Educational Supervisors (87%; n=160) and health boards (72%, n=131), provides leverage to negotiate supporting professional activities’ (SPA) time (76%, n=142) and continuing professional development (CPD) activities (71%, n=131). Factor analysis identified three principal factors: professionalisation of the educational supervisor role, supporting practice through training and feedback and implementation of the Agreement.
Conclusions Our evidence suggests that respondents believed the Agreement would professionalise and support their Educational Supervisor role. Respondents showed enthusiasm for the Agreement and its role in maintaining high standards of training.
- Educational supervision
- MEDICAL EDUCATION & TRAINING
- HEALTH SERVICES ADMINISTRATION & MANAGEMENT
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
- Educational supervision
- MEDICAL EDUCATION & TRAINING
- HEALTH SERVICES ADMINISTRATION & MANAGEMENT
Strengths and limitations of this study
The study contributes to the limited research on the impact of greater trainer recognition.
The sample includes Educational supervisors from all health boards in Wales working in many different specialties.
The study represents a snapshot in time at the introduction of the Educational Supervision Agreement; longitudinal follow-up is required.
Securing and maintaining high-quality education for trainee doctors is key to ensuring excellent patient care and patient safety. In August 2012, the General Medical Council (GMC) published details of new arrangements for the formal recognition of medical trainers in secondary care.1 These arrangements apply to four categories of trainer: in postgraduate training, (1) educational supervisors and (2) named clinical supervisors; and in undergraduate education, (3) those responsible for overseeing students’ progress at medical school and (4) lead coordinators at each Local Education Provider (LEP). These arrangements use existing standards for postgraduate training as outlined in the Trainee Doctor 2 and the Tomorrow’s Doctors 3 and map to seven headings requiring evidence in appraisal.1 4 Providing a process for the formal recognition of medical trainers is designed to offer assurance to patients and the public that new doctors are developed with the appropriate knowledge, skills and behaviours. Regulation should support trainers as good professional role models for students and trainees to emulate.5
The Wales Deanery, in a bid to promote high standards of postgraduate education and training, developed and piloted a tripartite Agreement in early 2011. This pilot preceded the publication of the GMC’s Implementation Plan (2012) and formed the foundation of the ‘Educational Supervision Agreement’ (EdSA)6 which was designed to meet requirements for the recognition of one of the GMC categories of postgraduate trainer, namely the Educational Supervisor. Educational supervisors are clinicians who are responsible for the overall supervision and management of a trainee’s trajectory of learning and educational progress during a placement or series of placements.1
The Educational Supervision Agreement
The EdSA is a signed agreement between three parties: an individual Educational Supervisor, a LEP (the NHS Health Board or Trust, ie, the medical director) and the Wales Deanery (the Postgraduate Dean). Currently, it is an Agreement with educational and not named clinical supervisors who are trained individuals responsible for overseeing a specified trainee’s clinical work throughout a placement in a clinical or medical environment.1 We note that extension to this group is planned. At present, EdSA is for foundation, core and higher specialty training, not training in general practice. The Agreement is designed to be an explicit demonstration of the three parties’ responsibilities and is intended to enhance communication and accountability between those responsible for delivering and supporting educational supervision in secondary care. In signing the Agreement, the Educational Supervisor commits to undertaking a minimum of 8 hours of continuing professional development (CPD) per year that maps to at least two of the seven framework areas adopted by the GMC5 and to cover all areas within a 5-year cycle. In addition, educational supervisors must provide evidence of suitable attitudes and behaviours (for example, from multisource feedback, evaluations of teaching, placement feedback forms, GMC survey results, trainee audits and analysis of critical incidents) which is reviewed in an annual appraisal of the trainer role.
The LEP responsibilities under the Agreement include building into a trainer’s job plan a time tariff of 0.25 ‘supporting professional activities’ (SPA), equivalent to 1 hour per week, per trainee. This was established following the initial pilot and consideration of the workload involved in supervising trainees. Supervision requires time not only in direct educational supervision but also in mentoring, preparing reports, supporting trainees in difficulty, sitting on annual review of competence progression (ARCP) panels and undertaking CPD for the trainer role. The Agreement also requires the LEP to provide annual appraisal for the trainer role and participate in quality management processes which include feedback to the Deanery.
Wales Deanery responsibilities include monitoring the provision of time and implementing quality management processes relating to Educational Supervision. The Deanery also signs up to responsibilities related to selection of educational supervisors, supporting mechanisms for the delivery of induction and other essential training, promoting annual appraisal and rewarding excellence.
The EdSA sets out to achieve five main objectives:
To enhance communication and accountability between the Deanery, the LEP and the Educational Supervisor.
To recognise the role of the Educational Supervisor and raise the profile of their work.
To promote high standards of postgraduate medical education and training in Wales.
To contribute to safe, high-quality patient care.
To support the GMC’s implementation plan.
Roll-out of EdSA commenced across Wales on 13 November 2013. This paper reports on the formative evaluation of the EdSA after 1 year.
We adopted mixed methods, collecting data from telephone interviews and a questionnaire. In reporting these data we followed standard guidance (see supplementary material).
Eleven semi-structured telephone interviews were conducted by KW, with a purposive sample of Educational Supervisors across Wales (February–July 2014), from a range of specialties including medicine, obstetrics and gynaecology and public health and both men and women (n=8 male; n=3 female). Participants were sent an invitation email which included an information sheet and consent form. The telephone interview is widely used in research,7–9 and we judged it to be an appropriate means by which to gather opinion about the effectiveness and impact of EdSA from individuals across a wide geographical area.
The telephone interviews lasted between 20 and 40 min, were recorded and transcribed. We conducted ongoing thematic content analysis.10 An initial coding framework was developed informed by the evaluation protocol and the literature. Two members of the research team (KW and AB) who have substantial experience in qualitative research and analysis independently read a selection of early transcripts, discussed their coding and further revised the coding frame before reaching consensus. Data were organised and analysed using Nvivo V.10. At 11 interviews, we determined data coding saturation had occurred as no new themes were raised in the later interviews.11
The questionnaire was informed by the aims of the EdSA and the interviews. Drafts were piloted and discussed with members of the Deanery responsible for the EdSA roll-out who were also Educational Supervisors. The questionnaire consisted of both closed and open free-text questions, thereby generating both quantitative and qualitative data.
All Educational and Named Clinical Supervisors across Wales were eligible to take part. Questionnaires were distributed in paper-based format at three Trainer and Educator Development Days (TEDD) hosted by the Wales Deanery across Wales (Cardiff, Swansea and Wrexham) between October and December 2014. These were free, CPD-accredited training days, for all practising Educational and Named Clinical Supervisors across Wales. Participants received questionnaires within their delegate packs and were asked to place completed questionnaires in collection boxes on their departure.
All questionnaire data were entered into IBM SPSS V.20. We reviewed variable frequencies and used exploratory factor analysis. Open comment data underwent thematic content analysis.10 A random selection of data entries (10%), including qualitative open comments, was checked for accuracy.
Ethical approval was granted by Cardiff University’s School of Postgraduate Medical and Dental Education Research Ethics Committee (03/01/2014).
We present and discuss results from the questionnaires and interviews. As themes are replicated across interviews and questionnaires, we integrate results in our presentation of findings. Box 1 provides an overview of the themes raised in the telephone interviews. These shaped the survey questions.
Overview of themes raised in telephone
Formal recognition of role
Consolidation of tripartite agreement
Legitimation of time spent supervising
Professionalising the role
Points of concern
Recognising supporting professional activities’ time in practice
Monitoring the agreement
Managing administrative workload
Barriers/enabler to implementation of the agreement
Size and staffing specialty
Predictability of work demands
Ability to attend training
Attitude of the Health Board and Deanery
Responses were received from individuals across all seven LEPs in Wales. The number of returns and the response rates for each of the regional events are presented in table 1. Overall, 191 surveys were returned, of these 62% of respondents were male.
Eighty-seven per cent of respondents (n=167) indicated having already signed the EdSA. At the time of data collection, 1265 Educational Supervisors had signed the Agreement; our sample thus represents 14% of signed agreements across Wales. Respondents were from a range of specialties; well-represented specialties included anaesthetics (n=29), paediatrics and child health (n=26) and core medical training (n=25). The majority indicated being both an Educational and Named Clinical Supervisor (63%, n=121), while 26% (n=49) indicated solely being an Educational Supervisor and 9% (n=17) solely a Named Clinical Supervisor.
Survey respondents were asked to indicate for how many trainees they acted as Educational Supervisor. The majority reported having one trainee (39%, n=60; table 2).
Although 80% indicated having a job planning meeting within the last 12 months, 18% (n=34) had not yet had their job planning meeting and three individuals responded that their last job planning meeting was in excess of 12 months ago. Of those having a job planning meeting within the last 12 months, 76% (n=135) reported having their SPA time recognised within the job plan, while 24% (n=43) reported they had not. Most of those who did have their SPA time recognised reported getting time in work time (78%). Respondents were asked further about this; 66 respondents (35%) provided comment which we summarised (see table 3). Some respondents made more than one comment. Overall, more commented on difficulties.
Many comments were about the reality of getting the SPA time in practice. Some respondents reported not having their SPA time recognised and not feeling supported:
Trainer role not recognised/not discussed in job plan yet. (w.120.f)
I have discussed but do not feel supported. (w.13.f)
Some individuals indicated not having adequate time recognised within their job plan or being confused and unsure about it:
I’m not sure it has been taken into account. (c.194.f)
Not enough time in job plan given. Confusion about time allocated for Educational vs. Clinical Supervisors. (s.123.f)
A sizeable number of respondents reported clinical workload pressures which take precedence and impinge on allocated SPA time; these respondents found ‘it difficult to fit in’ and commented on completing work at home:
Clinical commitments can take precedence over SPA time. (c.14.f)
Yes in theory but in practice not possible due to current acuity of patients, gaps in junior doctor rotas, pressure of targets (eg, cancer), absent SpRs etc. (c.154.f)
My SPA time is in my job plan and allocated within work time, but in practice all my SPA is done ‘out of work’ time. (c.168.m)
I specifically have an SPA that is in the evenings and what I like about that is nobody can interrupt me and I can get those things done. (19: telephone interviewee)
Impact of the EdSA
To understand the perceived impact of the EdSA, we asked respondents to indicate along a 10-point scale (where one signified negative impact and 10 positive impact) the effect of the Agreement in relation to their roles as educational and named clinical supervisors, the quality of training, time with trainees and patient care. It is noteworthy from table 4 that all means are above 6, indicating a positive impact.
We asked individuals to rate along a five-point scale, from ‘strongly disagree’ to ‘strongly agree’, whether they believed the Agreement enhances accountability for themselves as supervisors, for the LEP and the Deanery. Responses indicate that the great majority of participants believed the Agreement would enhance accountability for all parties (see table 5).
In telephone interviews, nearly all spoke of the Agreement enhancing accountability (10/11) for all parties. Some interviewees hoped that the Agreement would improve Health Board compliance (7/11), “the important thing is that it forces our Health Board to make sure that we are looking after our junior doctors” (18: telephone interviewee).
When asked, the majority of survey respondents (85%) thought that the Agreement professionalises the role of the Educational Supervisor (table 6). The majority (76%) also thought the Agreement provides leverage to negotiate for the recognition of SPA time in job planning meetings. The comments reported earlier (table 3) provide some insight into what those disagreeing with this statement might be experiencing. From the telephone interviews, all agreed that the EdSA had the potential to professionalise their role as an Educational Supervisor. Some interviewees (4/11) mentioned that it would do so by screening out those who could not or did not want to commit to the supervisory role: “because now you have to train at it and you have to show that you’re continuing to train at it. I think that means that it should be seen in a different light, and it’s not now seen as automatic” (110: telephone interviewee).
The majority of respondents agreed (63% ‘agree’/‘strongly agree’) that the EdSA would ensure the quality of Educational Supervision, although it is worth noting a sizeable proportion (27%, n=50) ‘neither agreed nor disagreed’ (table 6). The results were similar for whether they thought that having their supervisory role recognised formally would help in the management of difficult trainees (table 6).
The responses to another set of statements are displayed in table 7. These report views on factors influencing the implementation of the Agreement. All telephone interviewees suggested that successful implementation of the Agreement would vary by specialty. This perspective was not fully supported by the wider questionnaire data. Fifty-six per cent (n=99) of respondents ‘agreed’/‘strongly agreed’ implementation would be easier in some specialties than others although 40% (n=72) ‘neither agreed nor disagreed’. However, when asked if implementation of the EdSA within their own specialty would be difficult, only 17% (n=32) of respondents ‘agreed’/‘strongly agreed’.
All interviewees were positive about the inclusion of Named Clinical Supervisors in theory: “it would be brilliant because…we could actually get together and talk about it beforehand and organise [trainees’] activity better” (111: telephone interviewee). From the questionnaire data, 47% (n=86) either ‘agreed’ or ‘strongly agreed’ that extending the EdSA to Named Clinical Supervisors would increase communication between the Educational and Named clinical supervisor, although almost as many ‘neither agreed nor disagreed’ (41%, n=75).
Data from our telephone interviews suggested that the Agreement was a ‘tick-box’ exercise (8/11), “to me it has become a bit of a paper exercise, meaningless” (17: telephone interviewee). To explore whether this was a view shared more widely, we included a statement about this within the survey. Results from respondents suggested that this was not the broader view, as 40% (n=73) ‘disagreed’/‘disagreed strongly’ and a further 34% (n=63) ‘neither agreed nor disagreed’ (see table 7).
Respondents were also asked to indicate whether they were confident that the Agreement had enough ‘teeth’ to ensure a high standard of trainee supervisory practice. Many were undecided (48%; n=90) (table 7). Interestingly, among the interviewees the issue of monitoring and a lack of trust was raised (6/11); some interviewees felt that increasing the monitoring of Educational Supervisors indicated lack of trust. However, from the questionnaire data, the idea of monitoring (policing compliance) was welcomed by many. The notion of the Agreement having enough ‘teeth’ was also alluded to within the ‘free-text’ comments. This view was further supported by 62% (n=114) of respondents who indicated that they did not think or were not sure there was sufficient information available on how EdSA would be enforced. Respondents commented on a lack of information, concerns about the ability of the Agreement to sift out poor trainers and whether it would be properly enforced. Telephone interviews and open comments from questionnaires revealed robust and no nonsense approaches to upholding quality and accountability of Educational Supervisors and LEPs:
Know what penalties we must/will have if can’t fulfil responsibilities for training. (c.170.m)
Publish names of those trainers who have not signed Agreement, and remove SPA time from them. (w.122.f)
The accountability of programme directors and Educational Supervisors should be cross-checked regularly by talking to their colleagues. (c.122.m)
It will need policing and feedback and there need to be instances where people are being taken off their in Educational Supervision because they don’t fulfil their role or aren’t able to fulfil their role. Unless you do that it becomes a paper exercise and toothless. (14: telephone interviewee)
A further set of statements were presented to participants. The results are presented in table 8. The majority of respondents, 82% (n=153), ‘strongly agreed/agreed’ that they would like to have more feedback on their role as a supervisor; this is in line with suggestions by interviewees (5/11). Some interviewees said that the Agreement would enhance their access to training but raised concerns about their ability to attend sufficient training. To explore these concerns more widely, we asked respondents whether the Agreement would ensure they had better access to educational training for their role: of the sample, 71% (n=131) ‘strongly agreed’/‘agreed’. Furthermore, 48% (n=87) of respondents reported they would no longer have difficulty in negotiating time away from work to attend educational training sessions (as indicated by disagreement with the statement).
A concern about workload identified in the telephone interviews (6/11), was supported by questionnaire data. When respondents were asked to rate whether they completed all their administrative responsibilities within work time, no participants ticked ‘strongly agree’ and 62% (n=114) disagreed (table 8).
A key element of EdSA is CPD (a minimum of 8 hours of CPD per year). The Wales Deanery provides training opportunities, and in general, interviewees were positive about the role of the Deanery in helping them to fulfil the requirements of the Agreement:
I haven’t come across something that the Deanery wouldn’t do for us. (111: telephone interviewee)
At the moment updates have all been provided by the Deanery. We get frequent updates locally in the postgrad centre, and they have all the all-Wales one …which were great. (18: telephone interviewee)
Seventy per cent of questionnaire respondents indicated they felt comfortable in being able to demonstrate required levels of CPD.
One of the primary aims of the Agreement is to raise standards of postgraduate training and education and to help support good supervisory practice. Despite an interview, observation that the Agreement could undermine current Educational Supervision practice this view was not supported by the wider questionnaire data: 62% (n=114) indicated disagreed that their supervisory practice was undermined (see table 8).
With a view to advancing the Agreement, respondents were asked to comment on what they thought were barriers to implementation. We present a summary in table 9.
Respondents’ suggestions about what might hamper implementation were largely pragmatic. These include extant time and clinical workload pressures which resulted in concern that existing commitments would inevitably lead to a squeeze on individuals’ ability to be a supervisor:
Time, multiplicity of roles. The best doctors tend to deliver the most roles. It is actually likely that very busy clinicians/ managers/ academics will necessarily have to drop supervision. (s.129.f)
Priorities of service delivery and inadequate job planning were further perceived barriers:
Employers still consider service delivery more important than training educational CPD needs of doctors. (s.11.f)
I feel this a very significant first formal step in the right direction. Will take some time before this is universally reflected in people’s job plans as appraisal information. (s.124.f)
Furthermore, issues around upholding quality were noted, for instance with regard to CPD training and the need for enforcement of the Agreement:
Relevance of courses & CPD which are required annually & avoidance of repetition. (w.15.m)
No ‘teeth’ to make radical change needed within health boards. (c.119.f)
In an effort to make sense of the complexity of responses, we conducted a factor analysis. Following Field,12 we first confirmed that a high number of responses to statements correlated (13/18 correlated), checked that the Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy was over the recommended value of 0.5 (KMO=0.87) and that all KMO values for individual items were greater than 0.5 (all were >0.87). Bartlett’s test of sphericity was significant (χ2(171)=1133.72, p<0.001) which indicated that correlations between items were sufficiently large for principal component analysis. Five components had eigenvalues over Kaiser’s criterion of 1 and in combination explained 56.3% of the variance. However, the scree plot showed a levelling off of eigenvalues after factor 3. Given this and the relatively small sample size, three factors were retained in the final analysis. Table 10 displays the factor loadings after varimax rotation. For our analysis, we have used a cut-off of 0.30.13 We described the factors as primarily being about: professionalising the role of Educational Supervisors (factor 1); supporting supervisory practice through training and feedback (factor 2) and implementation of the Agreement (factor 3).
This paper presents findings from a formative evaluation of the first year of the roll-out the EdSA. The results can help to shape development of the Agreement to fit with trainers’ expectations and inform subsequent evaluation of EdSA. Given the recent GMC trainer standards, this work is timely and can inform others working to meet requirements. The Agreement can be understood for the most part, as being well received by the sample we surveyed. The Agreement successfully addresses the objective of supporting the GMC’s implementation plan and goes some way to ensuring commitment from each of the parties to the tenets of the Agreement which are commensurate with the GMC’s intentions.
A strength of the study was the way in which data from the interview was triangulated in the questionnaire. We have revealed a level of similarity in opinion across those in our sample as well as highlighting some difference of opinion (eg, adequacy of SPA provision, hours of CPD, extent to which such an Agreement should be enforced). Findings from the explorative factor analysis help us to identify and simplify the main purposes of the Agreement (professionalising the role and providing support) and the challenge (implementation). These key elements are commensurate with the aims and objectives of the GMC implementation plan.
Our results should be interpreted in context however, acknowledging that these data reflect a snapshot in time from a relatively small self-selected sample of Educational Supervisors attending a training event during the introductory period of the Agreement. Self-selection operated at two levels: first our sample included only those individuals who chose to attend the CPD event; second, we only have data from those who chose to complete and return the questionnaire. In the future, to enhance response rate, we will seek to build time into the programme for questionnaire completion. We also note that the group of signed-up Educational Supervisors was not static, increasing in number as the Agreement rolled out. This makes cohort comparison problematic. However, we can report that these findings are commensurate with those reported from the GMC Trainer Survey14 for educational and Named cClinical Supervisors in areas such as recognition for the role, SPA time and support for CPD. Other studies looking at the effectiveness of Educational Supervisors acknowledge practice variability15 and low educator appraisal16 provide argument for professionalising the role which is part of the purpose of the EdSA. We recognise that in respect of the EdSA, longer-term follow-up is needed.
The findings suggested that the EdSA will fulfil the majority of its aims or objectives: the Agreement is seen as enhancing communication and accountability between the Deanery, the LEP and the Educational Supervisor; and professionalising the role of Educational Supervisors through recognition of the role, SPA time tariffs in job plans and annual appraisal of the trainer role. Results show that there is an expectation that the Agreement will promote and enhance standards of postgraduate medical education and training in Wales. Qualitative descriptive data provided by respondents through open comments and telephone interviews shed light on the positive expectations of the Agreement to deliver on its principles, providing the authority to support and recognise the role of Educational Supervisors in practice. However, while respondents are proponents of safe, high quality patient care, the impact of the Agreement on patient care cannot yet be judged and needs further evaluation.
The evaluation shows the positive expectations of the Agreement to deliver on its principles, providing the quasi-regulatory authority to support, recognise and monitor the role of Educational Supervisors in practice. These findings provide positive reinforcement for the development and roll-out of EdSA and its potential extension to Named Clinical Supervisors.
We gratefully acknowledge funding from the Wales Deanery in supporting this work. Without the award, we would have been unable to undertake this important study. We would like to express our appreciation to those educational supervisors who have taken the time to participate. In addition, we would like to express thanks to Dr Esther Muddiman for assisting with data collection and Elaine Russ for providing project support.
Contributors KW and AB were involved in the conception and design of the study and conducted analysis of the data; all authors contributed to data interpretation; the article was drafted by KW and revised by AB, CG and AGS. The final submission has been approved by all coauthors (AB, CG and AGS).
Funding This work was supported by the Wales Deanery.
Disclaimer Views expressed within this paper are those of the authors.
Competing interests The authors declare they have no personal competing interests. We note authors CG and AGS have roles in Wales Deanery associated with postgraduate education and quality.
Ethics approval This study received ethical approval from the Postgraduate School for Medical and Dental Education Research Ethics Committee, Cardiff University (03/01/2014).
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement No additional data available.
Correction notice This paper has been amended since it was published Online First. Owing to a scripting error, some of the publisher names in the references were replaced with 'BMJ Publishing Group'. This only affected the full text version, not the PDF. We have since corrected these errors and the correct publishers have been inserted into the references.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.