Getting lost in translation? Workplace based assessments in surgical training

doi:10.1016/j.surge.2013.03.001

The Surgeon

Volume 11, Issue 5, October 2013, Pages 286-289

https://doi.org/10.1016/j.surge.2013.03.001 Get rights and content

Abstract

Workplace based assessments (WBA) are integral to the competence-based surgical training curriculum that currently exists in the UK. The GMC emphasise the value of WBA's as assessments for learning (formative), rather than as assessments of learning (summative). Current implementation of WBA's in the workplace though, is at odds with their intended use, with the formative functions often being overlooked in favour of the summative, as exemplified by the recent announcement that trainees are required to complete a minimum of 40 WBA's a year, an increase from 24. Even before this increase, trainees viewed WBA's as tick-box exercises that negatively impact upon training opportunities. As a result, the tools are commonly misused, often because both trainees and trainers lack understanding of the benefits of full engagement with the formative learning opportunities afforded by WBA's. To aid the transition in mind-set of trainees and trainers to the purpose of assessment in the workplace, the GMC propose the introduction of ‘supervised learning events’ and ‘assessments of performance’ to supersede ‘WBA's’. The impact of this change and how these will be integrated into surgical training is yet to be seen, but is likely to be a step in the right direction.

Section snippets

Workplace based assessments

WBA is ‘assessment of working practices based on what trainees actually do in the workplace, and predominantly carried out in the workplace itself’.⁶ The ISCP requires trainees to complete a range of assessments: ‘mini clinical evaluation exercises (mini-CEX), case-based discussions (CbD), direct observation of procedural skills (DOPS), procedure based assessments (PBA's) and 360° appraisals using a multi-source feedback tool (MSF). WBA's are designed to integrate teaching, learning, assessment

Reliability demands that multiple assessments be performed

Accurately assessing the performance and skills of doctors in training remains a significant challenge.¹⁰ One question often asked of WBA's is how reliable they are – that is their reproducibility. Three factors are thought to impact upon reliability: the number of encounters observed, the number of assessors and the aspects of performance being assessed.¹¹ It has been recognised that doctors' performance is case specific and poorly predictive of performance in other cases.¹² A result of this

Trainees view WBA's as ‘tick-box’ exercises which leads to their misuse

Even as early as 2004, a study concluded that surgical training is being negatively impacted upon by WBA's due to the considerable administrative burden.¹⁴

This has been followed by a large independent survey of surgical trainees published in 2009 which highlighted the dissatisfaction of surgical trainees with the administrative burden placed on them by the WBA's they were required to undertake.¹⁵ The survey identified that 82% of trainees felt that the time taken to complete the required number

Lessons to be learnt from the Foundation Programme

A comprehensive review of the Foundation Programme has been recently performed. Regarding WBA, Collins concluded that the number of assessments required was ‘formidable’, and that urgent review should be made into their frequency.²⁰ The number of assessments required of surgical trainees is now much greater than that required at the time of this review of the Foundation Programme. It would appear a logical extrapolation that the same conclusion would therefore be true of surgical training, that

Are ‘supervised learning events’ the answer?

Although lack of time is clearly an issue, trainees lack understanding of the educational theories underpinning the use of WBA's, and the benefit of full engagement in the formative assessment process, which is why they are viewed by many as tick-box, summative exercises.16, 22

Trainers too, must possess understanding of the purpose, timing and frequency of WBA's to ensure the tools are implemented correctly.²³ However, engagement with training in the use of WBA's can be very low – just 11.5% of

Conclusion

Workplace based assessments are an integral part of surgical training in the UK. Unfortunately, they are negatively viewed, on the whole, by trainees – predominantly because of the administrative burden that is placed upon an already time constrained training programme. The GMC's view that WBA's should be formative learning events is rarely realised and the tools are commonly misused. This may be partly due to time constraints, but full understanding by trainees and trainers of WBA educational

References (25)

J. Beard
Workplace-based assessment: the need for continued evaluation and refinement
Surgeon
(2011)
Department of Health
Modernising medical careers: the response of the four UK health ministers to the consultation on ‘Unfinished business – proposals for reform of the senior house officer grade’
(2003)
Department of Health
A guide to the implications of the European working time directive for doctors in training
(2009)
G. Purcell-Jackson et al.
How long does it take to train a surgeon?
BMJ
(2009)
Intercollegiate Surgical Curriculum Programme
The purpose and structure of the training programme
(2006)
Modernising Medical Careers. The Gold Guide: a reference guide for postgraduate specialty training in the UK....
G.E. Miller
The assessment of clinical skills/competence/performance
Acad Med
(1990)
General Medical Council
Workplace based assessment: a guide for implementation
(2010)
A. Pentlow
Workplace-based assessments in surgery: are we heading in the wrong direction?
Med Teach
(2013)
L.W.T. Schuwirth et al.
Challenges for educationalists
BMJ
(2006)

J.J. Norchini

The validity of long cases

Med Educ

(2001)

A.S. Elstein et al.

Medical problem-solving: an analysis of clinical reasoning

(1978)

Cited by (28)

Exploring mentors’ interpretation of terminology and levels of competence when assessing nursing students: An integrative review
2018, Nurse Education Today
Citation Excerpt :
In support of this argument, Boud and Falchikov (2006) and Schuwirth and Van der Vleuten (2011) identified that the central purpose of educational curricula is for learning to take place, so constructively aligned assessment impacts positively on future learning through providing targets and feedback that focus and drive deeper learning. The concept of assessment as a way of providing constructive feedback for learning emerged to challenge the traditional view that the aim of assessment is to judge whether a student passes a test (Ali, 2013; Haines et al., 2013; Popham, 2001). Assessment for learning focuses on the formative potential of the assessment through individualised feedback on performance, and provides a continuum where improvements are documented in individual areas as moving from ‘working towards competence’ to ‘competent’ thus refocusing attitudes towards assessment as something that can facilitate learning rather than a process that highlights incompetence.
The purpose of this integrative review is to evaluate the empirical and theoretical literature on the challenges mentors face in interpreting and assessing levels of competence of student nurses in clinical practice.
An integrative review of the literature.
An extensive and systematic literature search was conducted covering the period 1986–September 2016 across twelve databases covering health and education related publications. Grey literature was searched from wide relevant sources.
Sources were eligible for review when they referred to mentor's interpretation or assessment of student nurses' level of competence in practice settings. Methodological rigor of the included studies was evaluated with the Mixed Methods Appraisal Tool.
After screening 1951 records by titles, abstracts and full text, 8 were selected for review. The methodological quality of the studies was moderate. The studies reported:
- •
  Difficulties in the language used to describe competencies.
- •
  The challenge of distinguishing between different levels of competence.
- •
  Lack of clear and constructive feedback to students.
Accurate and fair assessment of students is impeded by a lack of transparent and explicit criteria.
There is a need to establish a transparent and common language to distinguish between and facilitate interpretation of different levels of competence. Well-designed rubrics might offer the solution to the challenges faced in practice-based assessment and necessitates further research into their use.
Consensus Conference on North American Training in Hepatopancreaticobiliary Surgery: A Review of the Conference and Presentation of Consensus Statements
2016, American Journal of Transplantation
The findings and recommendations of the North American consensus conference on training in hepatopancreaticobiliary (HPB) surgery held in October 2014 are presented. The conference was hosted by the Society for Surgical Oncology (SSO), the Americas Hepato-Pancreatico-Biliary Association (AHPBA), and the American Society of Transplant Surgeons (ASTS). The current state of training in HPB surgery in North America was defined through three pathways—HPB, surgical oncology, and solid organ transplant fellowships. Consensus regarding programmatic requirements included establishment of minimum case volumes and inclusion of quality metrics. Formative assessment, using milestones as a framework and inclusive of both operative and nonoperative skills, must be present. Specific core HPB cases should be defined and used for evaluation of operative skills. The conference concluded with a focus on the optimal means to perform summative assessment to evaluate the individual fellow completing a fellowship in HPB surgery. Presentations from the hospital perspective and the American Board of Surgery led to consensus that summative assessment was desired by the public and the hospital systems and should occur in a uniform but possibly modular manner for all HPB fellowship pathways. A task force composed of representatives of the SSO, AHPBA, and ASTS are charged with implementation of the consensus statements emanating from this consensus conference.
Assessing the nontechnical skills of surgical trainees: Views of the theater team
2016, Journal of Surgical Education
Citation Excerpt :
In addition, consultant feedback also suggests trainer dissatisfaction with the process, with some seeing it as a “tick-box exercise” also. This has previously been reported in studies looking at perceptions of work-based assessments.15 Local education initiatives, workshops, and online training modules help to promote the NOTSS ethos and the potential that NOTSS has to enhance service delivery and patient outcomes, something that would be appreciated by scrub staff as much as the medical trainers as well as patient outcomes.
This study aims to explore the views of members of theater teams regarding the proposed introduction of a workplace-based assessment of nontechnical skills of surgeons (NOTSS) into the Intercollegiate Surgical Curriculum Programme in the United Kingdom. In addition, the previous training and familiarity of the members of the surgical theater team with the concept and assessment of NOTSS would be evaluated.
A regional survey of members of theater teams (consultant surgeons, anesthetists, scrub nurses, and trainees) was performed at 1 teaching and 2 district general hospitals in South Yorkshire.
There were 160 respondents corresponding to a response rate of 81%. The majority (77%) were not aware of the NOTSS assessment tool with only 9% of respondents reporting to have previously used the NOTSS tool and just 3% having received training in NOTSS assessment. Overall, 81% stated that assessing NOTSS was as important as assessing technical skills. Trainees attributed less importance to nontechnical skills than the other groups (p ≤ 0.016). Although opinion appears divided as to whether the presence of a consultant surgeon in theater could potentially make it difficult to assess a trainee’s leadership skills and decision-making capabilities, overall 60% agree that the routine use of NOTSS assessment would enhance safety in the operating theater and 80% agree that the NOTSS tool should be introduced to assess the nontechnical skills of trainees in theater. However, a significantly lower proportion of trainees (45%) agreed on the latter compared with the other groups (p = 0.001).
Our survey demonstrates acceptability among the theater team for the introduction of the NOTSS tool into the surgical curriculum. However, lack of familiarity highlights the importance of faculty training for assessors before such an introduction.
Workplace based assessments - Confusion and misuse continues to be a problem
2015, Clinical Medicine, Journal of the Royal College of Physicians of London
Surgical trainers' experience and perspectives on workplace-based assessments
2015, Journal of Surgical Education
Citation Excerpt :
Several studies have highlighted a level of dissatisfaction with the WBAs that form part of this new curriculum. Trainees have commented that they felt they lacked validity and reliability and have become no more than a tick-box exercise.4 A part of this stems from a lack of engagement with these new tools by both trainers and those being assessed.
Workplace-based assessments (WBAs) were designed to provide formative feedback to trainees throughout their surgical career. Several studies highlight dissatisfaction with WBAs, and some feel they lack validity and reliability and exist as a “tick-box exercise.” No studies have looked at the attitudes of the assessor.
The aim of this study was to evaluate perceptions and experience of the 4 intercollegiate surgical curriculum programme WBAs by assessors.
An 18-item electronic questionnaire, including 6-point Likert scoring questions, was e-mailed to all surgical program directors for distribution to general surgery trainers within their deanery.
In total, 64 responses were received. All trainers had been trained in using WBAs. Trainers had the most experience with procedure-based assessments (PBAs)—72% of trainers had completed more than 25 PBAs. Trainers felt PBAs were the most beneficial WBA, and both PBAs and case-based discussions were regarded as significantly more useful than mini–clinical evaluation exercise (p < 0.05). More than 74% stated that WBAs were mainly initiated by trainees, and only 10% had specific sessions allocated to complete WBAs.
WBAs are regarded as beneficial to trainees. The results suggest that assessors feel case-based discussions and PBAs, which assess higher thinking and practice of complex practical skills, respectively, are significantly more useful than assessments involved in observing more straightforward clinical and procedural interactions.
Trauma and Orthopedic Surgery Curriculum Concordance: An Operative Learning Curve Trajectory Perspective
2019, Journal of Surgical Education
Citation Excerpt :
Craft-based specialties are thus faced with unique challenges. Working time legislation has reduced the exposure that surgical trainees receive,4 while the work-based assessments developed to assess operative competence, PBAs have been criticized by both trainers and trainee alike.5-8 In order to be awarded a CCT, Trauma & Orthopedic (T&O) Higher Surgical Trainees (HSTs) must achieve a minimum total of 1800 cases over 6 years of training.3
Certification of completion of training in Trauma and Orthopedic (T&O) surgery in the UK requires the demonstration of operative competence in 12 index procedures, achieved through attaining a level 4 consultant-validated procedure-based assessment (PBA). The aim of this study was to evaluate the trajectory of operative learning curves related to PBA performance with respect to operative caseload and training time.
Logbook data from consecutive 24 higher T&O trainees were compared with PBA evaluations to determine the relationship between PBA level, operative experience, training time, and indicative numbers. Learning curve gradients were calculated using trigonometry related to operative experience and training time.
A higher surgical orthopedic training program serving a single UK (Wales) Deanery.
Twenty-four consecutive higher T&O surgery trainees.
Median caseload to achieve level 4 competences ranged from 9 (interquartile range 6-12) for tension band wiring (olecranon or patella) to 101 (61-127) for arthroscopy, with significant variation between all 12 procedures (p < 0.001). Median number of PBAs to reach level 4 competences was 4 (2-6) with significant variation between procedures (p < 0.001). Median learning curve gradients to achieve level 4 competence for tension band wiring were 68.2° and 33.7° by caseload and training time respectively, compared with 12.2° and 45° for arthroscopy, with significant learning curve variation for all procedures related to caseload between first level 3 and first level 4 PBA (p < 0.001). Competence ratios were <1 (median 0.99, range 0.70-2.53) for 6 of the 12 indicative procedures.
Significant learning curve trajectory variance was observed, with discrepancies between indicative operative numbers and the point at which competence was judged achieved. Numbers of index operations to achieve certification of completion of training warrant further examination.

View all citing articles on Scopus

View full text

ReviewGetting lost in translation? Workplace based assessments in surgical training

Abstract

Section snippets

Workplace based assessments

Reliability demands that multiple assessments be performed

Trainees view WBA's as ‘tick-box’ exercises which leads to their misuse

Lessons to be learnt from the Foundation Programme

Are ‘supervised learning events’ the answer?

Conclusion

Surgeon

Modernising medical careers: the response of the four UK health ministers to the consultation on ‘Unfinished business – proposals for reform of the senior house officer grade’

A guide to the implications of the European working time directive for doctors in training

How long does it take to train a surgeon?

BMJ

The purpose and structure of the training programme

The assessment of clinical skills/competence/performance

Acad Med

Workplace based assessment: a guide for implementation

Workplace-based assessments in surgery: are we heading in the wrong direction?

Med Teach

Challenges for educationalists

BMJ

The validity of long cases

Med Educ

Medical problem-solving: an analysis of clinical reasoning

Review
Getting lost in translation? Workplace based assessments in surgical training