Article Text

Download PDFPDF

Original research
Differences in progression by surgical specialty: a national cohort study
  1. Carla Hope1,
  2. Jonathan Lund2,
  3. Gareth Griffiths3,
  4. David Humes4
  1. 1Division of Graduate Entry Medicine and Health Sciences, University of Nottingham, Nottingham, UK
  2. 2Division of Medical Sciences and Graduate Entry Medicine, University of Nottingham, Derby, UK
  3. 3Department of Vascular Surgery, Ninewells Hospital and Medical School, Dundee, UK
  4. 4NIHR Nottingham Biomedical Research Centre (BRC), Nottingham University, Nottingham, UK
  1. Correspondence to Carla Hope; carlahope{at}nhs.net

Abstract

The aim of surgical training across the 10 surgical specialties is to produce competent day 1 consultants. Progression through training in the UK is assessed by the Annual Review of Competency Progression (ARCP).

Objective This study aimed to examine variation in ARCP outcomes within surgical training and identify differences in outcomes between specialties.

Design A national cohort study using data from the UK Medical Education Database was performed. ARCP outcome was the primary outcome measure. Multilevel ordinal regression analyses were performed, with ARCP outcomes nested within trainees.

Participants Higher surgical trainees (ST3–ST8) from nine UK surgical specialties were included (vascular surgery was excluded due to insufficient data). All surgical trainees across the UK with an ARCP outcome between 2010 and 2017 were included.

Results Eight thousand two hundred and twenty trainees with an ARCP outcome awarded between 2010 and 2017 were included, comprising 31 788 ARCP outcomes. There was substantial variation in the proportion of non-standard outcomes recorded across specialties with general surgery trainees having the highest proportion of non-standard outcomes (22.5%) and urology trainees the fewest (12.4%). After adjustment, general surgery trainees were 1.3 times more likely to receive a non-standard ARCP outcome compared with trainees in trauma and orthopaedics (T&O) (OR 1.33, 95% CI 1.21 to 1.45, p=0.001). Urology trainees were 36% less likely to receive a non-standard outcome compared with T&O trainees (OR 0.64, 95% CI 0.54 to 0.75, p<0.001). Female trainees and older age were associated with non-standard outcomes (OR 1.11, 95% CI 1.02 to 1.22, p=0.020; OR 1.04, 95% CI 1.03 to 1.05, p<0.001).

Conclusion There is wide variation in the training outcome assessments across surgical specialties. General surgery has higher rates of non-standard outcomes compared with other surgical specialties. Across all specialties, female sex and older age were associated with non-standard outcomes.

  • education & training (see medical education & training)
  • medical education & training
  • surgery

Data availability statement

Data are available upon reasonable request. Data may be obtained from a third party and are not publicly available. The data used in this study were made available upon application to UKMED and accessed via virtual desktop. Requests for data should be made directly to UKMED.

http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

Statistics from Altmetric.com

Strengths and limitations of this study

  • This is the first study investigating factors affecting the Annual Review of Competency Progression outcome across all surgical specialties.

  • The major strength of the study is the large sample size comprising all higher surgical trainees between 2010 and 2017.

  • Unlike previous studies, this study uses data from reliable sources and is not dependent on survey data.

  • Limitations include the inability to investigate the causes behind our findings due to the nature of the analysis.

Introduction

Surgical training in the UK is comprised of 10 specialties: cardiothoracics, ENT (Ear, Nose and Throat surgery), general surgery, neurosurgery, oral and maxillofacial surgery (OMFS), paediatric surgery, plastic surgery, trauma and orthopaedics, urology and vascular. Successful entrants to surgical specialty training in the UK are enrolled into a 6-year programme within their specialty of interest, termed Higher Surgical Training. This follows a minimum of 4 years’ clinical experience after medical school graduation. Specialty training is mapped to a national curriculum which defines the levels of specialty-specific knowledge and skills required for completion of training around academic, procedural and clinical competence.1 Progression to the next stage of training is dependent on meeting curriculum targets and is assessed by the Annual Review of Competency Progression (ARCP).2 ARCP is the formal and structured evaluation of a portfolio of evidence to ensure a doctor has achieved the required competencies to enable satisfactory progression through each stage of medical training.2 The endpoint of successful completion of surgical training is the award of Certificate of Completion of Training (CCT) which allows entry onto the General Medical Council (GMC) specialist register.3

The aim of surgical training is the same across all specialties: to enable surgeons to acquire the curriculum standards and the professional responsibilities to practise as a day 1 consultant surgeon.4 If the training programme is meeting the needs of trainees, we would expect similar proportions of ARCP outcomes across all surgical specialties. The GMC is committed to promoting excellence in the delivery of postgraduate medical education and therefore it is important to ensure consistency of training standards and assessment across surgical training.5

Differential attainment refers to the differences in performance between different demographic groups6 and within surgical training this is reported internationally.7–9 Other studies have demonstrated differences in surgical training outcomes by gender, age and place of medical qualification10 11; however, to date, there have been no studies comparing ARCP outcomes across surgical specialties.

Aim

To examine variation in ARCP outcomes within UK surgical specialty training and identify any differences between specialties.

Method

We performed a cohort study using anonymised data from the UK Medical Education Database (UKMED), approval number 3506.12 UKMED collates data on sociodemographic and educational information for UK and international medical students and follows these students through their medical training, adding data as they progress. The database includes all UK medical students commencing studies in 2002 onwards, and from 2012 onwards, those graduating from non-UK medical schools and entering UK postgraduate training are included. For each graduate, ARCP outcomes are linked to demographic information. These data can be modelled to assess the differences in ARCP outcomes between surgical specialties. These data have been used to study outcomes and progression in medical specialties and foundation programme extensively and have been shown to be reliable.13–15

Patient and public involvement

Patient and public involvement was not sought given the educational nature of the study and the fact no patients were included in the study.

Outcome measure

The outcome measure in this study was ARCP outcome. ARCP outcome was converted into an ordinal category following which has previously been validated.16

  • Group 1: ‘satisfactory progress/training completed’: ARCP outcomes 1 and 6 (see table 1).

  • Group 2: ‘insufficient evidence presented’: ARCP outcome 5.

  • Group 3: ‘targeted training required (no extended time)’: ARCP outcome 2.

  • Group 4: ‘extended training time required/left programme’: ARCP outcome 3 or 4.

Table 1

ARCP outcomes

Any non-training outcomes were not included in this analysis, for example, those out of programme for research or experience.

Cohort derivation

All surgical trainees (ST3–ST8) from cardiothoracics, ENT, general surgery, neurosurgery, OMFS, paediatric surgery, plastic surgery, trauma and orthopaedics, and urology with an ARCP outcome recorded between 2010 and 2017 were included. Trauma and orthopaedic surgery was taken as the reference category in the regression analyses as this had the largest number of outcomes. In the event of more than one ARCP outcome within 12 months, the first chronological outcome was used. It was not possible to include vascular surgery as this only became a separate specialty in 2013 and the number of outcomes was too small (N=189).

Predictor variable selection

Age, sex, region of primary medical qualification, less than full-time training and year of ARCP were selected as confounding factors due to the variation in these demographics across specialties. Region of primary medical qualification was categorised into UK, European Economic Area and International Medical Graduates.17 Less than full-time training requires the same standard of competencies to be met at ARCP; however, the training time may be longer.

Statistical analysis

Descriptive statistics were derived to describe the satisfactory and non-standard cohorts. Demographic data were rounded to ensure anonymity as per guidance from the Higher Education Statistics Agency.18 Missing data were coded as a separate category within each variable. Χ2 test was used to compare the difference in the proportion of satisfactory outcomes compared with general surgery, as general surgery had the highest proportion of non-standard outcomes. Predicted probabilities of obtaining each ARCP outcome were calculated for each specialty to allow comparison across all specialties and displayed in a caterpillar plot using Stata post-estimation commands. A multilevel ordinal regression was performed, with a random intercept at the trainee level. This takes into account the fact that trainees had multiple ARCPs over time. Wald’s test and likelihood ratio tests were used to obtain p values.

A multilevel multiple ordinal regression was performed to investigate the odds of non-standard ARCP outcome by specialty. ORs with 95% CIs adjusted for sex, age, region of primary medical qualification, less than full-time training and year of ARCP were reported. P values of <0.05 were taken as significant. Analyses were performed using Stata V.15 (StataCorp, College Station, Texas, USA).19

Results

The analysis included 8220 trainees across all specialties, comprising 31 788 ARCP outcomes. Seventy-six per cent of all trainees were male. The proportion of female trainees varied from 15% in trauma and orthopaedics to 49% in paediatric surgery (table 2). The median age at ARCP was 35 years (IQR 32–37). The majority of trainees (81%) had attended a UK medical school. Eighty-two per cent (25 991 of 31 788) of the outcomes were satisfactory; 8% provided insufficient evidence, 6% required targeted training and 4% either required extended training time or were asked to leave the programme (table 3). Between 2010 and 2017, of the non-standard outcomes, 1.3% (75 of 5797) were outcome 4 (released from training programme). Between 2010 and 2017, standard outcomes reduced from 85% to 78% (p<0.001).

Table 2

Demographics by specialty

Table 3

Number of ARCPs by specialty

Trauma and orthopaedics had the largest proportion of ARCP outcomes, followed by general surgery (table 3). There was wide variation in the proportion of non-standard outcomes between specialties (table 3). General surgery had the lowest proportion of standard outcomes (77.5%), compared with 87.6% in urology. Maxillofacial surgery had the greatest proportion (6.9%) of extended training/left programme outcomes while ENT had the lowest at 2.2%. All specialties except paediatric surgery had a significantly greater number of satisfactory outcomes than general surgery on Χ2 test (table 3). Figure 1 illustrates the variation in ARCP outcome by specialty. General surgery had the lowest predicted probability of a satisfactory outcome comparing across all specialties and urology had the greatest. General surgery and paediatric surgery had the greatest predicted probability of all other non-standard outcomes. The results from the univariable and multiple ordinal regression analysis can be found in table 4.

Table 4

Multilevel ordinal regression showing factors associated with non-standard ARCP outcome

Figure 1

Predicted probabilities of ARCP outcome by surgical specialty. General surgery has the lowest predictive probability of a satisfactory ARCP outcome across all specialties, while urology has the greatest. ARCP, Annual Review of Competency Progression; ENT, Ear, Nose & Throat surgery; OMFS, oral and maxillofacial surgery; T&O, trauma and orthopaedics.

Discussion

This is the first study investigating the differences in ARCP outcome between surgical specialties. When combining all surgical specialties, women and older trainees are less likely to receive a standard ARCP outcome. The likelihood of a non-standard outcome has increased throughout time. There is between-specialty variation in ARCP outcomes with general surgery trainees significantly more likely to have a non-standard ARCP outcome compared with other specialties. Across the specialties, 1.3% of trainees are released from training (outcome 4).

Surgery remains dominated by men and attitudes towards women in surgery have been challenged in recent years. One study reported that 59% of women in surgery had experienced workplace discrimination, there was also a perception that the surgical environment is better equipped to support male trainees.20 Other studies have found that female trainees have adverse outcomes in surgical training. In a meta-analysis comprising 17 407 surgical trainees across the world, women were found to have a higher pooled attrition than men.21 A 2019 study found that female general surgery residents in the USA were significantly less likely to be granted operative autonomy by faculty than men,7 similar findings were seen among New Zealand general surgery trainees.22 The finding that women across all specialties have adverse outcomes requires investigation at the specialty level.

Between 2010 and 2017, the likelihood of non-standard ARCP outcome has increased. The possible explanations for this include increasing experience of the ARCP panel in ensuring criteria are met, changes in training quality and difficulties meeting ARCP requirements. Other factors such as increased familiarity of the system and requirements since its introduction in 2007, more rigorous application of the criteria and the recognition that it is more beneficial to trainees to issue a non-standard outcome earlier than later may also play a role in this finding. The largest increase in non-standard outcomes was in the insufficient evidence group. This may be due to increased service provision demands leaving less time for portfolio admin and changing curriculum requirements. Previous studies have described a decrease in operative case load for surgical trainees over time.23–25 A large proportion of UK trainees struggle to meet operative case requirements, with 85% reporting coming in on off days to gain extra experience.26 The variation in ARCP outcomes over time is likely multifactorial and beyond the scope of this study.

There are a number of factors which may contribute to the difference in ARCP outcomes between surgical specialties. Until the introduction of the new surgical curriculum in August 2021, there were differences in requirements for CCT between specialties. These inconsistencies have been highlighted in a 2019 paper by Wood et al.27 For example, the academic requirement for CCT in general surgery requires three peer-reviewed publications by the end of training, urology trainees require two, while paediatric surgery requires four, cardiothoracic trainees are required to deliver six presentations while those in ENT have no specified number. These different thresholds of attainment may therefore contribute to variation seen in the outcomes awarded in this study. Regardless of surgical specialty, there are universal requirements of a day 1 consultant. In the 2021 Joint Committee on Surgical Training curriculum update, the academic requirements are now uniform across all surgical specialties; however, variation in operative case load and work based assessments persists.28 29 The academic and management and leadership responsibilities are likely to be the same across specialties and therefore we support this standardisation. The aim of the surgical training programmes is to produce competent day 1 consultants, this study should prompt a review of these requirements across specialties.

The top performing specialties (urology, plastic surgery and ENT) have a smaller number of trainees which may increase the number of training opportunities and the ease of completing work-based assessments. In smaller specialties, it is likely that trainees and trainers will have greater training continuity, this has been shown to increase intraoperative entrustment.30 31 Examples of good practice from well-performing specialties should be highlighted and shared in order to improve the quality of surgical training for all.

In the USA, general surgery residents have a higher rate of attrition compared with other specialties.32–34 However, the failure to complete residency in these studies is largely due to voluntary attrition as opposed to unsatisfactory progression. Similarly, in a Canadian study across surgical specialties, general surgery residents were the most likely to consider leaving the programme.35 The main reason for this was concerns regarding work–life balance. Reassuringly, we have found that a low proportion of trainees are asked to leave the programme. However, the increasing rates of non-standard ARCP outcomes found in our study may negatively impact on trainee satisfaction and have a knock-on effect on attrition rates in the future.

Surgical trainees have been found to have low rates of career satisfaction and higher rates of burnout compared with trainees in other medical specialties.36 In a 2019 meta-analysis, general surgery trainees had the highest rate of burnout among the surgical subspecialties with a 58% prevalence rate of burnout.37 Burnout was defined as a long-term stress reaction marked by emotional exhaustion, depersonalisation and reduced personal accomplishment.38 Orthopaedics and neurosurgery had a slightly lower rate at 55% and 52%, respectively.37

The major strength of this study is that it captures a large contemporary cohort of surgical trainees across specialties. UKMED contains data from reliable sources and avoids the reliance on survey data as studies in this field often do. The grouping of ARCP outcomes into an ordinal scale has been previously used for studies based on data from UKMED.10 11 16 39 Due to the small numbers of outcome 4 (released from training programme) awarded nationally, a subgroup analysis on this cohort is not possible due to anonymity issues. Therefore, outcome 4 is combined with outcome 3 (additional training time) in our analysis. There may be different factors contributing to the award of these outcomes that will not be accounted for by combining them. One limitation is the inability to include vascular surgery within the analysis, due to the small number of ARCP outcomes available. Ethnicity data were not available for a large proportion of the study cohort and therefore it was not possible to include ethnicity as a variable; this is an area that needs to be addressed in further research. It is not possible to determine the underlying causes for the findings in this study, which are likely to be multifactorial. Further detailed work examining individual ARCP outcomes is required and a repeat review of variation between specialties once the new curriculum is well established.

Conclusion

There is a wide variation in the use of non-standard ARCP outcomes across UK surgical specialties with female sex and older age at ARCP being associated with non-standard outcomes. Furthermore, the use of non-standard outcomes is becoming more frequent over time. Possible explanations for this include variation in CCT requirements, training quality or difficulty meeting ARCP standards. The finding that some specialties and groups of trainees have lower rates of standard ARCP outcomes requires further investigation to ensure consistent training quality and assessment regardless of specialty. An in-depth study including details of work-based assessments, logbook data and supervisor reports per specialty could help to further understand the variation in outcomes.

Data availability statement

Data are available upon reasonable request. Data may be obtained from a third party and are not publicly available. The data used in this study were made available upon application to UKMED and accessed via virtual desktop. Requests for data should be made directly to UKMED.

Ethics statements

Patient consent for publication

Ethics approval

This study does not involve human participants.

Acknowledgments

I am a postgraduate of an affiliated association which is funding the publishing charge.

References

Footnotes

  • Twitter @carlahope1988

  • Contributors All authors were responsible for editing and drafting the final manuscript. CH performed the statistical analysis with input from DH. DH and CH were the guarantors of the work.

  • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

  • Competing interests None declared.

  • Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

  • Provenance and peer review Not commissioned; externally peer reviewed.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.