Jump to comment:

• Strengthening the Validity of Educational Performance Measurement Scores to Predict Success in MRCS Examinations
  Dr Rebecca Williams, Professor Stuart Enoch
  Published on: 6 September 2021

• Published on: 6 September 2021

  Strengthening the Validity of Educational Performance Measurement Scores to Predict Success in MRCS Examinations

  • Dr Rebecca Williams, Department of Education University of Oxford
  • Other Contributors:
    • Professor Stuart Enoch, Director of Postgraduate Surgical Studies and Chairman of Faculty, MRCS Courses

  Dear Editor,

  We read, with profound interest, the paper entitled ‘Does performance at medical school predict success at the Intercollegiate Membership of the Royal College of Surgeons (MRCS) examination?’ by Ellis et al. This is a highly timely study, and we sincerely thank the authors for foregrounding the causal relationship between achievement in medical school and outcomes in postgraduate surgical examinations. As the paper rightly asserts, no previous studies have attempted to investigate this. A nuanced appreciation of predictors of success in the MRCS examinations can directly translate to the way in which surgical trainees are taught, and this issue is, therefore, unequivocally worthy of attention.

  Several studies have revealed that success at one level equates to success at another level. A notable example is the longitudinal studies conducted in Australia, Canada, Denmark and Switzerland, which have identified that students who performed highly in the Programme for International Student Assessment (PISA) tests at the age of 15 had an enhanced probability of attaining higher levels of education and were less likely to be unemployed (OECD, 2018). Specifically within the context of healthcare, Lipton et al. (1984) explored how secondary school results and personality test scores can act as predictors of achievement in medical school, whilst Wolkowitz and Kelley (2010) investigated how scores in science, mathematics, reading and English might forecast...

  Show More

  Dear Editor,

  We read, with profound interest, the paper entitled ‘Does performance at medical school predict success at the Intercollegiate Membership of the Royal College of Surgeons (MRCS) examination?’ by Ellis et al. This is a highly timely study, and we sincerely thank the authors for foregrounding the causal relationship between achievement in medical school and outcomes in postgraduate surgical examinations. As the paper rightly asserts, no previous studies have attempted to investigate this. A nuanced appreciation of predictors of success in the MRCS examinations can directly translate to the way in which surgical trainees are taught, and this issue is, therefore, unequivocally worthy of attention.

  Several studies have revealed that success at one level equates to success at another level. A notable example is the longitudinal studies conducted in Australia, Canada, Denmark and Switzerland, which have identified that students who performed highly in the Programme for International Student Assessment (PISA) tests at the age of 15 had an enhanced probability of attaining higher levels of education and were less likely to be unemployed (OECD, 2018). Specifically within the context of healthcare, Lipton et al. (1984) explored how secondary school results and personality test scores can act as predictors of achievement in medical school, whilst Wolkowitz and Kelley (2010) investigated how scores in science, mathematics, reading and English might forecast success in nursing school. That strong performance and high achievement in medical school are predictive of success in postgraduate surgical examinations is, perhaps, unsurprising. What is important, however, is that the current study by Ellis et al. is now considered as a foundation from which to assist surgical trainees to better prepare for and achieve favourable outcomes in mandatory postgraduate examinations. This will, in turn, enable them to progress in their careers and to, ultimately, provide optimum patient care.

  The General Medical Council (GMC) allows the UK’s 35 medical schools to develop and administer their own assessments, but it requires that all newly qualified doctors can demonstrate the essential objectives and requirements outlined in its ‘Outcomes for Graduates’ (GMC, 2018). Since a standardised assessment practice across the UK’s medical schools does not exist, medical students throughout the UK are assessed differently. As the paper states, this is a reason for which the UK Medical Licensing Assessment (UKMLA) will be introduced into all British medical schools in the academic year 2024/2025. All final year medical students in the UK will need to pass this licencing test in order to be registered with the GMC as a medical practitioner.

  It is crucial to highlight that this variability in assessment practices questions the reliability of the use of educational performance measurement (EPM) scores as a predictor of success in the MRCS examinations. Referencing Devine et al. (2015), Ellis et al. acknowledge that the ‘volume, type and intensity’ of assessment vary between medical schools, and they appreciate that such distinctions in assessment practices could culminate in final year medical students being disadvantaged when their EPM score is compared to that of their peers who attended other medical schools in which assessment practices were less stringent. Additionally, the paper aptly mentions that medical schools rank their final year medical students internally, and graduates are, therefore, categorised in a norm-referenced manner that is local rather than national. However, we argue that it is imperative to bring to the fore the fact that a student’s EPM score will be shaped by the assessment practices that he or she encounters during his or her undergraduate medical studies, especially since each medical school engages in distinct assessment practices. Given that the assessment practices differ between medical schools, students awarded the same score by two different universities do not necessarily possess identical academic attributes, and the use of EPM scores to neatly categorise graduates from across the UK merits re-evaluation. In order to employ EPM scores as a comparative measure to determine if medical school performance predicts success in postgraduate examinations, greater consideration must be afforded to the heterogeneity of the EPM scores.

  We believe that the validity of the current study can now be strengthened with a consideration of the following three additional aspects:

  1. The way in which each medical school conducts summative assessments. The frequency of summative assessments, the way in which summative assessments are structured and marked, and how the marks obtained by a medical student throughout his or her undergraduate degree contribute to his or her EPM score vary between medical schools. An examination of these aspects will offer a greater understanding of the correlation between EPM scores, which are shaped by the type, frequency and magnitude of the assessments administered by individual medical schools, and success in the MRCS examinations.

  2. The overall ability of the cohort in each of the individual years under review. Since each cohort will, inevitably, be stronger or weaker than the previous group, an exploration of a particular cohort’s ability will enable the EPM score as a predictor of success in postgraduate examinations to be appropriately placed.

  3. The ways in which EPM scores can be standardised across medical schools. The identification of methods that will allow summative performances through medical school to culminate in a standardised outcome will make EPM scores a more valid predictor of success in postgraduate examinations.

  References:

  Devine O.P., Harborne A.C., and McManus I.C. 2015. Assessment at UK medical schools varies substantially in volume, type and intensity and correlates with postgraduate attainment. BMC Medical Education, 15:146.

  General Medical Council. 2018. ‘Outcomes for Graduates’. https://www.gmc-uk.org/-/media/documents/outcomes-for-graduates-2020_pdf...

  Lipton, A., Huxham, G.J. and Hamilton, D. 1984. Predictors of success in a cohort of medical students. Medical Education, 18:4, 203-210. DOI: 10.1111/j.1365-2923.1984.tb01011.x

  OECD. 2018. Equity in Education: Breaking Down Barriers to Social Mobility. Paris: OECD Publishing. DOI: https://doi.org/10.1787/9789264073234-en.

  Wolkowitz, A.A. and Kelley, J.A. 2010. Academic Predictors of Success in a Nursing Program. Journal of Nursing Education, 49:9

  Show Less

  Conflict of Interest:
  None declared.

  • Back to top