Discussion

In this study, we were able to show that the three main selection measures (SJT score, EPM and educational achievements) used in ranking applicants to the UK foundation programme were related to the odds of successful completion of this stage of training. However, the relationship between educational achievements and completion was not independent of the other two measures. In contrast, the SJT score appeared to offer some degree of incremental predictive validity over that provided by the EPM deciles, suggesting that it is capturing additional, and relevant, information on applicants as intended by its developers. As such, our findings add to the emerging evidence of the potential usefulness and validity of SJTs in medical selection across different career stages and for different clinical specialisations in relation to a number of educationally and clinically relevant outcomes. 24 25

Our findings in relation to trainee age are in keeping with those reported by an earlier study by Pyne and Ben-Shlomo,26 who reported that older doctors in their sample of specialty trainees were more likely to have problems with progression at ARCP than their younger colleagues. The relationship between disability and ARCP has not previously been reported on; here, we found that those trainees who had a disability (any category) as recorded by HESA were less likely to obtain an outcome 6.

Reflecting on our key findings, it is perhaps unsurprising that the educational achievements measure was not independent of the other measures. This metric would have been skewed by the relatively small number of students who had participated in research projects, often as part of intercalated degrees. Such students would also have been likely to have been ranked relatively highly, according to their medical school EPMs. It is also worth commenting that, at first glance, both the EPM deciles and the SJT scores appear equally predictive of completion of the foundation programme. However, when we attempt to place both measures, although be crudely, on the same scale (ie, divided into deciles), it is clear that EPM deciles are more predictive of this outcome, compared with SJT scores, with a ratio of roughly 2:1. Of course, the situation is complicated by the fact that EPM deciles are locally derived measures, while SJTs are nationally standardised tests. There may be ways in future research that the EPMs can be adjusted to make them more nationally comparable using ‘Peer-Competition Rescaling’.27 Nevertheless, despite the local nature of EPM deciles this finding remains relevant to policy as both SJT scores and EPM deciles are used in the national selection process. It is also known that SJTs are generally encountered as relatively easy tests by candidates, and therefore most of the information is available on those below the average level of performance.12 This infers that the SJT scores are likely to be relatively poor at differentiating more highly performing candidates from each other. We noted that once the scores were divided into deciles, this would have inevitably led to some loss of information, and resulted in a reduced degree of statistical significance in the relationship between this predictor and the outcome of interest, with the p value increasing from 0.01 to 0.02 when entered into a multivariable model with the other selection measures. In contrast, EPM deciles may be able to differentiate between both low and high performing candidates. This proposition is supported by our findings illustrated in figure 2: the centre portion of the graph is relatively flat with the slopes being steepest at the extreme ends of lower and highly EPM decile ranked applicants. We also noted that, in contrast to EPM, SJT scores were not independently and statistically significant predictors of outcome 6 s once three other background variables were put into a multivariable model (disability, number of applications and age). However, as applicant ranking does not take into account these latter three factors, this finding does not have direct importance to policy. Furthermore, it does suggest that at least some of the variance in SJT scores may be associated with these variables, although the directionality of influence could not be established from these data. Nevertheless, it is known that some demographic factors (such as female sex and ethnicity) are associated with SJT performance in general28 and therefore the use of such selection measures may have an indirect effect on the advantage (or disadvantage) experienced by certain subpopulations of applicant.

This was a relatively complete national dataset with few missing data. However, a number of limitations of the study are worth noting. First, the UKMED phase I cohort does not include graduates from non-UK medical schools who apply to foundation training. However, from 2015 onwards the non-UK students have been included in the data UKMED receives on foundation applications, permitting research into this area. Second, suboptimal ARCP outcomes are relatively rare in the foundation programme period of training. Unlike later periods of postgraduate medical training, there are fewer categories of ARCP outcome, and therefore at foundation stage they are a relatively information poor variable, picking up only cases where issues impacting performance were severe enough to prevent the doctor completing the programme. Moreover, there were no data on which trainees received remedial support during the foundation programme were available for this study. Thus, it is possible that a number of doctors who received outcome 6 s had required, sometimes considerable, support in order to achieve this. To establish whether the SJT and EPM deciles predict which trainees required additional support from their foundation schools would require the UKMED to have identifiable data on doctors in difficulty on the foundation programme. The UKFPO annual reports give figures for the numbers of doctors in difficulty: in 2014 there were 186 F1s and 163 F2s from UKMED medical school monitored via foundation schools’ doctors in difficulty (DiD) policies and processes. In 2015, there were 251 F1s and 210 F2s monitored via these policies.29 We do not know which ARCP outcomes these doctors were finally awarded. Similarly, at the other end of the scale, the outcome 6 category does not capture performance that exceeds that which is required to achieve the foundation competencies. Thus, in psychometric terms, the outcome would have generally yielded information on trainees at the lower end of performance. That is, the results may tell us relatively little about candidates who score relatively highly on the three selection measures.

More variance may also be present in the end of placement reports completed by trainees’ clinical supervisors in the e-portfolio each trainee is required to maintain. A score derived by summing across the six to eight clinical supervisor reports that are completed over the course of the foundation programme may provide a more granular outcome measure. Such an approach may be considered closer to the criterion used by Patterson et al, who employed supervisor ratings of trainee job performance at 1 year into training as an outcome to validate selection measures for entry to GP training against.7

Some variation in ARCP outcomes across foundation schools was noted. Moreover, EPM is a local rather than national measure. However, the use of a cross-classified multilevel model should have controlled for these potential clustering effects on the outcome of interest.

The present study relied on data that were already routinely collected and not specifically designed as a criterion to assess the validity of the selection measures. As Austin and Villanova noted using a general overall performance construct such as successful completion of the programme may not allow adequate matching to the predictive measures and it may be more fruitful to match selection measures to particular criteria representing particular aspects of job performance.30 Nevertheless, trainees undertaking the foundation programme and those responsible for managing the programmes are likely to regard it as an important criterion even if fails to capture the full range of foundation doctors’ performance. Other possible criteria include the multispecialty recruitment assessments used for selection into the next stage of training including GP training programmes.31

Our findings have clear indications for selection policy into the foundation programme. In particular, they provide evidence to support the intuitive sense, expressed by some, that excessive weight may be given to the SJT scores within the ranking process.16 Rather our results would suggest a 2:1 weighting ratio between the EPM decile and the SJT score may be the optimum, if the aim is to rank and advantage candidates most likely to successfully complete the foundation programme. Policywise, our findings provide less evidence to support using the educational achievements score in calculating applicant rankings for the foundation programme. If the aim is to select candidates most likely to complete this phase of training then our results suggest that the educational achievements score does not add any incremental value beyond the EPM deciles or SJT scores. However, we are aware we have used a somewhat circumscribed outcome, that is, completion of the programme. It may be that the selectors wish to also advantage applicants, according to their academic records, which are most likely to be educationally, and perhaps clinically successful in their long-term careers. If removal of the educational achievements and an increase in the weighting of the EPM deciles were to be considered then an exercise could be undertaken to model whether such a change would have a significant impact on the numbers of applicants whose rank changes to an extent that they would have been offered a different unit of application. Such weightings would be more in-line with research in selection outside of medicine, where measures of ability are more strongly predictive than personality-type measures. Schmidt and Hunter reviewed meta-analyses of selection methods: when predicting performance in job training programmes they report a mean correlation coefficient of 0.56 for cognitive ability, whereas for integrity and conscientiousness tests they report correlation coefficients of 0.38 and 0.30.32 One might argue that the EPM measure will inevitably reflect cognitive ability while the SJT is more akin to an integrity or personality test.

Further research could focus on the extent to which these foundation selection measures predict long-term success and career choices in medicine. Moreover, there is an immediate intention to further explore and describe the types of disability reported by medical graduates in a forthcoming UKMED project: UKMEDP54 declared disability in the UKMED dataset 2002–2016: an exploratory descriptive analysis. 33

In conclusion, the continued use of the SJT in selection into the foundation programme is justified by these findings, although it may be that excessive weight is being placed on the score, relative to the other two measures. Moreover, the UKFPO could consider discontinuing the use of educational achievements for ranking applicants to the foundation programme, unless it is shown to be associated with more distal markers of success in medical training.