What is the effect of secondary (high) schooling on subsequent medical school performance? A national, UK-based, cohort study

Objectives University academic achievement may be inversely related to the performance of the secondary (high) school an entrant attended. Indeed, some medical schools already offer ‘grade discounts’ to applicants from less well-performing schools. However, evidence to guide such policies is lacking. In this study, we analyse a national dataset in order to understand the relationship between the two main predictors of medical school admission in the UK (prior educational attainment (PEA) and performance on the United Kingdom Clinical Aptitude Test (UKCAT)) and subsequent undergraduate knowledge and skills-related outcomes analysed separately. Methods The study was based on national selection data and linked medical school outcomes for knowledge and skills-based tests during the first five years of medical school. UKCAT scores and PEA grades were available for 2107 students enrolled at 18 medical schools. Models were developed to investigate the potential mediating role played by a student’s previous secondary school’s performance. Multilevel models were created to explore the influence of students’ secondary schools on undergraduate achievement in medical school. Results The ability of the UKCAT scores to predict undergraduate academic performance was significantly mediated by PEA in all five years of medical school. Undergraduate achievement was inversely related to secondary school-level performance. This effect waned over time and was less marked for skills, compared with undergraduate knowledge-based outcomes. Thus, the predictive value of secondary school grades was generally dependent on the secondary school in which they were obtained. Conclusions The UKCAT scores added some value, above and beyond secondary school achievement, in predicting undergraduate performance, especially in the later years of study. Importantly, the findings suggest that the academic entry criteria should be relaxed for candidates applying from the least well performing secondary schools. In the UK, this would translate into a decrease of approximately one to two A-level grades.

shows the distribution of the entrants total UKCAT scores across the 18 medical schools in the different universities. The distribution of the total UKCAT scores seem to differ widely. This may be partly explained by the fact that different medical schools use the UKCAT differently in the selection process. Some use the UKCAT as a "borderline method" (to discriminate amongst a small number of applicants lying at a decision borderline, who are otherwise indistinguishable on the medical school's other selection criteria), or "factor method" (an applicant's UKCAT score or a proxy for that score is added to the score the applicant obtains in the medical school's usual method of selection, to provide a total score), or "threshold method" (minimum or threshold UKCAT score is adopted to create a hurdle that an applicant must cross to reach the next stage in the selection process) or "rescue" (to compensate for an applicants who would otherwise be rejected on account of their score on other selection criteria) [4].  Table 1 shows the same information depicted in Figure 1 by means of summary statistics.
To determine whether the distributional differences at university level may also be a factor of the quality of secondary school attended by a medical school entrant in a university, the total UKCAT matriculation scores were categorised into three (ranked) groups based on the standardised average performance of secondary schools attended by the entrants.  tively. The "[" and "]" indicate the limit is included in the group. The respective number of observations in the ranked groups were 622, 619 and 614 respectively. As may be observed from these values, the (ranked) groups had somewhat an equal number of ob-servations. There were 252 observations that were ungrouped due to missing values in average school level performance. For each of the groups, the corresponding standardised UKCAT matriculation scores were examined. The distribution of the total UKCAT matriculation scores for the three ranked groups are shown in Figure 2. The lowest UK-CAT performance was observed for entrants who attended secondary schools in group 1.
The distribution of the total UKCAT matriculation scores in this group seemed differentiated from the other two groups. The secondary schools represented in group 2 and 3 did not seem differentiated from each other in terms of total UKCAT matriculation scores of medical school entrants who attended them. To statistically confirm the trend observed in Figure 2, a one-way anova was conducted.
The factor of interest was the group which had an ordered level of 1, 2 and 3 based on average secondary school level performance as already described. Following a statistically significant mean difference (p-value < 0.001) in the total UKCAT matriculation scores between the groups, the Tukey's multiple group comparison was conducted. This was done to determine the full extent and direction of the differences between the groups. Table 2 shows the results of this comparison which confirm the observed trend in Figure 2.
Compared to group 1, the total UKCAT matriculation scores were higher for entrants who attended secondary schools in groups 2 and 3. There was no evidence that total UKCAT matriculation scores differed for entrants who attended secondary schools in groups 2 and 3.     Table 3 shows the predictive validity of the total UKCAT score and PEA as estimated by bivariate Pearson correlation coefficients. Generally, the predictive validity of PEA was higher than that of the total UKCAT score. It was also observed that the predictive validity for the knowledge-based outcomes were higher than that for skills-based outcomes. The predictive validity of both the total UKCAT score and PEA was highest in the first two years of medical school training.  In order to obtain a single metric of scholastic (or academic) ability from the reported GCSEs and A Level exam scores, a novel approach described by McManus et. al [1] which involved conceptualising educational achievement as a latent variable was used.
Thus PEA was estimated as a latent trait via an ordinal factor analysis using the most commonly taken A-level (both A1 and A2), and the grades obtained (e.g. A, B, C etc) used as (ordered categorical) indicators (see Table 4). The non-hierarchical version of McDonald's Omega was computed from the polychoric correlation matrix, since the factor analysis was of first order [2,3]. The non-hierarchical McDonald's Omega was found to be 0.91. Full Information Maximum Likelihood (FIML) which maximizes use of the available data was used for the analysis to deal with missingness in the data (e.g. for the subjects not taken by a particular candidate). Subsequently, factor scores were then estimated for all applicants in the data, the results of the factor analysis from Mplus are displayed on   Table 5: Results from the factor analysis for the derivation of factor scores for PEA 3 Mediation analyses

Single-level simple mediation analyses
It was aimed to determine the extent to which an entrants PEA would mediate the predictive power of the UKCAT for two separate domains (knowledge and skills) over the period of undergraduate training. To accomplish this a mediation model was considered. This is because, the overall total predictive power of the UKCAT for knowledge and skills-based undergraduate medical school exams would be partitioned into direct and indirect predictive power. This would then enable the accurate assessment of the relative, and unique, contribution UKCAT scores makes within the selection process. To demonstrate how this is done, consider Figure 4, which shows a simple mediation model. The term "simple" means that there is a one predictor, one mediator and one outcome variable under consideration. proportion of this non-unique contribution, which is the portion of the predictive power of the UKCAT that is explained by PEA, may be expressed as a * b c (see Figure 3 in text of main paper) where c is the total effect which has been shown to be equal to sum of the indirect and direct effects The significance of the indirect effect may be obtained by testing the hypothesis H 0 : a * b = 0 versus H 0 : a * b = 0, traditionally, this was done by assuming a normal distribution for the indirect effect of a * b thus necessitating the use of wald, score or likelihood ratio test with their corresponding p-value. This however, may lead to incorrect conclusions, when the indirect effect is not normally distributed as is often the case [5].

Multi-level simple mediation analyses
The structure of the data used for the study was hierarchical (clustered) because the outcomes (knowledge and skills) considered in each year of undergraduate training were nested within the 18 universities. This means that fitting a simple mediation analysis which essentially ignored the hierarchical structure of the data would potentially result in total, direct and indirect effects with induced attenuations. This may then lead to biased conclusions. For this reason, a multi-level mediation model was considered. In a nutshell, this model constitutes fitting a simple mediation for each cluster (university) separately and subsequently pooling the effects of interest together in some defined way to form population average total, population average direct and population average indirect effects.
A conceptual representation of this model may be viewed on Figure 5.
The subscript i denotes a student and subscript j a particular university. Further, ε PEA i j and ε K i j are level-1 residuals for the mediator PEA and Knowledge based outcome of interest respectively. Finally, d PEA j , d K j , a j , b j and c j are the random intercepts and slopes of the models. The assumptions of the 1→ 1→ 1 hierarchical mediation model are as follows 1. The predictor, UKCAT i j is uncorrelated with all the random effects ( d PEA j , d K j ,a j , b j and c j ) and the residuals (ε PEA i j and ε K i j ) in the model.
2. The residuals from the models, ε PEA i j and ε K i j , are each normally distributed with an expected value of zero and are uncorrelated with one another.
3. The level-1 residuals, ε PEA i j and ε K i j are uncorrelated with random effects d PEA j , d K j , a j , b j and c j in the model. 4. The random effects are normally distributed with means equal to the average effects in the population. This may be expressed as, and E(c j ) =c j = c for the slopes of interest. Further, the random effects covary with one another.
5. The distributions of PEA i j is normal conditional on UKCAT i j and K i j normal conditional on PEA i j and UKCAT i j .
These assumptions lead to the following matrix formulation of the model. Note that, it is possible to estimate the average of effects (which may be referred to as "population level effects", quantify the effects across all universities and their corresponding variabilities) The average mediation (indirect) effect and average total effects may then be estimated by making use of equations 3.5 and 3.6 respectively.
The multi-level simple mediation model was fitted in Mplus and the estimates of average total, average indirect and average direct effects estimated from equations 3.5 and 3.6.
The significance of the average total and average direct effects were obtained from the results in Mplus. To determine the significance of average indirect effect, a Monte Carlo 95% Percentile CI was programmed in R software by sampling 10,000 observations from the distribution in equation 3.7.
The individual elements of the distribution in equation 3.7 were obtained from the results of the multi-level mediation model in Mplus using the TECH 3 output command. Each of the 10,000 observations sampled for a, b and σ a j b j were plugged into equations 3.5 and 3.6 to obtain 10,000 average indirect effect values. Subsequently, the Monte Carlo 95% distribution of the 10,000 estimates for indirect effect. Figure 6 shows the plotted results from the models. It was observed that there were statistical significant average indirect effects in the first four years of undergraduate training of medical school for both knowledge and skills-based exams outcomes. The indirect effects represent the contribution of PEA towards the predictive power of the UKCAT. It was also observed that the range of the CIs widened in the third year onwards which is indicative of the missingness observed in the later years of the study (see Figure 1 and Table 1 in main text of the paper) which led to little information available for analysis in each of the university clusters in the data.

Choosing between single-level and multi-level simple mediation analyses
The multi-level mediation model fitted in section 3.2 is prone to convergence difficulties and is highly susceptible to missing data related problems. For instances where there are high attrition rates in later years of a longitudinal cohort study, it is highly likely that some or most of the clusters may have little or no data to contribute meaningfully to the analysis and this may further risk a lack of convergence. Therefore, for a given estimation problem, a single-level mediation model is preferred if there is evidence that there are no statistically significant clustering effects in the data.
To determine whether there were statistically significant clustering effects in the data equations 3.5 and 3.6 were considered. Note that from equation 3.5, when σ a j b j = 0, the resulting average indirect effect is equal to what would be estimated in a single-level simple mediation analysis in section 3.1. Therefore in seeking to determine whether a single or multi-level mediation analysis should be fitted to the data, it will be sufficient to test the hypothesis, H 0 : σ a j b j = 0 versus H 1 : σ a j b j = 0. Evidence in favour of the null hypothesis would also be evidence in favour of a simple single-level mediation analysis.
The results of the hypothesis test were available as part of the multi-level results in Mplus and are displayed on Table 6. It was observed that all of the p-values were > 0.05 implying that there were statistically non-significant clustering effects in the data. Further, Intra Cluster Correlations (ICCs) for the models computed by utilising the main diagonal of the covariance matrix from equation 3.7 and the residual variances from the model are displayed on Table 7. The observed ICCs (7 th and 13 th column of the Table) indicate that the proportion of variability explained by the multi-level mediation models is negligible. Therefore a simple single-level mediation model is appropriate for the data.
Following the results on Tables 6 and 7, a simple single-level mediation model was fitted using two models, for the case of knowledge-based exams outcomes, shown in equation (3.9) and (3.8) respectively using the same notation as in section 3.2.
PEA i = I PEA + a * UKCAT i + ε PEA (3.8) Knowledge-based exams Skills-based exams Academic year σ a j b j Std. Error P-value σ a j b j Std. Error P-value      The results from Tables 8 and 9 Table 10 shows the results of the statistical test conducted. It was observed that there were statistically significant differences in the proportions of the total UKCAT scores explained by the PEA between the knowledge and skills-based exams outcomes in all but the fifth year of medical training. It is was also observed that the fifth year of medical school training had very low sample sizes for the two outcomes under consideration. This contributed to a lack of sufficient power to detect differences in the proportions in that year.

Multi-level linear model
To address the second aim of the study, which was to appraise the influence of the performance of the previous secondary school attended on an undergraduates achievement in medical school, a multi-level linear model or Linear Mixed Model (LMM) was used due to its capability to handle clustering in instances where the outcomes are continuous and correlated. The term "mixed" in the Linear Mixed Model comes from the fact that the model estimates both fixed (mean structure) and random effects (random structure). The modelling framework of Linear Mixed Model may be expressed as follows: with b 1 . . . b N and ε 1 . . . ε N being independent. Y i is the n i -dimensional outcome (knowledge or skills-based exams), X i and Z i are the design matrices for the fixed and random effects of known predictors respectively, β and b i are fixed and university specific effects respectively, and ε i is the vector containing the residual components [9]. X i is a design matrix containing the predictors; average school level performance of the school in which an entrant sat for their A-level exam, an entrant's reported A-level grade (AAA, AAB,

ABB, BBB or BBC), interaction between average school level performance and reported
A-level grades and the tier of an entrant's secondary school as categorised based on their performance (see Figure 2). Z i is a design matrix containing a random intercept which modelled the correlation in the outcomes within a university by allowing the (predicted) outcomes to vary between universities.
As seen in Table 11, the effect of the secondary school group (ordered based on their performance as 1, 2 or 3) was not statistically significant. This implies the A-level grades earned by an medical school entrant and the average level performance of secondary school attended are sufficient in explaining the undergraduate medical school outcomes.
Further categorisation of secondary schools based on their performance adds no value in explaining undergraduate medical school outcomes. Therefore the proposed model fitted was in line with the predictors shown in Table 12.

Single-level simple mediation analyses
For the single-level simple mediation analysis, the models were fitted after imputation was conducted 30 times thus creating 30 datasets. These datasets were analysed and results later summarised through pooling of the estimates. The computation of associated standard errors of their estimates was also done. The MI was conducted in SAS using the Monte Carlo Markov Chain (MCMC) which imputes the missing values in the data in a way that retains the overall mean and covariate structure of the data assuming a joint multivariate normal distribution [10,11]. The results of the previous non-imputed data displayed in Table 8 and 9 for both knowledge and skills-based exams are further displayed in graphical form in Figure 7. These were compared to the results from the multiply imputed data which are found on Figure 8. It was observed that in as far as the aim of the analysis was concerned, there were no discernible difference in the estimates and conclusions regarding the indirect effects of UKCAT through PEA for both the knowledge and skills-based outcomes from both the multiply imputed and non-imputed data. This implies that the assumptions of ignorability and MAR were plausible and that the missingness though severe in later years of the study, did not adversely effect the results and conclusions of the statistical analysis. This is expected as the missing data was created when participating medical schools failed to submit outcome data the UKCAT database in a that particular year. Thus, it may be concluded that the missing data was unlikely to threaten the validity of the inferences drawn from the results. The results from MI data were compared to those from the original data shown in Figures   4 and 5 in the main text of the paper for both knowledge and skills-based exams outcomes.
The comparison revealed that the missingness did not an adverse effect on the analysis.
Like in the original unimputed data, for both knowledge and skills-based exam outcomes, at each level of average school level performance students with higher grades tend to perform better compared to their counterparts with lower grades throughout undergraduate medical school. Overall, compared to students from schools with high average school level performance, students from schools with low average school level performance tend to have better scores in both knowledge and skills-based exam outcomes throughout undergraduate medical school. This suggests that the assumption of MAR invoked for the study was plausible.

Year 5
Average school level performance MI predicted standardised score Grade legend AAA AAB ABB or lower

Year 5
Average school level performance MI predicted standardised score Grade legend AAA AAB ABB or lower Figure 10: Multiply imputed effect of average school level performance by reported grades on undergraduate medical school skills-based exams