Additional data analyses
| Component of assessment utility being evaluated | Method of analysis |
| Reliability: internal consistency | Cronbach’s alpha coefficient for each type of question compared using the method of Feldt17; the Spearman-Brown formula was then used to estimate the number of questions of each type required for an alpha of 0.8.18 |
| Cost: time taken to mark VSAs | The total minutes of consultant time required to mark the VSA, costed at the 2016/2017 hourly rate for a hospital consultant (including on-costs and overheads) of £10819 ($143). |
| Potential educational impact: effect on pass/fail rates | Cohen standard setting20 applied to both VSA and SBA total scores; pass/fail decisions for the two assessments were then compared using Cohen’s kappa. |
| Question discrimination | Pearson correlation coefficient (point-biserial) between students’ scores on each question and those on all other questions combined (item–rest correlation) for each type of question; the difference between question types was compared using a Wilcoxon signed-rank sum test (for paired, skewed data). |
SBA, single-best-answer; VSA, very-short-answer.