Observational cohort study with internal and external validation of a predictive tool for identification of children in need of hospital admission from the emergency department: the Paediatric Admission Guidance in the Emergency Department (PAGE) score

Objectives To devise an assessment tool to aid discharge and admission decision-making in relation to children and young people in hospital urgent and emergency care facilities, and thereby improve the quality of care that patients receive, using a clinical prediction modelling approach. Design Observational cohort study with internal and external validation of a predictive tool. Setting Two general emergency departments (EDs) and an urgent care centre in the North of England. Participants The eligibility criteria were children and young people 0–16 years of age who attended one of the three hospital sites within one National Health Service (NHS) organisation. Children were excluded if they opted out of the study, were brought to the ED following their death in the community or arrived in cardiac arrest when the heart rate and respiratory rate would be unmeasurable. Main outcome measures Admission or discharge. A participant was defined as being admitted to hospital if they left the ED to enter the hospital for further assessment, (including being admitted to an observation and assessment unit or hospital ward), either on first presentation or with the same complaint within 7 days. Those who were not admitted were defined as having been discharged. Results The study collected data on 36 365 participants. 15 328 participants were included in the final analysis cohort (21 045 observations) and 17 710 participants were included in the validation cohort (23 262 observations). There were 14 variables entered into the regression analysis. Of the 13 that remained in the final model, 10 were present in all 500 bootstraps. The resulting Paediatric Admission Guidance in the Emergency Department (PAGE) score demonstrated good internal validity. The C-index (area under the ROC) was 0.779 (95% CI 0.772 to 0.786). Conclusions For units without the immediate availability of paediatricians the PAGE score can assist staff to determine risk of admission. Cut-off values will need to be adjusted to local circumstance. Study protocol The study protocol has been published in an open access journal: Riaz et al Refining and testing the diagnostic accuracy of an assessment tool (Pennine Acute Hospitals NHS Trust-Paediatric Observation Priority Score) to predict admission and discharge of children and young people who attend an ED: protocol for an observational study. BMC Pediatr 18, 303 (2018). https://doi.org/10.1186/s12887-018-1268-7. Trial registration number The protocol has been published and the study registered (NIHR RfPB Grant: PB-PG-0815–20034; ClinicalTrials.gov:213469).

recession were combined.

Continuous variables
During the regression modelling the fractional polynomial command made suggestions for the transformation of continuous variables. Age was included as reciprocal fractional polynomial of the original variable. Temperature and respiratory BMJ Publishing Group Limited (BMJ) disclaims all liability and responsibility arising from any reliance Supplemental material placed on this supplemental material which has been supplied by the author(s) For heart rate and oxygen saturation the plots of the fractional polynomials laid over the data were examined to visually assess how the models fit. These plots and the findings from the exploratory analysis jointly led decisions on the ultimate form of each of the variables.
• Heart rate: The fractional polynomial suggested a linear association with admittance. Examining the plot revealed that although a linear relationship is broadly appropriate it does a poor job of capturing the two large dips in probability of admittance. A more complex plot would have captured better the association between heart rate and admittance. With the large sample size such a curve could have been fitted without being overly concerned about overfitting. However, this would have defeated the overall aim of an easily used paper-based point scoring tool by requiring multiple coefficients in the final model. Heart rate was categorised as under 75, 75 to 125 and over 125.
• Oxygen saturation: The multivariable fractional polynomial modelling suggested that the raw oxygen saturation variable taken to the power of -2 was the best fitting form. On examination of the fractional polynomial plot it was clear that the plot fitted the data very well for oxygen saturations up to 90%, but above 90%, where 99% of the data lay, the polynomial failed to capture the more erratic association between saturation and admittance.
Accordingly, a decision was taken to categorise the oxygen saturation variable based on the frequencies of the responses at each saturation BMJ Publishing Group Limited (BMJ) disclaims all liability and responsibility arising from any reliance Supplemental material placed on this supplemental material which has been supplied by the author(s) percentage, its observed relationship with admittance from the graph and on the previous PAT-POPS model (which aligns with clinical opinion of oxygen saturation cut-offs): 95%-100%; 90%-94%; <90%.
Being aware of the statistical implications of categorising continuous variables these decisions were not made lightly, however it was clear that no fractional polynomial would be able to capture the nature of the relationship better than categorisation. Figure A shows the internal validation calibration plot. The blue 'lowess' line fitting the reference line so well is a demonstration of how well the model is fitting the data.

Internal validation
Given the large sample size however this is to be expected.

Model reproducibility or transportability
The C-index for this membership model was 0.8078. The closer this number is to 1 the more likely it is that this represents model transportability rather than model reproducibility. So, in carrying out external validation using the Site 2 dataset an assessment is being made of whether the model works well in a different but related population. This was as expected given that from the exploratory analysis it could be seen that Site 2 represent a different mix of ethnicities as well as generally older and healthier patients.

Calibration and discrimination
The Brier score was lower in the Site 2 data (0.0653) than in the model development dataset, suggesting very good prediction accuracy in this external dataset. At 0.9864908 (0.896986 to 1.075996) the calibration slope was also close to the ideal of 1. Calibration-in-the-large (CITL) was less impressive at -1.218977 (-1.305886 to -1.132068). This suggests that the predicted probabilities were higher than the observed proportions. So, children in Site 2 were less likely to be admitted than the model predicted; a child with equivalent characteristics presenting at Site 1, for example, would be more likely to be admitted. This can be seen in the calibration plot ( Figure B) where the observed probabilities are consistently lower than the expected probabilities, especially as the expected probabilities near 1. This was also evident in the E/O ratio of 2.7. This is the expected/observed ratio, how many were expected to be admitted based on the model versus how many actually were BMJ Publishing Group Limited (BMJ) disclaims all liability and responsibility arising from any reliance Supplemental material placed on this supplemental material which has been supplied by the author(s) admitted, so ideally this would be 1. That it is higher here suggests less people were admitted than would have been expected from the model. In terms of discrimination, the C-index was 0.7626 (0.74244 to 0.78279) (Figure B).

Model reproducibility of transportability
The C-index for this membership model was 0.8154. This suggests that Site 1 and Site 3 are also quite different from each other in terms of their populations and therefore model transportability was assessed with the following results. That these two sites differ is expected; aside from the difference in ethnic make-up of the populations, Site 1 is an ED whereas Site 3 is an Urgent care centre and therefore sees healthier children.

Calibration and discrimination
As with Site 2, when the model was applied to Site 3 the Brier score was a very low 0.0732. The calibration slope was over 1 at 1.075, this time suggesting overfitting, but as with Site 2 the CI also crossed 1 (0.976 to 1.123). The CITL was -1.055 (-1.123 to -0.987) suggesting predicted probabilities were higher than the observed proportions, again evident in Figure C. The E/O ratio was 2.3. The C-index was 0.7533 (0.734 to 0.770).

Overall
From these results it can be determined that the model transportability is good overall. With only slight over/under-fitting present at each of the external validation sites and C-indexes only modestly smaller than was found in the original model. It is