Development and validation of automated computer-aided risk scores to predict in-hospital mortality for emergency medical admissions with COVID-19: a retrospective cohort development and validation study

Objectives There are no established mortality risk equations specifically for unplanned emergency medical admissions which include patients with SARS-19 (COVID-19). We aim to develop and validate a computer-aided risk score (CARMc19) for predicting mortality risk by combining COVID-19 status, the first electronically recorded blood test results and the National Early Warning Score (NEWS2). Design Logistic regression model development and validation study. Setting Two acute hospitals (York Hospital—model development data; Scarborough Hospital—external validation data). Participants Adult (aged ≥16 years) medical admissions discharged over a 24-month period with electronic NEWS and blood test results recorded on admission. We used logistic regression modelling to predict the risk of in-hospital mortality using two models: (1) CARMc19_N: age+sex+NEWS2 including subcomponents+COVID19; (2) CARMc19_NB: CARMc19_N in conjunction with seven blood test results and acute kidney injury score. Model performance was evaluated according to discrimination (c-statistic), calibration (graphically) and clinical usefulness at NEWS2 thresholds of 4+, 5+, 6+. Results The risk of in-hospital mortality following emergency medical admission was similar in development and validation datasets (8.4% vs 8.2%). The c-statistics for predicting mortality for CARMc19_NB is better than CARMc19_N in the validation dataset (CARMc19_NB=0.88 (95% CI 0.86 to 0.90) vs CARMc19_N=0.86 (95% CI 0.83 to 0.88)). Both models had good calibration (CARMc19_NB=1.01 (95% CI 0.88 to 1.14) and CARMc19_N:0.95 (95% CI 0.83 to 1.06)). At all NEWS2 thresholds (4+, 5+, 6+) model, CARMc19_NB had better sensitivity and similar specificity. Conclusions We have developed a validated CARMc19 scores with good performance characteristics for predicting the risk of in-hospital mortality. Since the CARMc19 scores place no additional data collection burden on clinicians, it may now be carefully introduced and evaluated in hospitals with sufficient informatics infrastructure.

The calculator integrated with the medical record will generate a number, but then what will be posted in the medical record? a text variable (low, intermediate, high)? Or a numerical probability of death? How frequently will the model need to be calibrated. You have tested your models many times now. How are they changing over time (connects with earlier point)?
Death in the patients included in the analysis occurred despite medical care. Do the authors worry that clinicians will withhold effective treatments if the model predicts death?
Did any of the patients have a terminal illness included? Is so, can the analysis rerepeated without them?
Consider also removing all the patient that died within 2-3 days of admission. These might have clear prognosis on admission and a model is not needed.
How does the model preform if the patients with extreme values are removed. Examples of extreme values are respiratory rate > 35, heart rate > 120, SBP < 80, etc. These are situations that are easy to identify by the clinician and a model is not needed to detect severity of illness. You can calculate the c statistic for patients above and below each of these value. You can also calculate the c for patients with no extreme values and patients with any. How did the model perform in COVID patients?

VERSION 1 -AUTHOR RESPONSE
Reviewer: 1 Dr. Julian Solis Garcia del Pozo, Albacete University Hospital Complex Comments to the Author: I thanks BMJ open to the opportunity to review this paper. The authors develop a computed aided-risk score to predict in-hospital mortality for hospital admissions, including COVID-19 patients. The authors performed the development of the score in York Hospital and the validation in Scarborough hospital. The prevalence of COVID-19 patients in the development dataset was 8.7%, and in the validation dataset, 11%. However, the title seems to reflect that the score has been developed exclusively for COVID-19 patients. The title should be improved to fit the content of the article better. On the other hand, I think that the methodology used is adequate, the manuscript is well written, and the information provided is relevant.
Response: Development and validation of automated computer aided-risk scores to predict in-hospital mortality for emergency medical admissions with COVID-19: a retrospective cohort development and validation study Reviewer: 2 Dr. Aiman Tulaimat, Cook County Health and Hospitals Comments to the Author: The authors attempt reexamine the CARM during the COVID pandemic. They have mastered this particular analysis technique. I do have few question to ask. Did they derive new coefficient for the various variables in the model? If yes, have they changed from prior models they have published? What does this change mean? What was the accuracy of the NEWS2?
Response: There are two major aims of statistical modelling: 1) causal inference (focussed on individual variable impact by inspecting the regression coefficients) and 2) predictive modelling (focused on the predictive performance of outcome in terms of discriminating the adverse outcome patients). In this study, we adopt a later approach and aimed to develop a model (CARMc19_N/NB) for predicting in-hospital mortality by updating the previous version (CARM_N/NB) for COVID-19 patients using NEWS2 data. We found the CARMc19 models are better than previous models in terms of discrimination (c-statistic or AUC).

NEWS (_N)
NEWS + Blood ( What is the threshold used to calculate the sensitivity and specificity for the CARM? Was it the same at each NEWS2? Response: We calculated the predicted risk of death from a simpler logistic regression model (died~NEWS2) at three NEWS2 cut-offs (4, 5, 6). We used these predicted risk of death probabilities as thresholds for calculating the sensitivity and specificity.
But the ultimate question is, how will this be implemented? Response: CARMc19 is currently at the implementation stage. Please see the screenshot below that show how its implementation will look like. We nevertheless agree that sound implementation and rigorous evaluation remain key considerations.
The calculator integrated with the medical record will generate a number, but then what will be posted in the medical record? a text variable ( How frequently will the model need to be calibrated? Response: Our aim is to assess the model predictive performance (in terms of discrimination and calibration) annually or if there is any indication from medical staff that they feel the scores lack face validity.
You have tested your models many times now. How are they changing over time (connects with earlier point)? Response: The model predictive performance (in terms of discrimination and calibration) has been improved with the addition of COVID-19 status.
Death in the patients included in the analysis occurred despite medical care. Do the authors worry that clinicians will withhold effective treatments if the model predicts death? Response: CARM is comparable with medical judgements in discriminating in-hospital mortality following emergency admission to an elderly care ward (Faisal et al., 2019). CARM may have a promising role in supporting medical judgements in determining the patient's risk of death in the hospital. It is important to note that we have designed CARM to support the medical decision-making process, not replace it, without placing any additional data collection burden on staff. We nevertheless agree that such questions are very important and need further research. Consider also removing all the patient that died within 2-3 days of admission. These might have clear prognosis on admission and a model is not needed.