Introduction Accurate surgical risk prediction is paramount in clinical shared decision making. Existing risk calculators have limited value in local practice due to lack of validation, complexities and inclusion of non-routine variables.
Objective We aim to develop a simple, locally derived and validated surgical risk calculator predicting 30-day postsurgical mortality and need for intensive care unit (ICU) stay (>24 hours) based on routinely collected preoperative variables. We postulate that accuracy of a clinical history-based scoring tool could be improved by including readily available investigations, such as haemoglobin level and red cell distribution width.
Methodology Electronic medical records of 90 785 patients, who underwent non-cardiac and non-neuro surgery between 1 January 2012 and 31 October 2016 in Singapore General Hospital, were retrospectively analysed. Patient demographics, comorbidities, laboratory results, surgical priority and surgical risk were collected. Outcome measures were death within 30 days after surgery and ICU admission. After excluding patients with missing data, the final data set consisted of 79 914 cases, which was divided randomly into derivation (70%) and validation cohort (30%). Multivariable logistic regression analysis was used to construct a single model predicting both outcomes using Odds Ratio (OR) of the risk variables. The ORs were then assigned ranks, which were subsequently used to construct the calculator.
Results Observed mortality was 0.6%. The Combined Assessment of Risk Encountered in Surgery (CARES) surgical risk calculator, consisting of nine variables, was constructed. The area under the receiver operating curve (AUROC) in the derivation and validation cohorts for mortality were 0.934 (0.917–0.950) and 0.934 (0.912–0.956), respectively, while the AUROC for ICU admission was 0.863 (0.848–0.878) and 0.837 (0.808–0.868), respectively. CARES also performed better than the American Society of Anaesthesiologists-Physical Status classification in terms of AUROC comparison.
Conclusion The development of the CARES surgical risk calculator allows for a simplified yet accurate prediction of both postoperative mortality and need for ICU admission after surgery.
- surgical risk
- postoperative mortality
- icu stay
- risk prediction
- risk calculator
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Strengths and limitations of this study
The Combined Assessment of Risk Encountered in Surgery (CARES) model predicts risk for both 30-day mortality and need for intensive care unit (ICU) admission postoperatively with good accuracy. Prediction of the risk for postoperative ICU admission is novel and not currently available for most risk stratification tools. This adds to the utility of the model and aids in decision-making process and health resource planning.
The CARES surgical risk calculator comprises of simple and easily accessible variables available from routine preoperative evaluation for most surgeries.
It is the first risk stratification tool to incorporate the use of red cell distribution width, a novel haematological biomarker which has been shown to be of value in predicting mortality risk.
This is a retrospective study design.
CARES has only been validated in a single centre, hence, there is a need for further prospective studies to externally validate the tool in other institutions across the region.
Introduction and background
About 250 million surgeries are performed worldwide each year, and this number is increasing rapidly.1 As access to surgery improves, the number of patients with postoperative complications will also increase.2 3 Previous studies demonstrated that a large proportion of postoperative mortality occurs in a smaller, distinct group of patients with high-risk characteristics, and <15% from this group were admitted to intensive care units (ICU) postoperatively.4 5 In the preoperative assessment of a surgical patient, it is prudent to counsel the patient on the risks of postoperative mortality and need for critical care monitoring after surgery. Therefore, accurate preoperative prediction and stratification of surgical risks is becoming even more important for perioperative shared decision-making process, guiding allocation of resources and improving patient outcomes.
However, predicting postoperative risks and identifying patients at a higher risk of adverse events have traditionally been based on individual surgeon experience and augmented by published rates in the literature, either from single institution studies or clinical trials.6 Unfortunately, these estimates are typically not specific to an individual patient’s risk factors. Moreover, existing risk stratification tools have their own limitations. The currently available risk stratification tools, for example, American Society of Anaesthesiologists-Physical Status (ASA-PS), Physiological and Operative Severity Score for the enUmeration of Mortality and Morbidity (POSSUM), Surgical Outcome Risk Tool (SORT) and American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP) were all derived from the Western population and healthcare systems, from which Asians would serve as outliers in view of the their different socioeconomic, cultural, genetic makeup and healthcare systems. There is a paucity of surgical risk stratification models which is derived from or has been validated in the Asian population. This limits the uptake and applicability of these models in the region.
Furthermore, the individual risk stratification tools suffers from wide inter-user variability (ASA-PS),7 need for data which are not available during the preoperative period (POSSUM),8 9 lack of validation outside the derived population’s region (SORT, ACS-NSQIP) and the complexity of the model itself (ACS-NSQIP). Hence, the inertia to use them may be due to concerns over the accuracy, complexity and also the requirement for a large number of variables. To improve the performance of predictive models, there are recent interests in the use biomarkers such as B-type natriuretic peptides10 11 and cardiac troponin12 to predict mortality. However, these markers may not be easily available across all laboratories and most often are not part of the routine preoperative investigations. More recently, readily available haematological biomarkers such as red cell distribution width (RDW) and degree of anaemia (if present) have been shown to be associated with postoperative mortality risk.13–15 Incorporation of these biomarkers with preoperative clinical factors may improve the accuracy of a surgical risk model.
We aim to develop a locally derived, simple and accurate surgical risk calculator that consisted of readily available preoperative clinical and laboratory variables, which can predict both mortality and ICU admission with just a single set of variables. We hypothesise that the development of this risk stratification tool could help us accurately predict the risk of (1) 30-day postsurgical mortality and (2) requiring admission to ICU for >24 hours during the surgical admission, which may serve as a surrogate for major postoperative complications in the Singapore population.
Data source and patients
Institutional Review Board approval was obtained (Singhealth CIRB 2014/651/D) prior to the start of the study, which waived the requirement of individual informed consent. Retrospective data were collected and analysed from the electronic medical records of 90 785 patients aged 18 and older who underwent surgery under general or regional anaesthesia between 1 January 2012 and 31 October 2016 in Singapore General Hospital, a 1700-bedded tertiary academic hospital in Singapore. These clinical records were sourced from our institution’s clinical information system (Sunrise Clinical Manager, Allscripts, Illinois, USA) and stored in our enterprise data repository and analytics system (SingHealth-IHiS Electronic Health Intelligence System (eHINTS)). eHINTS collects reliable data on patient demographics, laboratories, comorbidities and 30-day postoperative outcomes for patients undergoing surgeries from all surgical subspecialties. It integrates information from multiple healthcare transactional systems including administration, clinical and ancillary systems. Mortality data on the system were synchronised with the National Electronic Health Records, ensuring a near complete follow-up. We excluded patients who underwent cardiac surgery, neurosurgery, transplant and burns surgery, and evaluated only the outcomes of the index surgery for patients who underwent multiple surgeries during the study period. After excluding patients with missing data, the final data set consisted of 79 914 cases (figure 1).
Procedures and definitions
Data collected include patient demographics as well as preoperative comorbidities and laboratory data (table 1). These are routine clinical and laboratory data that were electronically collected during the preoperative anaesthesia assessment visit. Priority of surgery (emergency or elective) and surgical risk classification were based on the 2014 European Society of Cardiology (ESC) and the European Society of Anaesthesiology (ESA) guidelines16 17 . Emergency cases in our hospital are classified as cases requiring operation within 24 hours, and they are further subcategorised into their degree of urgency. Missing data were excluded and complete case analysis was done.
Preoperative laboratory results including full blood count (FBC) and renal panel (RP) were taken as the latest blood results within 90 days before the surgery, and up to the day of surgery but before the start time of surgery. These results include preoperative haemoglobin, red cell distribution width levels and serum creatinine levels. The presence of anaemia was defined by WHO’s gender-based classification of anaemia severity.18 Mild anaemia was defined as haemoglobin concentration of 11–12.9 g/dL in males and 11–11.9 g/dL in females; moderate anaemia was defined for both genders to be haemoglobin concentration between 8–10.9 g/dL and severe anaemia defined as haemoglobin concentration <8.0 g/dL. Pre-existing chronic kidney disease, if present, is graded based on the estimated glomerular filtration rate by the Modification of Diet in Renal Disease equation according to 2012 Kidney Disease: Improving Global Outcomes (KDIGO) guidelines.19 Red cell distribution width (RDW) is reported as a coefficient of variation (percentage) of red blood cell volume with the normal reference range for RDW in this hospital laboratory to be 10.9%–15.7%. Levels >15.7% were defined a priori as high RDW, and this corresponded to the 89th centile of RDW values in our study population. A high RDW has been shown to be associated with an increased risk of mortality that is independent of the severity of anaemia.20 The chosen cut-off value of 15.7% was shown to have a sensitivity of 39.5%, specificity of 89.3%, positive predictive value of 5.3% and negative predictive value of 99.0%.13 The individual components of revised cardiac risk index were defined as per the original study by Lee et al,21 and the ASA status follows that of the ASA-PS definitions.22
All analyses were performed using IBM SPSS V.21.0. The data set (90 785 subjects) was randomly divided to 70%:30%, with the former used as the derivation cohort and the latter as validation cohort.
The Combined Assessment of Risk Encountered in Surgery (CARES) surgical risk calculator was developed using ORs of the risk variables obtained from the logistic regression for postsurgical 30-day mortality or ICU >24 hours within the admission. The ORs were assigned rank scores. The model was then validated on the 30% cohort. The Hosmer-Lemeshow (H-L) test for calibration was used to show the goodness of fit for the models developed.
In the initial development of the model, we looked at each outcome individually before combining the significant variables to create a combined risk prediction model for both mortality and need for postoperative ICU stay of >24 hours (ICU >24 hours). For the initial analysis, we included potentially significant variables for each outcome and performed stepwise multivariate logistic regression to obtain the minimum number of variables that retained the accuracy of prediction. The accuracy of prediction was estimated using the receiver operating curve (ROC) and the area under the receiver operating curve (AUROC).
The significant variables for each outcome were then combined to construct a single risk prediction model. We tested the accuracy of the model in predicting either mortality or ICU >24 hours using the ROC and AUROC. H-L tests were performed to show goodness of fit for the model. The performance of the CARES surgical risk calculator was then compared with that of the ASA-PS and the ASA-PS with propensity scoring to adjust for possible differences in the development of the models. Statistical significance was set at P<0.05.
From a total of 90 785 cases, 63 715 (70%) patients were randomly selected into the derivation cohort and the remaining 27 070 (30%) into the validation cohort. Observed mortality in the derivation and validation cohorts were similar, at 0.6%. 1.2% (770 patients) in the derivation cohort were admitted to ICU for more than 24 hours, while that in the validation cohort is 1.4% (375) patients. Descriptive data for these cohorts are summarised in table 1.
Model development and derivation
For mortality, 12 variables which were found to be significant in univariate analysis (age, surgical risk, race, anaemia, chronic kidney disease, RDW, presence of cerebrovascular accident, ischaemic heart disease, congestive heart failure, insulin-requiring diabetes mellitus, ASA status and gender) were included. The AUROC (the highest ever AUC to be obtained for this set of data) for this 12-variable black-box model was 0.931 (0.916–0.946). Stepwise logistic regression retained only seven significant variables (table 2) which produced the same AUROC of 0.931 (0.915–0.947) of the 12-variable black-box model.
Both the ORs and rank scores are presented in table 2. While rounded ORs are commonly used for risk prediction models, some of our ORs are too high and may skew the final score. Hence, to handle these ‘extreme scores’, a ranking system was developed for scoring in the calculator. Additionally, this final rank-based model facilitates utilisation of the risk calculator by keeping it simple enough for regular clinical consult use. This mortality model with an AUROC of 0.928 (0.912–0.945), compared with the original AUROC of 0.931 (0.915–0.947) from the OR-based model, does not compromise accuracy of the risk prediction and at the same time offers increased usability (even in health systems without regular electronic medical records use). The AUROC are shown in online supplementary appendix figure 1.
Supplementary file 1
The H-L test for calibration demonstrated good fit for the final model (P=0.79) predicting postoperative mortality for both the derivation and validation cohort (online supplementary appendix figures 2 and 3).
For the ICU >24- hour outcome, 13 variables which were significant in the univariate analysis (age, surgical risk, race, anaemia, RDW, presence of cerebrovascular accident, ischaemic heart disease, congestive heart failure, chronic kidney disease, insulin-requiring diabetes mellitus, ASA status, surgical priority and gender) were included. The AUROC (the highest ever AUC to be obtained for this set of data) for this 13-variable black-box model was 0.876 (0.861–0.890). Stepwise logistic regression retained seven significant variables (age, surgical risk, anaemia, congestive heart failure, ASA status) with AUROC of 0.873 (0.858–0.888). Table 3 shows the seven significant variables and their corresponding OR and rank scores.
For the similar reasons as with the mortality model above, a risk score model using the rank of the ORs was developed, with an AUROC of 0.867 (0.852–0.882), retaining accuracy of the risk prediction. The AUROC are shown in the appendix (online supplementary appendix figure 4).
The H-L test for calibration showed good fit for the final model (P=0.81) predicting need for ICU stay >24 hours in both the derivation and validation cohorts (online supplementary appendix figures 5 and 6).
Combined modelling for both mortality and ICU
To further increase the ease of use for CARES surgical risk calculator, we explored a combined model that is accurate in predicting both mortality and morbidity with just a single set of variables. The above results of separate modelling for each outcome predictions demonstrated robust and accurate individual model in predicting respective outcomes. We now combine the predictors to create a single model. This model consists of nine variables: age, surgical risk, anaemia, RDW, ischaemic heart disease, ASA, surgical priority, gender and presence of congestive heart failure.
The combined predictors are shown in table 4.
These OR-based nine variables yielded an AUROC of 0.936 (0.920–0.953) for mortality and 0.874 (0.859–0.889) for ICU. Using the rank scores, the AUROC are 0.934 (0.917–0.950) and 0.863 (0.848–0.878) for mortality and ICU, respectively, which again show that accuracy was not compromised. The ROCs are shown in figure 2. The corresponding H-L plots are available in the appendix (online supplementary appendix figures 7 and 8).
We tested the rank score model on the validation cohort. The AUROC was 0.934 (0.912–0.956) for mortality and 0.837 (0.808–0.868) for ICU >24 hours. The ROCs are shown in figure 3. The corresponding H-L plots are also shown in the appendix (online supplementary appendix figures 9 and 10).
The cumulative rank scores were subsequently categorised into different bands classifying the risk of mortality as low, low–moderate, moderate–high or high, and a corresponding mortality probability was assigned to each band (table 5). Clinical decision-making suggestions are included in the table.
Applicability of the CARES model for combined mortality and ICU prediction
We provide a hypothetical example of a patient to illustrate the application of the model. A 76-year-old Chinese man with history of hyperlipidaemia, hypertension, previous ischaemic heart disease with stent inserted with no evidence of congestive heart failure is scheduled for an elective right hemicolectomy for colorectal cancer. FBC shows a starting haemoglobin of 12 and an RDW of 12.1.
Using the scoring method, the patient would have a combined rank score of 8+5+2+3+7+2=28, which places him in the moderate–high-risk category predicting a 1.9% risk of 30-day postsurgical mortality and 4.9% risk of need for postoperative ICU stay for >24 hours.
Using these figures, we would be able to counsel the patient appropriately and plan for optimisation of modifiable risk factors preoperatively and appropriate perioperative monitoring and surveillance.
Comparison with ASA-PS
When the performance of combined model CARES was compared with both the ASA-PS and its propensity model, CARES performed better than both with an AUROC of 0.934 (0.917–0.950) for mortality and 0.863 (0.848–0.878) for ICU >24 hours (see table 6).
We developed and internally validated a simple, locally derived CARES surgical risk calculator comprising nine preoperative variables to predict both 30-day mortality and need for ICU (ICU admission for >24 hours), respectively, in adults undergoing non-cardiac and non-neurological surgery. The nine variables are age, gender, ASA status, surgical priority, surgical risk, presence of anaemia, RDW, presence of ischaemic heart disease and congestive heart failure. The web-based version of this risk calculator is currently under construction.
Presently, the use of preoperative risk stratification systems is not routinely done due to combination of factors such as the complexity of the system, the need for additional non-routine preoperative tests, use of intraoperative and postoperative variables and the inability to calculate an individual percentage mortality and morbidity risk.23 The CARES surgical risk calculator comprises simple and easily accessible variables available from routine preoperative evaluation for most surgeries, with the exception of healthy patients coming for the lowest risk surgeries, which may not even need an FBC.24 The inclusion of more variables would greatly increase the time taken to collect data and thereby reduce the usability of the tool. Ease of use and face validity are two important factors that may encourage widespread, routine use of risk prediction tools,25 both of which are true for the CARES surgical risk calculator. The CARES surgical risk calculator stood out from most other risk stratification tools not only in that it was derived from the intended Southeast Asian population but it also predicts both postoperative mortality and the need for postoperative ICU care using a single set of easily accessible variables. This provides convenience and improves efficiency to the clinician in a busy outpatient setting.
The prediction of postsurgical 30-day mortality risk is an important clinical outcome that is of interest to both surgeons and patients and therefore aids in shared decision making.26 Thirty-day all-cause postoperative mortality is a widely accepted, valid and relevant outcome measure of surgical care.27 For this study, the postoperative mortality data were synced with the National Registry of Death data, ensuring the integrity and completeness of the data.
The prediction of the risk of ICU admission for >24 hours postoperatively is novel and not available for most current risk stratification tools. The capability to predict ICU admission for >24 hours after surgery would aid physicians to determine the postoperative patient disposition plan before surgery and therefore healthcare resource allocation. This could improve patients’ outcome by reducing failure to rescue events28 and improve the efficiency of the valuable ICU bed allocation. The disposition of a patient immediately after surgery usually depends on both objective and subjective factors. Objective variables include patient comorbidities, preoperative evaluation, surgical risk and intraoperative haemodynamic. Subjective factors are usually operator-dependent and involve the clinical judgement and comfort level of the anaesthetist and surgeon involved.29 30 While an ICU admission by itself would not be a useful measure of morbidity, length of stay in ICU may be seen as an indirect measure of morbidity-related outcome.31 We defined ICU admission for >24 hours as a significant outcome on observation that patients who were discharged from ICU within the first 24 hours may have been safely monitored postoperatively in a lower intensity unit.
CARES reflects the critical significance of age as a predictor of risk. Age is a significant independent predictor, and this should be reflected in the risk counselling. This will help to increase the awareness of the impact of ageing on mortality among both the clinicians and patients and their careers.
Despite being a single-centre, locally derived risk stratification tool, CARES has a number of advantages. It is the largest study to develop and validate a surgical risk stratification tool in the heterogeneous, multiracial cohort of patients undergoing non-cardiac surgery in the Southeast Asia region. The CARES tool is a parsimonious model, consisting of only 9 preoperative variables, of which includes 7 clinical variables and 2 simple preoperative blood tests which are used to calculate both mortality and ICU admission risk, compared with 18 preoperative, intraoperative and postoperative variables for POSSUM and 22 preoperative patient risk factors for the ACS-NSQIP model.8 32 Despite the small number of variables used to compute surgical mortality and morbidity risk, CARES demonstrated high performance in our population cohort for both outcomes. This may be due to the use of biomarkers, which could increase the performance of risk prediction tools. Our CARES model incorporated the use of RDW which is obtained from the almost routine preoperative FBC test. Recent studies have shown the value of RDW in predicting postoperative mortality.13 14 20 33
In the development of our risk prediction tool, we focused on the utility and usability of the tool. We computed the ORs of each variable category and assigned a rank score to each of them. The cumulative rank scores were matched to corresponding risks for the outcomes. This resulted in a more user-friendly calculator, without compromising on the accuracy. This could lead to better physician and patient acceptance in using the tool for shared decision making as well as healthcare resource planning (see tables 2 and 5).
The ASA-PS is one of the most commonly used risk prediction tools currently in Singapore. We compared the AUROC of the CARES surgical risk calculator with that of ASA-PS and found that CARES showed considerable superiority in performance in both the mortality and need for ICU prediction. With easily available preoperative data, good accuracy and the availability of a web-based calculator, it is hoped that adoption of the CARES tool in calculating preoperative surgical risk may exceed that of other models.
The limitations for this study include the inherent nature of retrospective data.
In our data collection, we also did not include the presence of chronic obstructive pulmonary disease (COPD) as a variable for the development of our risk stratification tool. COPD itself has been not been conclusively shown to be strongly associated with increased postoperative mortality in non-cardiac or non-thoracic surgeries.34 35 One major weakness of our study is the lack of proper definition in our classification on the priority of surgery. As this is a retrospective study, we were unable to give an a priori definition of emergency surgery. We classified the priority of surgery according to that recorded in surgical records. However, in our centre, emergency surgery is further subdivided into four categories, with category A being surgeries that require operation within 6 hours of admission and category D being surgeries that can wait up to 24–48 hours. While the Surgical Risk Tool36 uses the National Confidential Enquiry into Perioperative Deaths classification of surgical priority which differentiates between scheduled, elective, urgent and emergency surgeries,6 the lack of discrimination in the priorities of our emergency surgery data contributes to possible classification bias.
Furthermore, while the CARES tool is based on a large data set, it has only been validated in a single centre, hence, there is a need for further prospective studies to validate the tool in other institutions across the region. Despite this limitation, the sample analysed was representative of the local population and the demographics of hospital admissions in Singapore37 38 The results are generalisable owing to the broad representation of the range of surgical specialties. It predicts both postoperative mortality risk and need for ICU in our local population and has the potential to be a specific risk prediction tool in the Southeast Asian population.
The CARES surgical risk calculator has many firsts. Besides being the first locally derived preoperative risk prediction tool and the first calculator to predict both mortality and ICU need after surgery, it is also the first risk stratification tool to incorporate the use of RDW. The generalisability of the model in international cohorts remains unknown and needs to be explored. External validation of the CARES tool is integral to test its validity further, as is the periodic recalibration and re-evaluation of the model to maintain validity with healthcare advancements, and demonstrate both utility and accuracy in predicting mortality and morbidity. The CARES tool has a potential to be expanded to be used for prediction for other postoperative morbidities, and further prospective studies should be focused on this. Likewise, studies evaluating the impact of risk stratification on improving patient outcomes through individual care planning should be a research priority as there is an opportunity to improve outcomes substantially.
The CARES tool provides an accurate prediction for mortality and need for postoperative ICU among surgical patients in Singapore. It is easily accessible and should be used in conjunction with clinical judgement to aid shared decision making and plan for ICU resource allocation. The development of the web-based calculator further facilitates user accessibility and utility of the tool.
The authors thank Mr Koh Yee Jin (principal systems specialist, Department of Health Insights, Integrated Health Information Systems Pte Ltd, Singapore) for his invaluable help in the data extraction process , Ms Sudha Harikrishnan, from the Department of Anaesthesiology, Singapore General Hospital for her help in data extraction and Associate Professor Ong Biauw Chi, Chair of Medical Board, Sengkang General Hospital, Singapore for her insightful mentorship.
Contributors DXHC: data analysis and interpretation, prepared draft manuscript, revised draft manuscript and approved final manuscript for submission. YES: data acquisition and curation, data analysis and interpretation, revised draft manuscript and approved final manuscript for submission. YHC: data analysis and interpretation, revised draft manuscript and approved final manuscript for submission. RP: funding acquisition and approved final manuscript for submission. HRA: designed and conceptualised study, funding acquisition, data acquisition and curation, data analysis and interpretation, revised draft manuscript and approved final manuscript for submission. All the authors agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.
Funding This work was supported by the department fund of Department of Anaesthesiology, Singapore General Hospital. The funding sources had no role in the design of this study and the analysis and interpretation of the results.
Disclaimer The authors affirm that this manuscript is an honest, accurate and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned have been explained.
Competing interests HRA is a recipient of SingHealth Duke-NUS Nurturing Clinician Scientists Scheme Award, project number 12/FY2017/P1/15-A29.
Patient consent Not required.
Ethics approval This study has been approved by the SingHealth Centralised Institutional Review Board (CIRB).
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement Extra data can be accessed via the Dryad data repository at http://datadryad.org/ with the doi: 10.5061/dryad.v142481.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.