Article Text

Original research
Development and validation of a risk prediction model for in-hospital major cardiovascular events in patients hospitalised for acute myocardial infarction
  1. Chaoqun Wu,
  2. Xiqian Huo,
  3. Jiamin Liu,
  4. Lihua Zhang,
  5. Xueke Bai,
  6. Shuang Hu,
  7. Xi Li,
  8. Jiapeng Lu,
  9. Xin Zheng,
  10. Jing Li,
  11. Haibo Zhang
  1. National Clinical Research Center for Cardiovascular Diseases, NHC Key Laboratory of Clinical Research for Cardiovascular Medications, State Key Laboratory of Cardiovascular Disease, National Center for Cardiovascular Diseases, Chinese Academy of Medical Sciences and Peking Union Medical College Fuwai Hospital, Xicheng District, Beijing, China
  1. Correspondence to Dr Haibo Zhang; haibo.zhang{at}fwoxford.org

Abstract

Objectives Patients admitted to hospital with acute myocardial infarction (AMI) have considerable variability in in-hospital risks, resulting in higher demands on healthcare resources. Simple risk-assessment tools are important for the identification of patients with higher risk to inform clinical decisions. However, few risk assessment tools have been built that are suitable for populations with AMI in China. We aim to develop and validate a risk prediction model, and further build a risk scoring system.

Design Data from a nationally representative retrospective study was used to develop the model. Patients from a prospective study and another nationally representative retrospective study were both used for external validation.

Setting 161 nationally representative hospitals, and 53 and 157 other hospitals were involved in the above three studies, respectively.

Participants 8010 patients hospitalised for AMI were included as development sample, and 4485 and 11 223 other patients were included as validation samples in their corresponding studies.

Primary and secondary outcome measures The in-hospital major adverse cardiovascular events (MACE) was defined as death from any cause, recurrent AMI, or ischaemic stroke.

Results The proportion of in-hospital MACE was 11.7%, 8.8% and 11.4% among the development sample and two external-validation samples, respectively. Nine predictors (ie, age, sex, left ventricular ejection fraction, Killip class, systolic blood pressure, creatinine, white blood cell count, heart rate and blood glucose) were independently associated with in-hospital MACE. The model performed well on both discrimination and calibration capability, with areas under the Receiver Operating Characteristic Curve (ROC) curve of 0.85, 0.74 and 0.80, and calibration slopes of 0.98, 0.84 and 0.97 in the development sample and two external validation samples, respectively. A point-based risk scoring system was built with good discrimination and reclassification ability.

Conclusions A prediction model using readily available clinical parameters was developed and externally validated to estimate risks of in-hospital MACE among patients with AMI, thereby better informing decision-making in improving clinical care.

  • myocardial infarction
  • risk management
  • stroke

Data availability statement

No data are available. No additional data available.

http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

Statistics from Altmetric.com

Strengths and limitations of this study

  • The analysis was based on a nationally representative cohort of hospitals in China using random samples of patients admitted with acute myocardial infarction.

  • We used logistic regression and Markov Chain Monte Carlo method to develop a prediction model, which evaluated the risk of in-hospital major adverse cardiovascular events among patients when at admission.

  • A simple risk score was further derived based on the prediction model.

  • We validated the model, using two data sets from a prospective cohort study and another national representative retrospective study.

  • Further external validations will be needed in the future.

Introduction

Acute myocardial infarction (AMI) is one of the leading causes of mortality and morbidity globally.1 2 Despite the fact that clinical management of AMI has greatly improved, in-hospital mortality and the rate of recurrent ischaemic vascular events (eg, recurrent myocardial infarction, ischaemic stroke) remains high over the past few decades.3 4 The ability to identify patients at risk of in-hospital major adverse cardiovascular events (MACE) using simple risk assessment tools may help physicians with proper clinical decisions regarding therapeutic strategies and hospital resources allocation.5 6 Additionally, such assessment tools should be easy to use at bedsides with routinely available clinical data.

Previous multivariable risk models, such as Global Registry of Acute Coronary Events (GRACE) and Thrombolysis in Myocardial Infarction (TIMI), have contributed important insights into the association between patient’s clinical data and in-hospital death or stroke.7–9 However, most of these models mainly focused on a single outcome, while major vascular events all adversely affect a patient’s quality of life and long-term outcomes. In addition, most models were developed among convenience samples from clinical trials or registry studies, which tended to recruit population from ‘centers of excellence’ or hospitals with high quality of care, and few studies had included representative samples from routine clinical care. Establishing a generalisable risk model is particularly important in China which is experiencing a growing burden of AMI with dramatic geographical variation in disease patterns, medical resources, and healthcare capability.10–12 Some previous models identifying risks of patients from China were established using self-reported data from trials or registry studies, or only focused on a single outcome or subtypes of AMI.13–15 Consequently, a practical prediction model derived from a large representative nationwide population would be imperative.

Accordingly, using data from China Patient-centered Evaluative Assessment of Cardiac Events Retrospective Study of Acute Myocardial Infarction (China PEACE-Retrospective AMI) Study and China PEACE-Prospective AMI Study, we aim to develop and externally validate a prediction model and a risk score to help clinicians quickly identify patients at admission with increased risks of in-hospital MACE and consequently improve their outcomes.

Method

The study was reported in accordance with the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis reporting guideline (online supplemental eTable 1).16

Patient and public involvement

Patients and /or public were not involved.

Study and validation populations

In this study, we established the model using data from China PEACE-Retrospective AMI Study (year 2011). Since it is widely accepted that prediction models should be externally validated in independent populations before being applied in clinical practice, we used data from the China PEACE-Prospective AMI Study and China PEACE-Retrospective AMI Study (year 2015) to perform external validation (ie, ‘validation #1’ and ‘validation #2’ samples).

The China PEACE-Retrospective AMI Study (year 2011) is a two-stage random sampling-designed cross-sectional study.17 In the first phase, a stratified random sampling procedure was used to identify participating hospitals according to regions. In the second phase, patients were selected from each sampled hospital through a systematic sampling approach (online supplemental appendix A). By this method, the study created a nationally representative sample of 9333 patients hospitalised for AMI across China during 2011. In 2015, the same approach was applied and 12 108 patients hospitalised for AMI were newly enrolled, yielding another nationally representative sample of China. The China PEACE-Prospective AMI Study is a nationwide prospective cohort study which consecutively enrolled patients with hospitalisation for AMI from 53 hospitals throughout 21 provinces in China from December 2012 to August 2014.18

Patients were excluded if they were hospitalised in another hospital first and transferred to the study hospital, or transferred out of the study hospital, or age <18 years old. Definitions for clinical risk factors or medical history were the same in the above three studies. All data from these studies was centrally abstracted from medical records following standardised operation procedures. The researchers monitored data abstraction quality by randomly auditing 5% of the medical records, with overall variable accuracy >98%.17 18

The study protocol conforms to the ethical guidelines of the 1975 Declaration of Helsinki as reflected in a priori approval. The central ethics committee at the China National Center for Cardiovascular Diseases approved the aforementioned three studies (ethics number: China PEACE-Prospective AMI Study and China PEACE-Retrospective AMI Study (year 2011): 2012–377; China PEACE-Retrospective AMI Study (year 2015): 2016–769). All collaborating hospitals either accepted central ethics approval or obtained local ethics approval by their internal ethics committees. All participants gave informed consents in the prospective study. In the retrospective studies, written informed consent of patients were not required.

Predictors

Potential predictors were selected if they were clinically meaningful, reliable and easy to collect, statistically meaningful and with a frequency of more than 1% occurrence. Candidate predictors included patient demographics characteristics (age, sex), medical histories (hypertension, diabetes mellitus, myocardial infarction, percutaneous coronary intervention, chronic kidney disease, ischaemic stroke) and clinical factors (Killip class 3 or 4, subtypes of AMI, pneumonia, left ventricular ejection fraction (LVEF), heart rate, systolic blood pressure (SBP), white blood cell count (WBC), blood glucose, serum creatinine, troponin) at admission (detail information in online supplemental appendix B). Missing rates of continuous variables among development sample ranged from 0.08% (age) to 8.7% (blood glucose). These missing values were at random and imputed by multiple imputation method with 10 imputations through SAS procedure, and the average of them was used as the imputation results (online supplemental appendix C). Variables after imputation was used to select predictors and develop the model.

Outcomes

The outcome for the prediction model was in-hospital MACE, defined as a composite of first occurrence of all-cause death, recurrent AMI or non-fatal ischaemic stroke during index hospitalisation. Outcomes were sought systematically by trained local clinic staff from relevant medical records and death certificates. All-cause death was defined as in-hospital death or withdrawal from treatment due to terminal status at discharge, since it is common that many severe patients are reluctant to die in hospital in China.17 Recurrent AMI was indicated if there was physician documentation of recurrent myocardial infarction from the beginning of hospital stay to discharge. Ischaemic stroke was defined as an acute symptomatic episode of focal or global neurological dysfunction caused by brain, spinal or retinal vascular injury as a result of infarction. We ascertained major adverse events with the same approach used in our large international multicentre trials.19 All outcomes were centrally adjudicated at the national coordinating centre by trained clinicians using standardized protocol.

Statistical analysis

We described patients’ characteristics in development and two external samples, respectively. Categorical variables were summarised as frequencies (%) and continuous variables were presented as means with SD. Observed events rate and 95% CI were also reported.

Model development and validation

Continuous variables were converted into categorical variables for easy using in clinical practice (age ≥65 years old, LVEF ≤40%, LVEF unable to be measured, heart rate >90 beats per minute, blood glucose >10 mmol/L, SBP <90 mm Hg, WBC >12×109/L and creatinine >100 μmol/L), and a stepwise multivariable logistic model was fitted in the development sample to determine potential predictors with a p value threshold of 0.2 for adding variables and 0.1 for removing variables. We then fitted the model with Markov Chain Monte Carlo (MCMC) simulation method to calculate a posterior probability for each selected predictor (online supplemental appendix D). In order to select stable factors, only potential predictors with posterior probability of 100% for positive association would be included in the final predictor list.20 Finally, we applied the prediction model to two external validation samples to further evaluate model stability.

Area under ROC curve (AUC) of the prediction model applied to development and external validation samples were reported, statistically indicating the discrimination power. The event rates in 10 strata defined by the lowest and highest deciles based on predictive probabilities were described, which also demonstrated model’s discrimination power.20 Calibration power was shown by graph with observed risks plotted on the y-axis against predicted risks on the x-axis, and patients were divided into 10 groups according to their predicted risks. The calibration slope and intercept, calculated by regressing the observed outcome on the predicted probabilities, was reported to evaluate the calibration statistically. A slope closer to 1 and intercept closer to 0 represented better calibration power. CIs of AUCs, calibration slopes and calibration intercepts were calculated by the bootstrap method, with which we randomly sampled 70% of all enrolled sampled and repeated 2000 times among three cohorts, separately. We did not impute missing values of predictors among two external validation samples, considering the original data sets were closer to the actual clinical status. Sensitivity analysis was conducted by building a model using data set without imputation for missing values, as well as applied the final prediction model among subgroups (gender, time from symptom to admission, AMI type, primary percutaneous coronary intervention (PCI) or not, and type of hospital) and reported AUCs.

Risk score

To simplify the use of prediction model, we developed a risk score system based on the regression coefficients estimated from the final prediction model. We calculated the percentage of each predictor’s coefficient among the sum of all coefficients (except intercept), and then rounded to integer as the assigned point value. A score was calculated for each patient by adding together the points corresponding to all predictors. In addition, we stratified patients into three groups based on the distribution of the risk score and calculated the average predicted event rates: low (about 10th percentile), average (about 10th–90th percentile) and high (about 90th percentile).

Analyses were conducted using SAS V.9.4 (SAS Institute). Statistical significance was defined by a two-tailed p value <0.05.

Results

Study samples baseline characteristics

A total of 8010 patients from 161 hospitals (65 tertiary and 96 secondary hospitals) were included in the development sample (online supplemental eFigure 1). The mean age of the population was 65.6±12.7 years, and 2585 (32.3%) were women. The most common comorbidities were hypertension (53.8%) and diabetes (22.6%). A total of 4485 patients from external validation #1 samples and 11 223 patients from external validation #2 samples were included. Compared with the development sample, the validation #1 population was younger (age 61.7±12.3 years) and had fewer women (25.1%), while the validation #2 population had similar demographic characteristics with the development sample (mean age of 65.9±12.7 years, women 32.2%). Table 1 summarised patients’ baseline demographic and clinical characteristics across study populations.

Table 1

Model-selected patient predictors by development and validation samples

Model development

In the development sample, 935 participants had MACE during hospitalisation, with the observed rate of 11.7% (95% CI: 11.0% to 12.4%). The observed rate for all-cause death, recurrent AMI and ischaemia stroke were 10.9%, 0.6% and 0.5%, respectively (online supplemental eFigure 2). The stepwise logistic regression identified 17 independent predictors and MCMC simulation kept 9 of them (figure 1 and online supplemental eFigure 3), including age, sex, LVEF, Killip class, SBP, creatinine, WBC, heart rate and blood glucose. We also performed a sensitivity analysis without missing data imputation and the results were consistent with the above findings (online supplemental eFigure 4).

Figure 1

Estimate coefficients and ORs of predictors of the prediction model. The predicted probability of outcomes can be calculated using the following formula: Probability of outcome (%) = (exp(B)) / (1+exp(B)) × 100%, where B=0.813 × (if age ≥65 years) + 0.313 × (if women) + 1.007 × (if Killip class 3/4) + 1.989 × (if LVEF unable to be measured) + 0.990 × (if LVEF ≤40%) + 0.390 × (if heart rate >90 bpm) + 0.669 × (if SBP <90 mm Hg) + 0.743 × (if WBC >12 000/μL) + 0.583 (if blood glucose >180 mg/dL) + 0.695 × (if serum creatinine >100μmol/L) − 4.881. bpm, beats per minute; LVEF, left ventricular ejection fraction; SBP, systolic blood pressure; WBC, white blood cell count.

The prediction model demonstrated good discrimination and calibration ability. The AUC for the final prediction model was 0.85 (95% CI: 0.83 to 0.86) (online supplemental eFigure 5). The median predicted event rate ranged from 0.75% in the lowest predicted decile to 47.57% in the highest predicted decile. For every 10% of patients, the mean predicted event rate ranged from 0.75% to 50.61%; while the actual numbers of events were from 0.47% to 50.38%, with a calibration slope of 0.98 (95% CI:0.96 to 0.99) and intercept of 0.003 (95% CI: 0.001 to 0.005) (table 2, figure 2).

Figure 2

Discrimination and calibration of the prediction model. (Upper panel) Distribution of predict events rate by 10 strata of predictive probabilities among development and external validation samples. (Lower panel) Correlation of observed events rate and predict events rate by 10 strata of predictive probabilities among development and external validation samples.

Table 2

Performance of prediction model among study samples

Model validation

The MACE rate was 8.8% (95% CI: 8% to 9.7%) among the validation #1 population. Compared with development and validation #2 samples, validation #1 sample had higher rates of ischaemia stroke (4.2% vs 0.5% and 0.8%) and lower rates of death (4.1% vs 10.9% and 9.9%). The event rate in validation #2 samples was 11.4% (95% CI: 10.9% to 12.1%) that was similar with the development samples (table 2).

AUCs of the final prediction model applied in the two external validation samples were 0.74 (95% CI: 0.70 to 0.77) and 0.80 (95% CI: 0.78 to 0.81), respectively. In subgroup analysis, the AUCs were also greater than 0.70 among all subgroups (online supplemental eTable 2). The median predicted event rate ranged from 0.75% to 25.58% in validation #1 sample and from 0.75% to 40.1% in validation #2 sample, respectively. Additionally, in two validation samples, the calibration slopes were 0.84 and 0.97, with intercepts of 0.035 and 0.020, respectively (table 2, figure 2).

Risk score

We developed a risk score based on the prediction model (table 3). The risk factor-specific points ranged from 4 (women) to 24 (LVEF unable to be measured). Overall, 10.7%, 79.2% and 10.1% patients were stratified to the low-risk (score: 0), intermediate-risk (score: 1–49) and high-risk (score: ≥50) groups, with corresponding predicted probabilities of 0.8%, 8.2% and 50.1% for in-hospital MACE outcomes in the development population, respectively (figure 3). All three samples had the score ranging from 0 to 87, which had a good correlation with the predicted probability of in-hospital MACE from the prediction model (figure 3, online supplemental eTable 3). The risk stratifications for the two external validation samples were also similar with the development sample (figure 3).

Figure 3

Distribution and performance of risk scores based on prediction model. (Upper-left panel) Distribution of risk scores among development and external validation samples. (Upper-right panel) Association between risk score and predicted probability of prediction model. (Lower panel) Risk stratification of patients from development and external validation samples by risk scores.

Table 3

Risk score based on prediction model

Discussion

In this study, we have developed and externally validated a risk-prediction model to estimate in-hospital MACE among patients hospitalised for AMI. The model indicated good model discrimination and calibration ability as suggested by external validations. The predictors in this prediction model were easy to collect and readily available for patients at hospitalisation. We have also developed a point-based risk scoring system based on the model, allowing clinicians to identify high-risk patients at admission and provide targeted treatments to improve health.

Our study expands on previous studies in several aspects. First, previous models were mostly developed from trial populations in developed countries including USA or European countries,9 21 22 which tended to enrol high-risk patients willing to participate from selective clinical sites, and study populations mainly focused on subtypes of AMI. Our prediction model was derived from a large and nationally representative patient cohort using rigorously abstracted information and externally validated among another two independent patient cohorts. Second, in contrast to most studies that used a single outcome, such as mortality or ischaemic stroke, we focused on a composite cardiovascular event including death and other major vascular events that may affect prognosis and impair quality of life. From a patient perspective, identifying the risk of in-hospital MACE is also important to ensure they could receive proper attention and evidence-based longitudinal care. Finally, the MCMC algorithm applied in this model guaranteed the reliability of included factors by providing a robust Bayesian variable selection based on marginal posterior probability.

Our prediction model included nine predictors which were consistent with prior studies.9 15 23–25 All the risk factors used in the risk scores are easy to collect, widely accepted and available on admission in clinical practice. In our model, LVEF unable to be measured is an important factor, and the reasons for the missing value of LVEF could be due to patients’ relatively worse heath status which lead to the reluctance towards further tests. The missing rate in our study was also consistent with previous reported data, such as 26.5% in a clinical trial26 and 29.4% in a US prospective cohort.27 Low blood pressure increased the risk of in-hospital MACE, which was also consistent with prior studies,23 28 29 and may be associated with worse health status, such as cardiogenic shock. To balance the model’s discrimination and complexity, we developed a risk score which could be easily calculated using demographics and clinical factors to simplify the model in this study. A score greater than 50 indicates a high risk (about 50% probability) of in-hospital MACE, suggesting more attention and evidence-based treatment for these patients during their hospitalisation for AMI.

The model and risk score performed well on discrimination and calibration capability in the development sample, and showed high consistency during external validation among populations that were enrolled 2–4 years later and with distinguished population characteristics. The results were also favourably comparable with those from previous published models. We also observed that patients from validation #1 had less MACE occurrence than the other two study populations. One possible explanation could be that validation #1 study was a prospective cohort and patients may tend to have better health status and adherence than those from retrospective ones, and validation #2 population was established in 2015 when treatment therapies and techniques for AMI had been improved over time. Additionally, we applied the model in different subgroups and the results remained consistent across subgroups (eg, ST segment elevation myocardial infarction (STEMI) or non-STEMI (NSTEMI), primary PCI treatment or not) with good discrimination. We also compared the model performance between our score and GRACE score for in-hospital mortality or myocardial infarction30 and for ischaemic stroke,7 and between ours and TIMI score for STEMI23 and for NSTEMI.9 The result shows if physicians need to predict the probability of in-hospital MACE in a patient admitted with AMI, our score is more effective than the direct application of GRACE and TIMI (online supplemental appendix E).

China has a growing burden of cardiovascular disease and AMI accounts for more than 80% of such events in the country.31 Our equations for predicting in-hospital MACE could not only enable accurate in-hospital adverse events assessment and improve patients’ quality of life, but also shorten the length of hospitalisation due to MACE. At the same time, when physicians are reminded that patients are at high-risk, they may also pay more attention in the treatment and management regarding therapeutic strategies and hospital resources allocation, thereby saving medical resources. The model and risk score from this study could play an important role in promoting better management of risk factors for preventing major vascular events during hospitalisation for AMI in China.

This study should be considered in the context of several limitations. First, we only assessed the risks of in-hospital MACE outcomes. Fixed-time outcomes, such as 30-day major vascular events, would not depend on length of stay, which is relatively longer in China than that of the Western countries and may be a potential confounder. Second, although the data sets we used to develop and validate are large nationwide samples or cohort of patients with AMI in China, the prediction model still needs to be validated and updated when additional data sets become available in the future. Third, the data for model development might be slightly dated; however, we externally validated in a sample in 2015 and a more recent prospective cohort, which have proved to discriminate well.

In conclusion, we developed and validated a useful and easily used prediction model and risk score to estimate risks of in-hospital MACE among patients hospitalised for AMI. The model evaluation indicates that it can predict MACE with good discrimination and calibration, making it helpful for identifying high-risk patients and effectively informing individualised decision-making in supporting quality improvement.

Data availability statement

No data are available. No additional data available.

References

Supplementary materials

  • Supplementary Data

    This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

Footnotes

  • CW and XH are joint first authors.

  • Contributors HZ contributed to the conception or design of the work. CW, XH, XB and SH contributed to the acquisition of data for the work. CW and XB contributed to the analysis of data for the work. CW, XH, HZ, JLiu, LZ, XL, JLu, XZ and JLi contributed to the interpretation of data for the work. CW and XH drafted the manuscript. HZ, JLiu, LZ, XB, SH, XL, JLu, XZ and JLi critically revised the manuscript. All gave final approval and agree to be accountable for all aspects of work ensuring integrity and accuracy.

  • Funding This work was supported by the National Key Research and Development Program from the Ministry of Science and Technology of China (2018YFC1311205, 2017YFC1310803, 2015BAI12B01); the Research Special Fund for Public Welfare Industry of Health (201202025) from the National Health and Family Planning Commission of China. The funder of the study had no role in study design, data collection, data analysis, data interpretation or the decision to submit the manuscript for publication.

  • Competing interests JLi reported receiving research grants, through Fuwai Hospital, from the People’s Republic of China for work to improve the management of hypertension and blood lipids and to improve care quality and patient outcomes of cardiovascular disease; receiving research agreements, through the National Center for Cardiovascular Diseases and Fuwai Hospital, from Amgen for a multicentre clinical trial assessing the efficacy and safety of omecamtiv mecarbil and for patient with dyslipidaemia registration; receiving a research agreement, through Fuwai Hospital, from Sanofi for a multicentre clinical trial on the effects of sotagliflozin; receiving a research agreement, through Fuwai Hospital, with the University of Oxford for a multicentre clinical trial of empagliflozin; receiving a research agreement, through the National Center for Cardiovascular Diseases, from AstraZeneca for clinical research methods training outside the submitted work; and receiving a research agreement, through the National Center for Cardiovascular Diseases, from Lilly for physician training outside the submitted work. No other disclosures were reported.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.