Article Text
PDF
Abstract
Objectives Risk scores are recommended in guidelines to facilitate the management of patients who present with acute coronary syndromes (ACS). Internationally, such scores are not systematically used because they are not easy to apply and some risk indicators are not available at first presentation. We aimed to derive and externally validate a more accurate version of the Global Registry of Acute Coronary Events (GRACE) risk score for predicting the risk of death or death/myocardial infarction (MI) both acutely and over the longer term. The risk score was designed to be suitable for acute and emergency clinical settings and usable in electronic devices.
Design and setting The GRACE risk score (2.0) was derived in 32 037 patients from the GRACE registry (14 countries, 94 hospitals) and validated externally in the French registry of Acute ST-elevation and non-ST-elevation MI (FAST-MI) 2005.
Participants Patients presenting with ST-elevation and non-ST elevation ACS and with long-term outcomes.
Outcome measures The GRACE Score (2.0) predicts the risk of short-term and long-term mortality, and death/MI, overall and in hospital survivors.
Results For key independent risk predictors of death (1 year), non-linear associations (vs linear) were found for age (p<0.0005), systolic blood pressure (p<0.0001), pulse (p<0.0001) and creatinine (p<0.0001). By employing non-linear algorithms, there was improved model discrimination, validated externally. Using the FAST-MI 2005 cohort, the c indices for death exceeded 0.82 for the overall population at 1 year and also at 3 years. Discrimination for death or MI was slightly lower than for death alone (c=0.78). Similar results were obtained for hospital survivors, and with substitutions for creatinine and Killip class, the model performed nearly as well.
Conclusions The updated GRACE risk score has better discrimination and is easier to use than the previous score based on linear associations. GRACE Risk (2.0) performed equally well acutely and over the longer term and can be used in a variety of clinical settings to aid management decisions.
- ACCIDENT & EMERGENCY MEDICINE
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 3.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/3.0/
Statistics from Altmetric.com
Strengths and limitations of this study
-
The Global Registry of Acute Coronary Events (GRACE) 2.0 risk score is derived from the largest multinational registry in acute coronary syndromes (ACS) and validated in an entirely independent dataset with comprehensive long-term outcome data.
-
This risk score employs non-linear functions and is more accurate than the original version. It is now validated over the longer term (to 1 and 3 years) and with substitutions possible for creatinine values and Killip class (performing almost as well).
-
This electronic risk score is designed to be used in mobile electronic devices (approximately 30 s to enter data) and presents the risk of death (or death/myocardial infarction) and relative to the entire ACS population.
-
The score is designed to assist clinical management decisions and is not a substitute for individual patient clinical assessment. However, it may help to address the current ‘treatment-risk paradox’ whereby low-risk rather than high-risk patients are more likely to receive interventional therapies.
-
Additional factors may influence outcome, especially in geographical populations and healthcare systems not evaluated in the multinational GRACE programme.
Introduction
Acute coronary syndromes (ACSs) comprise a heterogeneous spectrum of patients who are currently stratified for management mainly on the basis of ECG characteristics and biomarker results. National Institute for Health and Care Excellence (NICE), Scottish Intercollegiate Guidelines Network (SIGN), European Society of Cardiology (ESC) and North American guidelines separate patients into ST-elevation MI or non-ST-elevation ACS, and they also recommend use of a risk score such as the Global Registry of Acute Coronary Events (GRACE) score.1–4 However, systematic risk stratification is not widely performed, despite the evidence and the guidelines.
Why should risk assessment be important for the triage and management of patients with acute coronary disease?
Whether a patient proceeds to an immediate, urgent or delayed coronary angiography and revascularisation and which of the acute antithrombotic regimens is chosen depends on patient risk characteristics. Evidence from randomised trials and guideline recommendations all support the use of different strategies according to risk status.1–4
In the development of NICE guideline 94 (http://www.nice.org/cg94), the guideline states that single variables (eg, troponin) were not as good as multiple variables in predicting outcome.1 NICE independently tested all of the published risk scores (GRACE,5 ,6 TIMI,7 PURSUIT,8 PREDICT,9 EMMACE,10 SRI,11 AMIS,12 UA13 risk score) in 64 312 patients from the MINAP dataset. They employed a ‘mini-GRACE score’ as many of the MINAP patients lacked creatinine values and Killip classification (substituting a history of renal dysfunction and the use of diuretics) and this approach also demonstrated good performance in an independent assessment.14 The c statistics was 0.825 with 95% confidence bounds 0.82 to 0.83 and this was superior to the performance of the other risk scores, and hence the recommendation from NICE to employ the GRACE risk score.1 However, the use of substitutions for creatinine and for Killip class has not been validated in an independent dataset and the prediction of long-term outcome had not been tested. In addition, non-linear functions for continuous variables and for Killip class may improve model discrimination and could be implemented in hand-held electronic devices.
Resolving the ‘treatment-risk paradox’
We and others have revealed a treatment-risk paradox in the management of acute coronary disease.15 ,16 In contrast to the evidence and the guideline recommendations, lower risk rather than higher risk patients are more likely to undergo interventional procedures and receive more aggressive antithrombotic and other therapies.15 ,16 This phenomenon has now been reported across widely different healthcare systems and different geographical settings. Why is this? First, current treatment decisions rely on clinical assessment and it is difficult for the clinician to weigh up potential benefits against potential hazards, and hence lower risk patients are commonly selected for more aggressive treatment (an unintended risk-averse approach). However, evidence demonstrates that even excluding those with contraindications, higher risk cohorts potentially have more to gain.15
Why are risk scores not more widely used?
Internationally, risk scores are not systematically applied for the management of ACS despite the evidence and guideline recommendations. Several factors contribute to this, including the misperception that clinician assessment or the use of individual risk indicators is sufficient.1 ,2 ,17 In addition, the most accurate risk scores have been cumbersome to compute (eg, requiring look-up tables and many use arbitrary score results). Finally, the parameters necessary for their implementation may not be available at the time of the patient's initial presentation.
What this study adds?
We aimed to develop and validate a revised and more accurate version of the GRACE risk score suitable for the acute and long-term prediction of risk. Instead of assuming that continuous variables such as age and the categorical variable Killip class were linearly associated with risk, we tested for non-linear associations and included them in the revised prediction tool where appropriate. In contrast to the earlier version of the GRACE score, which required the computation of a numerical score (without absolute risks), we derived and externally validated an electronic version with absolute percentage risks. This is suitable for use in hand-held electronic devices and smartphones, and the clinical applicability is broadened by using substitutions for creatinine and Killip class. Creatinine values may only be available after hospital admission and many settings do not routinely use Killip class for evaluating heart failure symptoms. Thus, the aim of this study was to develop a simplified risk score suitable for applications in a variety of settings and to test the accuracy of the revised GRACE risk predictor (GRACE score 2.0) to predict early and long-term risk, as an aid to clinical management.
Methods
GRACE risk score
The GRACE registry was designed to reflect an unbiased population of patients with ACS and was undertaken over 10 years, in 94 hospitals and 14 countries.5 ,6 ,18–20 The design has been reported previously.18 ,20
In-hospital and up to 6 months outcomes and risk scores were derived based on independent predictors of outcome. These have been described previously (ST segment deviation, age, heart rate, systolic blood pressure, creatinine, Killip class, cardiac arrest at admission and elevated biomarkers of necrosis).5 ,6 The GRACE risk score was derived from the original population of 26 267 patients (11 389 for hospital score for patients enrolled through 31 March 2001; 21 688 were used to derive the 6-month risk score for patients enrolled through 30 September 2002) with suspected ACS, validated prospectively in a further set of 22 122 patients and validated externally.5
Risk characteristics of populations may evolve over time (as management changes) and it is appropriate that the GRACE score should be tested in a more recent cohort of patients with ACS and with extended follow-up.21
The original GRACE score was estimated in hospital risk of death or the combination of death or MI and the same outcomes up to 6 months postdischarge. The new version of the GRACE risk score for 1-year outcomes was derived in the more recent dataset of 32 037 patients from the GRACE registry enrolled between January 2002 and December 2007. For three 3-year mortality, the UK cohort of 1274 patients with long-term follow-up was employed. The characteristics of this study population have been reported previously.20 The algorithm employed the same independent predictors of outcome as originally derived and reported, but non-linear associations were incorporated to improve model discrimination. In addition, a simplified version of the risk score was developed with substitutions for creatinine (history of renal dysfunction) and substitutions for Killip class (diuretic usage). As previously validated, a parsimonious model of only eight factors conveyed more than 90% of the predictive accuracy of the complete multivariable model.5 ,6
Consistency of estimates in different GRACE risk models
The GRACE risk score V.2.0 contains slightly more precise estimates of V.1.0 hospital6 and 6-month death5 probabilities. Instead of converting model estimates to a point system, and using intervals for continuous variables such as age, as in V.1.0, V.2.0 directly utilises model estimates themselves to compute cumulative risk (see: http://www.outcomes-umassmed.org/grace/files/GRACE_RiskModel_Coefficients.pdf).
Because GRACE models were derived in different patient populations from different study periods, differences in cumulative rate estimates for the same interval exist. The 1-year death model contains the most recent and largest patient populations. Therefore, 6-month and 3-year death models were standardised to conform to estimated Kaplan-Meier cumulative rates for the 1-year model. The revised V.2.0 6 month cumulative estimates now conform to V.2.0 1-year model estimates as of 6 months, and the 1-year estimates for the V.2.0 3-year model as of 1 year also conform to V.2.0 1-year estimates for the 1-year model.
External validation
The updated GRACE risk score was validated by testing the algorithm in its full version and simplified version in an entirely separate registry population, the French registry of Acute ST-elevation and non-ST-elevation Myocardial Infarction (FAST-MI).22–24 FAST-MI 2005 is a nationwide French registry conducted over a 1-month period at the end of 2005 and it included 3059 patients with ST segment elevation myocardial infarction (STEMI) or non-STEMI from 223 centres. Follow-up was conducted by a research team from the Société Française de Cardiologie and investigators.22 ,23 Sequentially, they consulted death registry data, wrote to family doctors and/or cardiologists and wrote to patients. In many instances, written contact was followed by telephone interviews.22 ,23 All variables required to calculate the new GRACE risk score were available in 2959 of the 3059 patients (96.7% of the full cohort). The GRACE algorithm was applied to the 2959 patients using logistic regression and the c statistics calculated for mortality at 1 year, mortality at 3 years and then for the subsets of patients with STEMI and non-STEMI. In addition, c statistics were calculated for death or MI. The same analyses were then repeated for hospital survivors only (n=2806). In addition, goodness of fit was tested using the Hosmer-Lemeshow test. Likewise, the simplified score was tested in the 3035 patients in whom all variables needed for its calculation were available.
Statistics
The Kaplan-Meier method was used to estimate 1-year and 3-year outcome rates.
Cox multiple regression models were fitted to outcomes of death and death or MI within 1 and 3 years of hospital admission.24 The same eight factors used in the original GRACE risk scores were used.6 The method of restricted cubic splines25 employs a smooth polynomial function and was used to test for possible non-linear associations between outcomes and age, creatinine, pulse and systolic blood pressure. Also, Killip class using four categories was compared with linear Killip class. Associations that improved model likelihood at the α=0.05 level were retained in final models. Such associations were also plotted and examined for clinical plausibility.
Model performance was evaluated using the May-Hosmer goodness of fit test,26 and Harrell's c index for model discrimination.27 A prediction tool based on these models uses point estimates and baseline survival to arrive at predicted outcomes for a given patient's covariate experience.28 Plots of estimated model event probabilities for non-linear covariates were produced using baseline survival estimates and risk factor parameter estimates (on the log hazard scale), evaluated at covariate means. These plots describe the shape of the association between the non-linear factors and outcomes, but they do not substitute for entering all of the patient's risk factor information into the risk tool.
Results
Patient characteristics
For the 32 037 patients from the GRACE registry (table 1), there were 2422 deaths within 365 days of initial admission, and complete covariate data. The distribution of deaths was as follows: 1275 in hospital (53%), 983 deaths after discharge within 180 days of admission (41%), 164 deaths from 181–365 days after admission (7%). The estimated 365 day cumulative death rate is 9.3% using the Kaplan-Meier method.
Characteristics on admission of the GRACE patients with ACS used in 1-year death model and the FAST-MI patients
For the 3-year model derived from 1274 patients from the UK, there were 261 deaths: 59 in-hospital (23%), 51 after discharge within 180 days of admission (20%) and 151 in the remaining 2.5 years since admission (58%). The estimated 3-year cumulative death rate is 20.5%.
Performance of the model using non-linear functions
Analyses were undertaken first using categorical variables and linear associations for continuous variables and Killip class (as in the original description of the GRACE risk score),5 ,6 and then using non-linear associations for age, heart rate, systolic blood pressure and creatinine (figure 1A–D). Differences were observed between the non-linear and the linear model with the former more likely to classify patients as at high risk (data not shown).
Non-linear associations for the 1 year mortality model were found for four continuous measures: systolic blood pressure (A), pulse (B), age (C) and creatinine (figure 1D); p<0.001 vs linear for each comparison.
Non-linear associations for the 1-year mortality model were found for all four continuous measures: systolic blood pressure, pulse, age and creatinine (p<0.001 vs linear). The restricted cubic spline (polynomial curve) functions for age and systolic blood pressure had three knots (‘inflection points’) at the 10th, 50th and 90th centiles of their distributions, and four knots at the 5th, 35th, 65th and 95th centiles of pulse and creatinine distributions. HR estimates are reported for selected intervals, to provide a sense of how associations change over covariate ranges (table 2). Killip class is modelled as four distinct groups (p<0.001 vs linear class). The 1-year death/MI model has similar non-linear associations, while the 3-year death model has four knot cubic spline associations for systolic blood pressure and pulse and linear associations for the remaining factors. Also shown are estimates for the substitute factors of renal insufficiency and diuretics, which can be used to replace creatinine and Killip when they are unavailable. Sample sizes increase somewhat for models using the substitute factors, and model discrimination is only slightly diminished.
Summary of Cox regression models
The goodness of fit test is partly a function of sample size with larger sample sizes increasing the chance that a small difference between observed and expected numbers of death will be detected. This was observed, with differences mainly in the 9th risk decile, (the model predicted a 3-year risk of 17%, estimated observed death 19.5%). The largest difference in the remaining deciles is 1.2%.
Based on the relative model χ2 values, age is the most important factor in all three models, followed by systolic blood pressure, creatinine and Killip class in the 1-year model (all have similar χ2 values), creatinine and Killip class in the 1-year death/MI model, and systolic blood pressure and pulse in the 3-year death model. All models show good discrimination (c indices ≥0.74), although combining MI with death in the 1 year model reduces model discrimination, because death and MI are not interchangeable with respect to patient risk profiles.
External validation of the non-linear GRACE risk score in the FAST-MI 2005 registry
The characteristics of the FAST-MI 2005 registry are reflective of the range of patients presenting with ACS (mean age 66.9 years±SD 14.4 years, 31% women, 53% STEMI, 47% non-STEMI, history of coronary artery disease 30%, history of stroke 5%, documented diabetes mellitus 24%, documented hypertension 57%, current smoking 30% and documented hypercholesterolaemia 47.5%). The FAST-MI 2005 registry has excellent completeness of follow-up (3-year follow-up 98% complete). Overall survival was 79% and infarct-free survival was 73%.
Using the FAST-MI 2005 cohort of 2959 patients, c-statistics for death exceeded 0.82 for the overall population at 1 year and also at 3 years (table 3). Discrimination for death in the model was higher in the ST-elevation MI population (c=0.84) at 1 year compared to the non-STEMI population (c=0.80). Discrimination for death or MI was slightly lower than for death alone (c=0.78) both at 1 and 3 years. Similar figures were obtained for hospital survivors (see tables 3 and 4).
The full GRACE risk score tested in FAST-MI 2005
The simplified GRACE risk score, with substitutions for Killip and creatinine (n=3035), tested in FAST-MI 2005
The c-statistics for 3-year death were calculated using the same approach for the whole ACS population and at 3 years were 0.82 for death and 0.75 for death or MI.
The c indices using the simplified GRACE model with substitutions for Killip class and serum creatinine were available for 99.2% of patients; these were 0.82 for both 1 and 3-year models).
In summary, use of non-linear functions for continuous variables improved model performance over the original GRACE risk score using linear functions. The external validation demonstrated good model discrimination at 1 and 3 years for both death and death or MI, and in subtypes of MI, ST-elevation and non-ST-elevation MI. This has not been tested previously. The risk score performs similarly when considering only the survivors of hospitalisation. The simplified risk score using a history of renal dysfunction in place of creatinine values, and using of diuretics in place of Killip class, performed almost as well as the full GRACE score.
Discussion
This study aimed to develop an improved version of the GRACE risk predictor (GRACE score 2.0) incorporating non-linear associations between continuous risk factors and outcomes in a format suitable for ease of use in hand-held electronic devices and smartphones (figure 2). In addition, the revised GRACE risk score allows readily available substitutions for missing variables at the time of first patient presentation (creatinine, Killip score), and this allows the healthcare professional to risk score a more complete range of patients hospitalised with ACS. The score is not dependent on key variables—it allows flexibility in light of data availability. Further, the GRACE score had not been tested for predictive accuracy beyond 6 months, and the simplified version of the risk score with substitutions for creatinine and for Killip class had not been tested in an independent population. A key finding is that model likelihood using individual non-linear functions for heart rate, systolic blood pressure, age and creatinine was significantly improved over a model using linear functions for these factors. In brief, the model with non-linear functions matches observed data more closely. Further, the updated GRACE risk score demonstrated similar high model discrimination at 1 and 3 years as had previously been demonstrated for in-hospital outcomes and outcomes to 6 months. In addition, the reduced version of the GRACE risk score, with substitutions for creatinine and Killip class (with a history of renal dysfunction and use of diuretics, respectively), performs nearly as well as the model with original factors.
Illustration of the Global Registry of Acute Coronary Events (GRACE) Score 2.0 on a mobile device (suitable for use in iOS, android or web versions). Left panel: values for percentage risk of death or death/myocardial infarction (or numerical GRACE Score). Remaining panels show the individual patient results as a vertical column superimposed on the entire acute coronary syndrome distribution curve (green column=low risk illustration, yellow column=medium risk and red column=high risk).34 For further information see http://www.gracescore.co.uk and http://www.outcomes.org/grace.
What are the implications?
In a diverse range of hospitals in 14 countries worldwide, with on-site angiographic facilities, the frequency of catheterisations and percutaneous coronary interventions (PCI) exhibited a paradoxical pattern, whereby most interventions were performed in low-risk rather than high-risk patients (the ‘treatment-risk paradox’).15 ,16
To counter the criticism that not all high-risk patients will be suitable for revascularisation, we undertook further analyses in a previous publication according to the frequency of angiography (hospitals with on-site angiographic facilities were divided into tertiles according to the rate of coronary angiography).15 Hospitals with a high rate of coronary angiography performed substantially more interventions in higher risk patients than those performed in low rate hospitals, despite a similar range of risks to patients, demonstrating that these patients were amenable to the intervention procedures.15
It is possible to estimate the ‘deficit’ in the frequency of revascularisation based on the actual differences between high rate and low rate hospitals observed in the GRACE programme. From the overall population, 37.8% of patients were in the GRACE high-risk group, 36.1% in the GRACE medium risk group and 26.1% in the GRACE lower risk group (categories according to the ESC guidelines).3 As previously reported,15 individuals in the highest third of the GRACE risk score had catheterisation performed in 51% and PCI or coronary artery bypass grafting (CABG) in 31.4% of patients, whereas those in the medium GRACE risk group had catheterisation performed in 68% and PCI or CABG in 42.9% and those in the lower risk group had catheterisation in 72% and PCI or CABG in 47.6%.
Taking the performance of hospitals that were in the highest third for the rate of coronary angiography (they performed PCI and CABG in 60.2% of the presenting population), it is possible to calculate the deficit compared with the hospitals with the lowest rate of angiography and revascularisation. The calculation assumes that the low performance hospitals increased their rate of PCI and CABG to the same extent as was observed in the highest third of hospitals. This projection is based on observed performance data for the rate of angiography. The calculation assumes that no more PCI or CABG was performed than was observed in the high rate hospitals. In brief, 700 more patients per 10 000 would undergo revascularisation if the same patients presented to high performance hospitals.
The impact of revascularisations on outcomes can be estimated from the pooled analysis of all the randomised trials where patients were randomised to an interventional strategy as a routine, or to a selective strategy based on symptoms and ischaemia.29 We previously reported this combined analysis based on individual patient data from the FRISC-2,30 RITA-331 and ICTUS32 trials, and the absolute reduction in cardiovascular deaths and MIs was 11.1 per 100 patients in the highest risk group and 4 per 100 in the medium risk group, over 5 years.29–32 Thus, based on the impact of a systematic interventional strategy in the randomised trials, there would be between 30 and 80 fewer cardiovascular deaths or MIs for each 10 000 patients with non-ST-elevation ACS. The estimate is conservative as it excludes the impact on medium risk patients and the number would be higher if the top quintile of performance was used as the reference standard rather than the top tertile. Thus, consistent with the guideline recommendations, a systematic approach for evaluating risk has the potential to increase the rate of revascularisation in high-risk patients without contraindications. Based on the combined analysis of all the randomised trials with long-term outcomes, this risk-related strategy has the potential to reduce the frequency of cardiovascular death and MI over the longer term. The ‘High’, ‘Medium’ and ‘Low’ risk categories may help guide practice decisions and they correspond with categories used by the ESC guidelines.3 However, for more precise estimates, the GRACE risk score also provides the numerical risk of death (or death/MI) at various time points.
Strengths and limitations
Recognising that population characteristics may differ in comparison with that of the originally derived GRACE model, we employed the most recent GRACE dataset in this study and externally validated the risk model in an entirely separate dataset (FAST-MI 2005 with yearly follow-up to 2010). We have previously reported that the GRACE risk prediction is not subject to significant change over time.33 The purpose of providing 1 and 3-year risk estimates was to aid the clinician regarding secondary prevention.34 The risk prediction estimates at 3 years were consistent with those at 1 year (although the 3-year data derive from a smaller dataset).
The GRACE programme is the largest multinational programme in acute coronary artery disease and was designed to ensure that the included patients were reflective of the broad spectrum of patients presenting with ACS, as well as of the range of hospitals in clinical practice. The sites were trained, audited and quality control measures were enacted throughout the study. Use of the UK cohort allowed estimation of long-term outcomes (as previously reported) with complete mortality data to 5 years.20 The external validation of the updated risk score was performed in the FAST-MI 2005 registry with inclusion of the full spectrum of hospitals admitting patients with ACS and excellent completeness of follow-up.
Although the updated GRACE risk score provides a reliable estimate for stratifying patients both acutely and in the long term, additional factors contribute to longer term risk. Further refinement of the risk score for long-term outcomes may require the inclusion of additional risk factors and biomarkers to increase precision, but the current risk score discrimination allows separation of patients into broad categories relevant for decisions on clinical management. Future studies will determine the impact of risk scoring strategies in various populations including the frail and elderly.
Conclusions
The updated GRACE risk score has better model discrimination and is easier to use than previous scores based on categorical variables. It is accurate in the acute phase and over the longer term and can be used in a variety of clinical settings to aid management decisions.
References
Footnotes
-
Contributors KAAF initiated the programme of work, performed the analyses in conjunction with coauthors and wrote and revised the manuscript. GF led the work of deriving the revised risk score, in conjunction with FA and WH. KC analysed and interpreted the data. ND, in conjunction with EP, TS, PC, JM and P-GS, performed the work on the French registry of Acute ST-elevation and non-ST-elevation MI (FAST MI) dataset. All authors contributed to the revisions of the manuscript and the interpretation of the findings.
-
Funding This work was supported by the British Heart Foundation (CH92010), an award from the Chief Scientist Scotland (CZG/2/455), and an unrestricted grant from AstraZeneca to the University of Edinburgh. The French registry of Acute ST-elevation and non-ST-elevation MI (FAST-MI) registry is a registry of the French Society of Cardiology, supported by unrestricted grants from Pfizer and Servier. Additional support for FAST-MI was obtained from a research grant from the French National Health Insurance System (Caisse Nationale d’Assurance Maladie).
-
Competing interests None.
-
Ethics approval Ethics approval was obtained in all of the participating centres of the GRACE study.5 ,6
-
Provenance and peer review Not commissioned; externally peer reviewed.
-
Data sharing statement The GRACE risk score has been made available to download without cost (http://www.gracescore.co.uk and http://www.outcomes.org/grace).