Article Text


A cross-sectional multicentre study of cardiac risk score use in the management of unstable angina and non-ST-elevation myocardial infarction
  1. Josien Engel1,
  2. Ineke van der Wulp1,
  3. Martine de Bruijne1,
  4. Cordula Wagner1,2
  1. 1Department of Public and Occupational Health, EMGO Institute for Health and Care Research, VU University Medical Center, Amsterdam, The Netherlands
  2. 2NIVEL, Netherlands Institute for Health Services Research, Utrecht, The Netherlands
  1. Correspondence to Josien Engel; j.engel1{at}


Objectives Quantitative risk assessment in unstable angina (UA) and non-ST-elevation myocardial infarction (NSTEMI), by using cardiac risk scores, is recommended in international guidelines. However, a gap between recommended care and actual practice exists, as these instruments seem underused in practice. The present study aimed to determine the extent of cardiac risk score use and to study factors associated with lower or higher cardiac risk score use.

Setting 13 hospitals throughout the Netherlands.

Participants A retrospective chart review of 1788 charts of patients with UA and NSTEMI, discharged in 2012.

Primary and secondary outcomes The extent of cardiac risk score use reflected in a documented risk score outcome in the patient's chart. Factors associated with cardiac risk score use determined by generalised linear mixed models.

Results In 57% (n=1019) of the charts, physicians documented the use of a cardiac risk score. Substantial variation between hospitals was observed (16.7–87%), although this variation could not be explained by the presence of on-site revascularisation facilities or a hospitals’ teaching status. Obese patients (OR=1.49; CI 95%1.03 to 2.15) and former smokers (OR=1.56; CI 95%1.15 to 2.11) were more likely to have a cardiac risk score documented. Risk scores were less likely to be used among patients diagnosed with UA (OR=0.60; CI 95% 0.46 to 0.77), in-hospital resuscitation (OR=0.23; CI 95% 0.09 to 0.64), in-hospital heart failure (OR=0.46; CI 95% 0.27 to 0.76) or tachycardia (OR=0.45; CI 95% 0.26 to 0.75).

Conclusions Despite recommendations in cardiac guidelines, the use of cardiac risk scores has not been fully implemented in Dutch practice. A substantial number of patients did not have a cardiac risk score documented in their chart. Strategies to improve cardiac risk score use should pay special attention to patient groups in which risk scores were less often documented, as these patients may currently be undertreated.

  • acute coronary syndromes
  • risk assessment
  • guideline adherence
  • predictors
  • chart review

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See:

Statistics from

Strengths and limitations of this study

  • This multicentre study is the first to provide thorough insight into the extent of cardiac risk score use in clinical practice by reviewing patient charts in 13 hospitals throughout the Netherlands.

  • Generalised linear mixed models were used to study hospital-related and patient-related factors associated with lower or higher cardiac risk score use.

  • The results can serve as a basis for future improvement initiatives aimed to enhance cardiac risk score use in practice, by tailoring strategies to patient groups in which risk scores are less often documented.

  • It was not possible to study associations between the extent of cardiac risk score use and the occurrence of adverse patient outcomes (eg, death) due to practical constraints.

  • Using retrospective review of medical charts as the underlying method of data collection made it impossible to determine the actual influence of cardiac risk score use on clinical decision-making of healthcare providers involved in the management of patients with unstable angina or non-ST-elevation myocardial infarction.


In the past decade mortality rates in acute coronary syndromes, including unstable angina (UA) and non-ST-elevation myocardial infarction (NSTEMI), decreased significantly due to substantial improvements in treatment possibilities.1 ,2 Despite these advancements, these conditions still account for a large part of the annual deaths worldwide and are expected to be the leading cause of death and to account for the largest disease burden worldwide by 2020–2030.3–5 Part of these deaths may be prevented, as it has previously been reported that a substantial number of patients were not treated according to the current standards of care.6–8 Patients with diabetes mellitus, renal insufficiency, signs of heart failure and patients aged 75 years or older were often neglected guideline recommended care.6 On the other hand, patients presenting to academic hospitals and to hospitals with revascularisation facilities onsite (eg, percutaneous coronary intervention (PCI) or coronary artery bypass grafting (CABG)) were more often treated in accordance to the guidelines.7 ,9 Patients diagnosed with UA or NSTEMI can be treated with medication or invasive procedures such as PCI or CABG. According to international cardiac guidelines, the decision to treat such patients with one or the other may be made on the basis of a quantitative assessment of the patient's risk of reinfarction or death.10–12 To assist clinicians in identifying patients at high risk of adverse cardiac events that would benefit most from invasive therapies, several instruments have been developed.10–12 The GRACE (Global Registry of Acute Coronary Events),13 ,14 TIMI (Thrombolysis in Myocardial Infarction),15 FRISC (fast revascularisation in instability in coronary disease),16 PURSUIT (Platelet glycoprotein IIb/IIIa in Unstable angina: Receptor Suppression Using Integrilin)17 and HEART risk scores18 are examples of validated cardiac risk scoring instruments. In estimating risk, these instruments incorporate and combine several diagnostic elements including a patient’s history, biomarkers and ECG findings, and can be used in the emergency department or coronary care unit. The predictive validity of these instruments was reported to be good.16 ,19 ,20 Previous research found that cardiac risk scores were effective in identifying patients at high risk for cardiac events.21 ,22 However, a gap between recommended care in the guidelines and actual practice seems to exist, as it has been suggested before that cardiac risk scores are not routinely used in clinical practice.21 ,23 ,24 This possibly contributes to perpetuating the ‘treatment risk paradox’, in which patients with low risk of adverse cardiac events, opposite to cardiac guideline recommendations, were more likely to receive invasive cardiac treatment compared with high-risk patients.6 ,25–30 Prior to creating future improvement initiatives aimed to increase cardiac risk score use, knowledge about the extent of this gap and associated factors is necessary. The present study, therefore, aimed to determine the extent of cardiac risk score use in Dutch clinical practice and to study factors associated with lower or higher cardiac risk score use.


This study concerns a cross-sectional multicentre study. A detailed description of the study protocol has been published previously.31 Where required, approval from hospitals’ local ethics board was obtained.


In 2008, all hospitals in the Netherlands committed themselves to the implementation of a quality improvement programme aimed to enhance patient safety in Dutch hospitals. The programme comprised several themes, including the theme ‘Optimal care for Acute Coronary Syndromes’ which, among other things, aimed to increase the application of cardiac risk scores in clinical practice.32 A random selection of 40 hospitals participated voluntarily in the evaluation of the nationwide quality improvement programme. By a multistage random sampling procedure, initially 12 hospitals were selected from the pool of 40 hospitals to participate in the current study (ie, evaluation of cardiac risk score use). Three PCI-capable hospitals declined participation, for which three additional PCI-capable hospitals were selected. Additionally, one hospital was selected to obtain optimal diversity in onsite revascularisation facilities and teaching status. The final sample consisted of 13 hospitals, of which 2 university hospitals, 7 tertiary teaching hospitals and 4 general hospitals. Bed capacity in the hospitals varied between 200 and 1200 beds.

Data collection

The primary study outcome was the extent to which cardiac risk scores were used in the management of patients with UA and NSTEMI, reflected in a documented risk score outcome in the patient's chart. Data were collected monthly by means of retrospective chart review.

Potentially eligible charts were selected from the hospitals’ billing system based on diagnostic-related group codes for UA and NSTEMI. All patients discharged in 2012, 18 years or older, with a diagnosis of UA or NSTEMI (as confirmed in the discharge letter) were considered for inclusion. Charts of patients who were transferred in from another hospital were excluded, as these patients were initially treated elsewhere and therefore the necessary data could not be obtained. In addition, charts of patients who provided insufficient information regarding the discharge diagnosis, who were hospitalised for an elective procedure, or who had an underlying illness or condition, other than a coronary stenosis, causing UA or NSTEMI (eg, anaemia) were excluded.

Charts of patients were selected per month in chronological order of discharge, until the screening capacity of the chart abstractors was reached. Charts of potentially eligible patients were manually reviewed to confirm a discharge diagnosis of UA or NSTEMI. In case a patient's final discharge diagnosis was unclear, a physician of the cardiology department was consulted. The following patient-related and hospital-related information was registered on standardised data extraction forms: demographic characteristics, cardiac history, presence of cardiac risk factors, presenting symptoms, biochemical and ECG findings and treatment practices. In addition, information regarding cardiac risk score use was registered, including the use of a validated risk score (yes/no), date of application, type of risk score used and risk score outcome and classification. Besides patient-related information, the following hospital factors were registered: teaching status (yes/no) and the presence of onsite revascularisation facilities.

The data were entered into a database using fixed entry fields (BLAISE V.4.7, Statistics Netherlands) and data reliability checks were conducted. To ensure reliable data extraction, more than 5% (103/1933) of the charts were screened by two chart abstractors independently. The total percentage of agreement between these abstractors was 95.1%, and ranged for the variables of interest (table 1) between 80.6% (ECG findings) and 100% (gender), indicating good to excellent data reliability.

Table 1

Independent variables in generalised linear mixed model

Missing data

In total, 1.5% of the values in the data set were missing, ranging from 0.1% to 22% per variable. Eleven variables had no missing values, including cardiac risk score use. Despite the small amount of missing data and the spread of missing data in the data set, a complete case analysis would have led to a large loss of information and power. Therefore, missing values were imputed using a multiple imputation procedure following the approach of van Buuren and Groothuis-Oudshoorn,33 resulting in five imputed data sets. In imputing missing values, it was assumed that the data were missing at random. The estimated values were corrected for the variables ‘hospital’ and ‘cardiac risk score use’ as these variables were of primary interest in the analyses. By means of the Kolmogorov-Smirnov test, and density and residual plots, it was determined whether the missing at random assumption was sustainable and the imputed values were plausible. In addition, a sensitivity analysis was conducted by comparing the results from the analyses of the imputed data with the results of a complete case analysis. Between these models, only small differences were found. The missing value analyses and their imputations were conducted in R (V.3.0.2 for Microsoft Windows) using the MICE package.33 ,34

Data analysis

Sample characteristics were calculated using descriptive statistics, and included frequencies and percentages for categorical variables, and means and SDs for continuous variables. Associations of independent variables (table 1) with the use of cardiac risk scores (yes/no) were studied with a generalised linear mixed model (GLMM), taking into account the clustering of data within hospitals.35 ORs, that are based on median probabilities over hospitals for cardiac risk score use, are presented. To facilitate interpretation, relevant explanatory variables were transformed into categorical variables (ie, age, heart rate and systolic blood pressure). Furthermore, month of discharge was represented by a categorical variable with 12 levels in every model, to account for the fact that chart abstractors were present on hospital departments to abstract data. In univariate analyses, associations between cardiac risk score use and the independent variables were tested. All variables with a significance level of p≤0.15 were entered in a multivariable model. Variables significantly associated (p≤0.05) with cardiac risk score use in the multivariable model were considered important in predicting risk score adherence. In addition, based on previous literature two factor interactions with onsite revascularisation options, teaching status, age and gender were tested. All analyses were conducted in R for windows (V.3.0.2) using the package lme4 on pooled data of five imputed data sets.34 The script of the pool function in MICE was rewritten for pooling GLMM models.


Study population

A total of 1933 charts of patients with a confirmed diagnosis of UA or NSTEMI were screened. Of these, 145 (7.5%) were excluded from the study as these concerned patients transferred from one hospital to another, leaving 1788 patients for further analysis (figure 1). The majority (62.6%) of these patients had a discharge diagnosis of NSTEMI (table 2). Males accounted for 66.9% of the patients, and more than a third (35.9%) of the patients were aged 75 years or older. Three-quarters (75.3%) of the total population underwent coronary catheterisation. The average length of hospital stay was 5 days (SD=4.97).

Table 2

Patient and admission characteristics of study sample

Figure 1

Inclusion and exclusion procedure of chart selection and screening (NSTEMI, non-ST-elevation myocardial infarction; UA, unstable angina).

Cardiac risk score use

In 57% of the patient charts, a cardiac risk score was documented, though substantial variation between hospitals was observed, that is, 16.7–87% (table 3). Six out of the 13 hospitals used more than one risk scoring instrument to calculate a risk score, being the following: GRACE (12/13 hospitals), TIMI (3/13 hospitals), FRISC (1/13 hospitals) and the HEART risk score (6/13 hospitals; table 3). The variance component for the random hospital effect in the GLMM ranged between 1.29 and 1.31 in the five imputed data sets, confirming the great variety between hospitals in the use of cardiac risk scores. When, for instance, the effects for two hospitals are equal to the 5th and 95th centiles of the normal distribution with variance 1.3 for hospital effects, the OR of one hospital relative to the other for cardiac risk score use is 42.6.

Table 3

Adherence to cardiac risk score use per hospital (pooled data)

In univariate analyses, 15 patient-related factors were significantly (p≤0.15) associated with cardiac risk score use (table 4). No significant associations with hospital-related factors were found (teaching status p=0.25, onsite revascularisation facilities p=0.67). In multivariable analyses, patients with obesity (OR=1.49; 95%CI 1.03 to 2.15; p=0.04) and former smokers (OR=1.56; 95% CI 1.15 to 2.11; p≤0.01) were more likely to have a cardiac risk score documented. Conversely patients with UA (OR=0.60; 95% CI 0.46 to 0.77; p≤0.01), in-hospital heart failure (OR=0.46; 95% CI 0.27 to 0.76; p≤0.01), tachycardia (OR=0.45; 95% CI 0.26 to 0.75; p≤0.01) or who had been resuscitated at admission (OR=0.23; 95% CI 0.09 to 0.64; p≤0.01) were less likely to have a cardiac risk score documented (table 4).

Table 4

Univariate and multivariable associations between risk score documentation in patient charts and hospital-related and patient-related factors (pooled data; n=1788)†


Besides the interactions with onsite revascularisation options, teaching status, age and gender, it was decided to also test whether interactions with former smoker were present. This, because an unexpected significant association between former smoker and risk score use was found. Significant interactions were found between the variables former smoker and discharge diagnosis (p=0.03), age and previous PCI (p=0.02), age and in-hospital heart failure (p=0.04), age and history of peripheral artery disease (p=0.03), and age and heart rate (p=0.04; table 5).

Table 5

Estimated ORs and 95% CIs for significant (p≤0.05) interactions terms added to multivariable model of GLMM (pooled data)†

Looking at the interaction effects with age, it was found that patients aged 75 years or over presenting with a previous PCI had a higher odds of cardiac risk score documentation compared with patients without a previous PCI (OR=1.53; 95% CI 1.00 to 2.34; p=0.05). In contrast, older patients were less likely to have a cardiac risk score documented in case they presented with heart failure (OR=0.29; 95% CI 0.14 to 0.57; p<0.001), with a history of peripheral artery disease (OR=0.47; 95% CI 0.24 to 0.91; p=0.02) or with tachycardia (OR=0.20; 95% CI 0.08 to 0.52; p≤0.001).


This study aimed to provide insight in the extent of cardiac risk score use in Dutch hospitals as recommended by international cardiac guidelines. In addition, associations with patient-related and hospital-related factors were studied. Substantial variation between hospitals’ cardiac risk score use was observed, with in approximately 40% of patient charts a cardiac risk score was not documented. Several patient-related factors including a diagnosis of UA, the presence of in-hospital heart failure, tachycardia and resuscitation at admission were associated with a lower likelihood of cardiac risk score use. Although evidence is not conclusive, the probability of cardiac risk score use was often lower in older patients (≥75 years) with additional conditions, such as in-hospital heart failure, a history of peripheral artery disease or tachycardia.

Previous studies also reported advanced age, heart failure and tachycardia as important predictors of lower guideline adherence in patients with acute coronary syndromes.6 ,36–41 Moreover, several of these studies also reported a decreased likelihood of survival.37 ,38 ,40 Implying that patients at high risk for adverse cardiac outcomes are less likely to receive guideline recommended care. However, according to the European guidelines, these high-risk subgroups of patients benefit most from early invasive treatments.10 It may, however, be discussed to what degree an invasive treatment may be desired in these high-risk subgroups of patients. Also, it could be questionable to what degree risk stratification using a cardiac risk score adds value in deciding on the treatment for these patients, for example, in the case of resuscitation the decision for a certain procedure may be evident. The European Society of Cardiology guidelines, however, do not take these circumstances into account and recommend to estimate risk levels with a cardiac risk scoring instrument for every patient suspected of UA/NSTEMI.10

Obese patients and former smokers were more likely to have a cardiac risk score documented. The association of former smoking and the use of a cardiac risk score, however, was unexpected and difficult to explain. There are no indications for partial confounding with other factors in the model as ORs for former smoker in univariate and multivariable models are sizeable and similar. Possibly, former smoking is an alias for some other underlying and unknown variable. For instance, former smoking is seen as an indication of a former more high-risk lifestyle and that way affects judgement. Further research may provide more insight on this.

Another interesting finding, that contrasted the findings of previous studies, was that a hospitals’ teaching status or the presence of onsite revascularisation facilities were not significantly associated with cardiac risk score use.7 ,36 These differences may be explained by the relatively small number of hospitals participating in the present study compared with previous studies. A large variation between hospitals in adherence scores regarding cardiac risk score use was found. The large component of variance, explained by the random hospital effect, suggests that cardiac risk score use in patients presenting with the same characteristics may heavily depend on which hospital the patient is presented in, and that other factors, beside a hospital's teaching status or onsite revascularisation facilities, are of influence. Common barriers in the implementation of cardiac risk scores, including the absence of necessary resources for implementation and cultural differences, may explain this substantial variation.42 Also, it has been suggested that physicians find the evidence underlying cardiac risk scores unconvincing.24 To increase the use of cardiac risk scores in clinical practice several implementation strategies, which pay explicit attention to patients with suspected UA, may be employed. A recent improvement initiative in the USA for instance, in which continuous education was the primary intervention, led to a significant increase in cardiac risk score documentation in patients with UA and NSTEMI.43 The use of continuous education has proven to be effective in achieving change in practice, however, it is recommended to also take into account facilitating factors and barriers on a patient, provider and organisational level.44 Therefore, further research is needed to carefully understand factors that explain the variation between hospitals’ cardiac risk score use.

Study limitations

Several limitations potentially affect the interpretation of the results of this study.

First, the use of cardiac risk scores was measured by screening charts on the documentation of a cardiac risk score. As a result, it is unknown to what degree a cardiac risk score influenced physicians’ decision-making regarding appropriate management strategies. However, it is plausible that when a cardiac risk score was documented, it was also used in practice.

Second, four predictors reported in previous studies of risk score use, that is, aspirin use prior to admission, creatinine level, troponin level and biomarkers, were not considered in the present study. These data could not be abstracted reliably. As a result, the precision of the model reported in this study might be smaller compared with other studies. In addition, it was not possible to reliably extract at what time point a risk score was recorded. The time registered in the patient's file was often imprecise (ie, time was entered retrospectively and did not represent the actual time point at which the risk score was used) or lacking. Making it impossible to provide any additional contextual information regarding the use of cardiac risk scores in clinical practice.

Third, in two hospitals, the method of selection of patient charts differed, as in these hospitals it was not possible to select patients based on the hospital's billing system. This could have influenced the selection of patients. However, their effects may be limited as it appeared that the random effects of these two hospitals were well in range with those of the other hospitals.

Fourth, it was not possible to extract all data from the charts at one time point per hospital. Therefore, monthly data collection visits were deemed necessary. For this reason, the reported associations were corrected for month of discharge. However, the frequent presence of the researchers onsite may have led to more awareness of the healthcare providers using cardiac risk scores, and as a result have higher adherence scores than hospitals not participating in the evaluation of the quality improvement programme. This overestimation of adherence rates can also be a result of the fact that the evaluation of the improvement programme took place in a cohort of highly motivated hospitals, as they all voluntarily agreed to participate.

Finally, three of the randomly selected hospitals declined participation in this study, which may have introduced selection bias. Hospitals that declined participation were possibly lagging behind in implementation. The actual use of cardiac risk scores in practice might therefore be even lower than estimated in this paper.


The results of the present study indicate that cardiac risk scores have not been fully implemented in Dutch clinical practice, as a substantial number of patients had no risk score documented in their chart. The large variation between hospitals could not be explained by the presence of onsite revascularisation facilities or a hospitals’ teaching status, as well as by several patient-related factors that were associated with higher or lower usage of cardiac risk scores in clinical practice. It is recommended that further research should focus first on explanatory factors for differences between hospitals, which could provide a basis for future improvement initiatives in which strategies are targeted towards patient groups in which risk scores were less often documented, as these patients may currently be undertreated.


The authors would like to thank Marie-Julie Heeren, Felix van Urk, Suzanne Vonk and Rixt Zuidema for their support in data collection.


View Abstract


  • Contributors All authors made substantial contributions to conception and design of the study. JE carried out data collection, performed statistical analysis, and drafted the manuscript, and prepared the manuscript for publication. IvdW provided substantial support in imputation of data and statistical analysis. IvdW, MdB and CW made substantial contributions to interpretation of the data and revised the manuscript critically for important intellectual content. All authors read and approved the final manuscript.

  • Funding The study was funded by the Dutch Ministry of Public health, Welfare and Sports

  • Competing interests None declared.

  • Ethics approval Medical ethical committee of the VU University Medical Center Amsterdam.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Data sharing statement No additional data are available.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.