Article Text
PDF
Abstract
Objective To externally validate and extend a recently proposed prediction model to diagnose obstructive coronary artery disease (CAD), with the ultimate aim to better select patients for coronary angiography.
Design Analysis of individual baseline data of a prospective cardiology cohort.
Setting Single-centre secondary and tertiary cardiology clinic.
Participants 4888 patients with suspected CAD, without known previous CAD or other heart diseases, who underwent an elective coronary angiography between 2004 and 2008 as part of the prospective Coronary Artery disease Risk Determination In Innsbruck by diaGnostic ANgiography (CARDIIGAN) cohort. Relevant data were recorded as in routine clinical practice.
Main outcome measures The probability of obstructive CAD, defined as a stenosis of minimally 50% diameter in at least one of the main coronary arteries, estimated with the predictors age, sex, type of chest pain, diabetes status, hypertension, dyslipidaemia, smoking status and laboratory data. Missing predictor data were multiply imputed. Performance of the suggested models was evaluated according to discrimination (area under the receiver operating characteristic curve, depicted by the c statistic) and calibration. Logistic regression modelling was applied for model updating.
Results Among the 4888 participants (38% women and 62% men), 2127 (44%) had an obstructive CAD. The previously proposed model had a c statistic of 0.69 (95% CI 0.67 to 0.70), which was lower than the expected c statistic while correcting for case mix (c=0.80). Regarding calibration, there was overprediction of risk for high-risk patients. All logistic regression coefficients were smaller than expected, especially for the predictor ‘chest pain’. Extension of the model with high-density lipoprotein and low-density lipoprotein cholesterol, fibrinogen, and C reactive protein led to better discrimination (c=0.72, 95% CI 0.71 to 0.74, p<0.001 for improvement).
Conclusions The proposed prediction model has a moderate performance to diagnose obstructive CAD in an unselected patient group with suspected CAD referred for elective CA. A small, but significant improvement was attained by including easily available and measurable cardiovascular risk factors.
- diagnosis
- obstructive coronary artery diseas
- prediction
- validation
- clinical epidemiology
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Strengths and limitations of this study
In our study of obstructive coronary artery disease diagnostics in a large cohort, we could analyse prospectively collected, good-quality data including various routine laboratory measurements.
The results reflect those of patients in daily clinical routine where one has to evaluate the available information for the diagnostic work-up, a situation in which a prediction model can be an aid to make the best possible decisions.
Not all information was complete, but multiple imputations counteracted this deficiency.
An effect of verification bias cannot be ruled out and more elaborate evaluation of chest pain symptoms might be advantageous.
Introduction
Since conventional coronary angiography (CA) is expensive and has a small, but not negligible risk of complication, prior diagnostic testing is recommended, especially in patients with an intermediate pretest probability of obstructive coronary artery disease (CAD). However, the use of multiple non-invasive tests in an individual patient is common which again might be costly and not free of risk (because of radiation and exposure to contrast medium). In this context, Diamond and Forrester1 presented the now widely known risk prediction model about 35 years ago, which was followed by efforts to expand it for an even better diagnostic work-up.2 Genders et al3 proposed an update and extension using recent data of a consortium from predominantly European countries.
Although being solely a ‘luminography’, invasive CA is still the reference standard to diagnose an obstructive CAD. Given the limitations of available clinical prediction models and non-invasive tests, a high proportion of CA in daily practice reveal no obstructive CAD or even no CAD at all and, hence, offer no improvement in patient management in terms of subsequent revascularisation for symptom relief or prognosis. For example, in about 43% of the cases in a study of 18 hospitals involving 2062 patients, an obstructive CAD was excluded.3 If a CA could be more precisely restricted to those patients with obstructive CAD (by use of a thorough cardiovascular evaluation including laboratory measures and a careful clinical assessment, summarised in a prediction model), the potential benefits of avoiding unnecessary invasiveness, non-invasive testing, costs, inconvenience to the patients and probably anxiety are obvious.
We can estimate the probability of obstructive CAD for adult patients with a diagnostic model based on age, sex and type of chest pain (the Diamond and Forrester model,1 labelled ‘basic model’ by Genders et al3). A contemporary and more extensive clinical model also considered traditional cardiovascular risk factors like diabetes, hypertension, dyslipidaemia and smoking status.3 Here, we aim to validate this proposed model in an unselected cohort of patients with suspected CAD referred for elective invasive CA and to test the incremental predictive value of an extension with standard laboratory parameters.
Materials and methods
Patients
Between February 2004 and April 2008, 8296 consecutive patients, referred for elective CA to invasively evaluate the prevalence and severity of obstructive CAD, were included in the Coronary Artery disease Risk Determination In Innsbruck by diaGnostic ANgiography (CARDIIGAN) cohort. Patients were 18 years of age or older with chest pain or symptoms suggestive of CAD (predominantly dyspnoea) and/or non-invasive evidence of CAD referred for elective CA. Patients were excluded when undergoing (1) an elective CA before or after heart transplantation, (2) an elective CA prior to solid organ transplantation, (3) an elective CA before heart valve repair or replacement, or with valvular heart disease as leading clinical diagnosis, (4) an isolated right heart catheterisation, (5) an electrophysiological procedure (pace-maker implantation or catheter ablation) as leading clinical indication, (6) an elective CA because of a known or suspected congenital heart disease as leading clinical diagnosis (eg, atrial septal defect, ventricular septal defect or patent foramen ovale), or (7) when referred for other reasons (like myocardial biopsy, aortic aneurysms, myxoma, endocarditis or prior failed angiography). Patients referred for an acute non-elective CA in the setting of acute coronary syndrome (ie, for a primary or rescue percutaneous coronary intervention (PCI) for ST elevation myocardial infarction or as an early invasive strategy for non-ST elevation myocardial infarction) had not been recorded in CARDIIGAN, but participated in the Austrian Acute-PCI registry. Also, those with a history of myocardial infarction, either recent (within 6 months) or prior (more than 6 months ago), were excluded as they have known CAD. Overall, there were 5606 patients included (68%) (with 5929 visits) and of these 4888 (87%) were first-time patients (without a history of coronary revascularisation, ie, PCI or coronary artery bypass grafting) available for the current study.
The data were primarily gathered in a prospective quality enhancement initiative with cardiovascular risk factor assessment, but in the normal daily routine, and extended with a retrospective patient record extraction. The hospital is a local secondary facility and also has a larger regional, tertiary function with specialised academic healthcare. The patients were asked for their written informed consent for the CA on a routine basis.4 The first author had access to all data in the study and is responsible for its processing.
Data
Information was recorded on basic characteristics of the patients, medical history, symptoms (present or absent), laboratory results and therapy decision immediately in the catheterisation laboratory. Concerning symptoms, patients were subdivided into unstable, stable and those without any symptoms of angina pectoris; in the current analysis those with unstable and stable symptoms were merged together with those with chest pain. The differentiation of the two types of angina is theoretically clear, but generally not in clinical practice where symptom description and allocation vary intraindividually and interindividually. The ‘unstable’ category was rather small (16% of patients). A sensitivity analysis was performed later to investigate the effect of other classifications of the predictor or of leaving out the unstable chest pain patients; the results were very similar. Past myocardial infarctions, prior PCI and prior bypass grafting were assessed, as well as family history of myocardial infarction (male first-degree relatives below 55 years and female first-degree relatives below 65 years), hypertension (including treatment), diabetes mellitus (including treatment),5 known cardiomyopathy and rhythm on baseline ECG. Also, baseline levels of total cholesterol, high-density lipoprotein (HDL) cholesterol, low-density lipoprotein (LDL) cholesterol, triglycerides, fibrinogen, C reactive protein, thrombocytes, haemoglobin, urea, creatinine, γ-glutamyltransferase (measured at 37°C), calcium, phosphate and uric acid were recorded. In addition, prothrombin time was measured and the glomerular filtration rate estimated according to the Modification of Diet in Renal Disease 2 (MDRD2) formula:6
estimated glomerular filtration rate=186×creatinine(−1.154)×age(−0.203)×(0.742 if woman)
In the multivariable analyses, triglycerides, fibrinogen and γ-glutamyltransferase were log-transformed because of skewed distributions. C reactive protein was dichotomised into ≤1.00 vs >1.00 mg/dL. Dyslipidaemia was defined as total cholesterol ≥200 mg/dL or treatment with lipid-lowering drugs.3
CA was performed by using standard Judkins technique, predominantly via a right femoral access. We used the routine ad hoc judgements of the treating interventional cardiologists, who all had ample experience in interpreting CAD by angiography, reflecting a real-life setting. Despite known limitations, visual estimation was applied to define obstructive CAD, instead of using quantitative coronary angiographic techniques or even pressure wire measurements, as it is the most commonly used method in clinical routine. The following segments of the coronary tree were separately evaluated and categorised by visual estimation of lumen diameter reduction into 0%, 1–49%, 50–69% and ≥70%: left main stem, left anterior descending artery (proximal and non-proximal separately), circumflex artery including relevant marginal branches (>2 mm diameter), diagonal branches (>2 mm diameter) and right coronary artery. Prior coronary interventions were recorded as plain old balloon angioplasty or stent implantation, graded as a significant restenosis absent or present. When dichotomising CAD in the main analysis as obstructive or not, the cut-off was set at 50% stenosis. Otherwise, the cut-off was 70%, and the left main artery counted as three vessels (where ≥50% stenosis was reset to ≥70%) and the three categories of the left anterior descending arteries counted as one.7 ,8
Statistical analysis
Patient characteristics are presented as proportions for categorical and by means and SDs (or medians and IQR) for continuous variables. Overall, 1.7% of the clinical data were missing, in contrast to the medication information with about 46% missing in seven variables (concerning 60% of the patients). Since the missing clinical data were not confined to a particular group of participants, in total ∼28% of them had one or more missing values. To avoid possible biases, we imputed the failing information numerous times using all available information9 after an extensive exploration of the missingness.10 The multiple imputations involved a total of 20 imputations, with application of the Markov Chain Monte Carlo technique.11
Multivariable logistic regression modelling was applied to update the CAD prediction model proposed by Genders et al.3 ,12 Since our cardiology department provides healthcare for a rather wide range of patients, the characteristics of the present cohort might differ to those of the development model. Therefore, a ‘case mix corrected c statistic’13 was estimated as a benchmark value, as well as the c statistic when refitting the regression model to the current data.14 The model updating included an extension with additional predictors, evaluated through likelihood ratio tests of the model extensions in a forward stepwise manner, each time considering the predictor with the strongest difference first.12
For assessment of the performance of a prediction model, discriminative ability and calibration are essential.15 Discrimination is the degree to which a prediction model distinguishes between patients with an outcome (significant stenosis) and those without. Calibration captures the correspondence between the observed outcomes (diagnosis of obstructive CAD) and the predictions.12 As the measure for discrimination, the area under the receiver operating characteristic curve is commonly used; in a situation with a binary outcome, as in this case, this is the same as the concordance (c) statistic used in generalised linear regression models. To evaluate the agreement of observed outcomes and predictions, calibration-in-the-large and the calibration slope are examined together with the calibration plot.16 This graph depicts the ascending prediction probability of CAD, commonly divided into 10 equal groups, set off against the corresponding actually observed relative frequency. Here, we estimated a flexible, non-linear, calibration curve with a 95% band of confidence.17 Additionally, we conducted a decision curve analysis to assess the clinical consequence of the models in comparison with a default approach of angiography for all included patients.18
The cut-off value for statistical significance was set at a two-sided p value of 0.05. The data preparation was performed with SPSS V.19.0, and the analyses with Stata/MP V.11.2 (14.1 for the decision curve analysis) and R V.3.1 software.
Results
Overall, 2127 of the 4888 (44%) patients had obstructive CAD as diagnosed by invasive CA. Patients with stenosis were predominantly men, older and more often reported chest pain, compared with those without an obstructive CAD diagnosis. Furthermore, the risk factor burden was higher among these patients (table 1).
Baseline characteristics of the study patients by 50% stenosis status
When differentiating obstructive CAD by its severity, the same univariate predictors showed a trend across the categories of no CAD, non-obstructive CAD, and one-vessel, two-vessel and three-vessel disease (table 2). For example, in the case of HDL cholesterol the mean level in the first category was 61 mg/dL and was lower in every next one, with 52 mg/dL among the two-vessel and three-vessel disease patients. The other predictors generally showed higher values with more severe CAD.
Baseline characteristics of the study patients by CAD* status
The c statistic for the previously proposed model was 0.69 (95% CI 0.67 to 0.70); at derivation the model had attained c=0.79.3 Based on the case mix, the expected discriminative ability was even higher than at development (c=0.80). Calibration was poor (figure 1A). The predictions encompassed practically the whole range from 0% to 100%, whereas the observed proportion of obstructive CAD cases was between 5% and 75%. Recalibration analysis showed a new intercept of −1.04 (95% CI −1.10 to −0.97; reflecting the lower observed than expected prevalence) and a calibration slope of 0.63 (95% CI 0.58 to 0.69; reflecting overall smaller than expected effects of the diagnostic characteristics). Additionally, the separate predictors were tested for model updating; sex and chest pain gave significant results (difference in coefficients 0.42 (p<0.001) and −0.42 (p<0.001), respectively). With complete re-estimation, all the new coefficients were considerably smaller than the original estimates, except for dyslipidaemia (see online supplementary table, re-estimated model). The discrimination of this re-estimated model was slightly higher than when applying the original model (c=0.70, 95% CI 0.68 to 0.71). The range of predictions is shrunk, with the maximum at about 85% (figure 1B).
Calibration plots of the predicted probability against the observed proportion of coronary artery disease (≥50% stenosis) of the 4888 CARDIIGAN patients with 95% confidence band for (A) the clinical model of Genders et al,3 (B) the re-estimated model and (C) the extended model; below the main graph the frequency distribution of coronary artery disease cases (upward) and without stenosis (downward) is shown. CARDIIGAN, Coronary Artery disease Risk Determination In Innsbruck by diaGnostic ANgiography.
supplementary table
Extending the model to include the laboratory results, HDL and LDL cholesterol, fibrinogen, and C reactive protein contributed exceedingly to the recalibrated model next to chest pain, hypertension and smoking (each p<0.02). The discrimination increased to c=0.72 (95% CI 0.71 to 0.74). The fully updated model (see online supplementary table, extended model) showed smaller coefficients than the previously proposed model for age, chest pain, diabetes, hypertension and smoking, and the interaction of chest pain and diabetes was negligible. The laboratory predictors had small coefficient values (comparison of the 75 vs 25 centiles: HDL cholesterol OR=0.67, LDL cholesterol OR=1.28, fibrinogen OR=1.27 and C reactive protein raised vs low OR=1.27).
With a cut-off at 50% estimated probability of a stenosis, the improvement in specificity of the extended model, in comparison to the re-estimated one, was moderate increasing from 74.5% (95% CI 72.9% to 76.1%) to 76.0% (95% CI 74.3% to 77.5%). Sensitivity however gained in the new model, from 52.3% (95% CI 50.2% to 54.4%) to 55.7% (95% CI 53.6% to 57.8%). From the decision curve analysis, it appeared that the extended model had an extra net benefit in the range of threshold probability between about 20% and 70% (figure 2), being larger than when subjecting none or contrarily all patients to angiography. Threshold probability (ie, the probability above which it is chosen to perform angiography) can vary according to patient preference and depends on appraisal by the physician. When set at 30%, sensitivity is 0.88, specificity 0.40 and the extra net benefit due to the model 0.04; at 40%, these measures are 0.74, 0.58 and 0.11, respectively. Per 100 patients about 10 and 16 angiographies, respectively, can thus be saved.
Decision curve for two prediction models of coronary artery disease (≥50% stenosis) of the 4888 CARDIIGAN patients. Note: the horizontal dotted line at net benefit=0.00 assumes that no patients are subjected to diagnostic work-up, the first descending dashed curve (cutting the horizontal dotted line of net benefit=0.00 at a threshold probability=0.44) is based on the assumption that all patients undergo diagnostic work-up for stenosis, the next long-dash-and-dot curve to the right displays the re-estimated model, and the upper solid line the extended model. CARDIIGAN, Coronary Artery disease Risk Determination In Innsbruck by diaGnostic ANgiography.
Discussion
Validation of the Genders et al3 model in the CARDIIGAN cohort showed a lower discriminative ability and poor calibration of risk predictions. We confirmed that the previously suggested predictors were associated with the presence of an obstructive CAD, but effects were weaker at validation than at development. We also found consistent results according to severity of obstructive CAD. An improvement in discrimination was obtained by optimising the model fit to the CARDIIGAN data and by extension with four laboratory risk factors (c from 0.69 to 0.72).
In practice, the updated and extended model predicts a probability of CAD diagnosis of only 5% in a non-smoking female patient aged 40 years with diabetes (no chest pain, no hypertension and no dyslipidaemia), HDL cholesterol of 66 mg/dL, LDL cholesterol of 103 mg/dL, fibrinogen of 293 mg/dL and C reactive protein of 0.11 mg/dL. Conversely, it predicts a probability of obstructive CAD as high as 72% in a man of 75 years of age who smokes, has chest pain and hypertension (but neither diabetes nor dyslipidaemia), HDL cholesterol of 44 mg/dL, LDL cholesterol of 152 mg/dL, fibrinogen of 440 mg/dL and C reactive protein of 0.55 mg/dL. Clearly, this information based on readily available standard parameters can support clinical decision-making on performing angiography or other diagnostic evaluations in these individual patients.
A limitation of our study is that certain low-risk patients, who might fit the inclusion criteria, were selectively not included in our study because they were not referred to the clinic. This could compromise the results because of verification bias.19 Since our university hospital also has an important secondary healthcare function, the effect of the bias is expected to be rather small, but some patients might have visited other departments (such as radiology) and not have been referred because of a low-risk profile. On the other hand, we noted a wide spectrum of included patients and our study population was somewhat more heterogeneous than the development population. Apart from considering the model reproducibility, transportability ‘across samples from different but related source populations’ is important to consider.13 Other studies have also applied similar models in another context, like taking the cut-off value for stenosis at 70% and applying CT angiography (but without presenting coefficients)20 or focussing on patients with valvular heart disease,21 and the results barely differed. Another study was based on coronary CT angiography to capture the disease status and concluded that applying an invasive angiography-based probability model led to poor risk estimates.22 External validation research can highlight the amount of variation in performance to select patients for further diagnostic work-up.
The increase in performance by laboratory results needs further validation, specifically whether this improvement would also apply in a patient group more similar to that of the development model. Since risk factor evaluation and the considered laboratory parameters are already part of the routine patient evaluation, the diagnostic work-up could easily be enhanced by applying the presented model.
In the prediction of obstructive CAD diagnosis, the presence of typical chest pain symptoms was originally a very strong factor. In our cohort, it was much less pronounced. Presented as ORs, it concerns the difference between 7.4 and 1.7, respectively. Actually, Genders et al3 had classified the symptoms according to recommended criteria, that is, substernal chest pain or discomfort, provoked by exertion or emotional stress and relieved by rest or nitroglycerine. The importance of chest pain categorisation has been questioned recently.23 In clinical practice, chest pain or discomfort evaluation is commonly more subjective and may be strongly influenced by sex and age with older women presenting with more atypical symptoms.24 ,25 Accordingly, Genders et al3 developed their model in a group with more male patients (70% vs 62% in our study). Although the symptom criteria are also rather subjective in nature, they seem to advance the diagnostic process whereas our routine-based categorisation is simpler. The large difference in coefficients found may emphasise room for improvement in evaluating chest pain in daily clinical practice. Further research on this topic would help to unravel the issues involved.
Besides a possible verification bias, our study was limited by the fact that some data were missing, not uncommon for an observational study. Neglecting it by applying a complete-case analysis would potentially have led to less valid and less precise results, actually a waste of resources.26 Therefore, multiple imputation modelling was used, a method that is appreciated for being able to address the problem adequately. Furthermore, the patient group studied here differed in many respects from that of the original model. Compared with Genders et al,3 there were less CAD cases (44% vs 60%) and fewer men (62% vs 70%) in our CARDIIGAN cohort, but the age distribution and mean body mass index were similar, as well as the prevalence of diabetes (15% vs 18%). Chest pain (61% vs 53%), hypertension (76% vs 67%) and smoking (46% vs 36%) were more prevalent in our study (table 1). This may compromise to some extent the generalisability of our findings, even though the inclusion of consecutive patients referred for invasive evaluation of suspected CAD reflects daily practice. However, the development study was a consortium of 18 hospitals from all parts of Europe and North America, in which the wide range of settings did not pose a problem.3 For example, in that multicentre study, typical chest pain proportions varied from 1% to 64%, male sex from 42% to 77%, diabetes from 6% to 40% and smoking 19% to 63%, and mean age from 51 to 66 years. So the target population can be represented by a diverse spectrum of characteristics and our cohort adds some further insights. Another limitation here concerns the time period since the inclusion of the patients, since technologies have developed further and nowadays more laboratory results are routinely available. However, the effects of predictors may not have changed within a decade and diagnostic CAD validation studies are still scarce. Our study might be a stimulus to further elaborate on this subject.
The European Society of Cardiology (ESC) guidelines on the management of stable CAD recommend different non-invasive tests, especially in patients with an intermediate pretest probability, like CT-based CA, exercise stress test or stress imaging testing.27 Patients with low pretest probability should not be further evaluated and those with a high probability ought to be directly referred for invasive angiography. However, most of the patients with suspected CAD in daily clinical practice are within the wide range of intermediate risk, highlighting the need for better performance of prediction models before selecting one (or multiple) non-invasive modalities or even referring to invasive CA. Besides different patient characteristics, some of the differences in model performance between the development patient group and our cohort might stem from the patient recruitment strategies used. In the study by Genders et al3 there was a high percentage of CT angiography applied, indicating a preselection of patients. In clinical practice, physicians are faced by the whole range of pretest probabilities in their patients including all non-invasive diagnostic strategies. Our data comprise of such patients as they were unselectively referred for invasive CA by different kinds of medical doctors (general practitioners, general internists, cardiologists, etc), with the limitation of no information about exact frequencies, results and quality of prior non-invasive tests.
However, we had the possibility to analyse a large cohort with prospectively collected, good-quality data. Also, several routine laboratory measurements were available, permitting assessment of their additive value for obstructive CAD diagnostics.
Although a new extended model needs validation, one can assess clinical usefulness of the current model. Specificity increased 1.4% through application of the extra predictors, while sensitivity improved 3.4%. Therefore, the true-positive rate was raised indicating an increase in affected patients being classified properly, with at the same time an increase in potential number of angiographies saved. We also found a wide range of threshold probabilities in which a diagnostic prediction model would be advantageous. So when deciding on an angiography, differential rating of the procedure disadvantages will not easily tip the balance against applying a model. In the future further evaluation of the optimal decision threshold to classify patients is necessary, since this allows assessing to what extent better decisions are made with a model than without one or with a comparison between models.16 More elaborate research differentiating obstructive CAD by its severity can surely be helpful to aid the decision-making process. Also, research should further be encouraged to advance knowledge about effects of preinvasive selections on patient management strategies (PCI, coronary artery bypass grafting or optimal medical therapy alone),28 to avoid unnecessary conventional CA.
In conclusion, this study externally validated a suggested CAD prediction model which in its original form might not automatically be transportable to other clinical settings. Additional laboratory predictors and updating led to a model that can be useful to predict an obstructive CAD diagnosis, next to a thorough clinical evaluation; both are warranted as a pre-requisite for better (ie, faster, less expensive, lower risk) diagnostic strategies.
References
Footnotes
Contributors MW, JD and HFA contributed to the conception, design and data collection of the cohort. ME, HU, HFA and EWS were responsible for the interpretation of the results. ME, HFA and EWS participated in the drafting of the manuscript. MW, JD, HU and EWS revised the work critically for important intellectual content. ME analysed the data and accepts full responsibility for the work and had full access to the data. ME affirms that the manuscript is an honest, accurate and transparent account of the study being reported.
Funding This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Ethics approval The Ethics Committee of the Medical University Innsbruck has given its approval (UN3266 and AN3266).
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement No additional data are available.