Article Text

Original research
Development and validation of a prediction rule for patients suspected of acute coronary syndrome in primary care: a cross-sectional study
  1. Loes T C M Wouters1,
  2. Dorien L M Zwart1,
  3. Daphne C A Erkelens1,
  4. Elisabeth J M Adriaansen1,
  5. Hester M den Ruijter2,
  6. Esther De Groot3,
  7. Roger A M J Damoiseaux1,
  8. Arno W Hoes1,
  9. Maarten van Smeden4,
  10. Frans H Rutten1
  1. 1Department of General Practice, Julius Center for Health Sciences and Primary Care, Utrecht, The Netherlands
  2. 2Department of Experimental Cardiology, Utrecht University, Utrecht, The Netherlands
  3. 3Julius Center for Health Sciences and Primary Care, Utrecht University, Utrecht, The Netherlands
  4. 4Department of Methodology, Julius Center for Health Sciences and Primary Care, Utrecht, The Netherlands
  1. Correspondence to Dr Loes T C M Wouters; L.T.C.Wouters-2{at}


Objective To develop and validate a symptom-based prediction rule for early recognition of acute coronary syndrome (ACS) in patients with acute chest discomfort who call out-of-hours services for primary care (OHS-PC).

Design Cross-sectional study. A diagnostic prediction rule was developed with multivariable regression analyses. All models were validated with internal-external cross validation within seven OHS-PC locations. Both age and sex were analysed as statistical interaction terms, applying for age non-linear effects.

Setting Seven OHS-PC in the Netherlands.

Participants 2192 patients who called OHS-PC for acute chest discomfort (pain, pressure, tightness or discomfort) between 2014 and 2017. Backed up recordings of telephone triage conversations were analysed.

Primary and secondary outcomes measures Diagnosis of ACS retrieved from the patient’s medical records in general practice, including hospital specialists discharge letters. Performance of the prediction rules was calculated with the c-statistic and the final model was chosen based on net benefit analyses.

Results Among the 2192 patients who called the OHS-PC with acute chest discomfort, 8.3% females and 15.3% males had an ACS. The final diagnostic model included seven predictors (sex, age, acute onset of chest pain lasting less than 12 hours, a pressing/heavy character of the pain, radiation of the pain, sweating and calling at night). It had an adjusted c-statistic of 0.77 (95% CI 0.74 to 0.79) with good calibration.

Conclusion The final prediction model for ACS has good discrimination and calibration and shows promise for replacing the existing telephone triage rules for patients with acute chest discomfort in general practice and OHS-PC.

Trial registration number NTR7331.

  • primary care
  • myocardial infarction
  • telemedicine

Data availability statement

Data are available upon reasonable request. Data can be made available for researchers whose proposed use of the data has been approved at request of the corresponding author, with a signed data access agreement.

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See:

Statistics from

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Strengths and limitations of this study

  • We could analyse the original and very first telephone conversation of patients with acute chest discomfort.

  • The developed prediction model can be well generalised to other out-of-hours services for primary care (OHS-PC) locations in the Netherlands, but also to similar OHS-PC settings in other countries or even emergency medical service settings.

  • A limitation is that a full external validation of the model in another OHS-PC was impossible because no other cohort similar to ours was available.

  • Another limitation is that the effects of the predictors were assumed to be similar for males and females while this is not exactly the case, but by incorporating a differential non-linear effect of age and interaction with sex in the analyses this effect is minimalised.


Chest discomfort is among the top five reasons for telephone contact in out-of-hours services for primary care (OHS-PC) and concerns 5% of all cases at the emergency department (ED) in the USA.1 In the Netherlands, around 80% of patients with chest discomfort first call the general practitioner (GP) or OHS-PC, while 20% directly calls the emergency medical service (EMS, or ambulance dispatch centre) or are self-referrals to the ED.2 Adequate triage and early diagnosis in these patients is vital, because in case of an underlying acute coronary syndrome (ACS) early effective therapeutic interventions (‘time is muscle’) improve the patient’s outcome and prognosis.3 For the diagnosis of ACS, a 12-lead ECG and troponin testing is needed.3 However, before the patient is referred to an ED where these diagnostic tests can be done, patient selection is necessary based on symptom presentation retrieved by telephone triage.4 5 Symptom-based differentiation of ACS from other causes of chest discomfort is notoriously difficult.6 Symptom-based prediction rules for diagnosing ACS in general practice and other prehospital settings are—although highly needed—scarce.7–9 The efficiency and safety of telephone triage in OHS-PC was poor in a population with a prior chance of ACS of 8.3% in females and 15.3% in males; almost 50% of the males and females with chest discomfort received a high priority ambulance, while 11% diagnosed with an ACS did not receive a high urgency (ie, was seen within 1 hour).10

Most prediction rules for diagnosing ACS were developed in the ED setting and include results from ECG and troponin testing.11 Such prediction rules cannot be straightforward implemented for telephone triage in general practice because (1) in the latter setting these diagnostic tests are not available, (2) the prior chance of ACS is rather low and (3) on average disease severity is less than in those seen in the ED.12 13 The prevalence of ACS among patients with chest discomfort who call OHS-PC or EMS is about 10%–15%, and among those seen at the ED between 10% and 30%.10 11 13 14 Only one prediction rule was developed in primary care for diagnosing ACS; the modified Grijseels prediction rule, which had moderate discriminative ability (c-statistic of 0.66) after external validation.7 8 Five other primary care prediction rules were developed to predict CAD; for example, the Marburg Heart Score (MHS) and INTERCHEST prediction rule (International Working Group on Chest Pain in Primary Care).9 In these studies both patients with acute and non-acute chest discomfort were included and the prevalence of stable CAD showed to be 10.9%–12.6%, while that of ACS was only 1.5%–2.5%.9 Thus, these prediction rules have limited applicability for specifically diagnosing ACS.

In most OHS-PC and about half of the EMS in the Netherlands, triage nurses use the semi-automatic ‘Netherlands Triage Standard’ (NTS) as a decision support tool to classify the urgency of the patient’s condition. Triage nurses have to choose one out of 56 ‘main complaints’ and based on answers linked to the triage criteria, the NTS automatically proposes one out of six levels of urgency, that is, a certain time frame in which patients should be seen (U0-U5, online supplemental appendix table 1). The NTS is a modified and shortened version of the Manchester Triage Standard which was developed in the ED setting.15 Although the NTS was explicitly developed for telephone triage, it has not yet been validated against clinical outcomes even though it is already implemented on large scale. Recent research showed that the NTS had a poor sensitivity of 0.73 (95% CI 0.68 to 0.78) for telephone triage of patients with acute chest discomfort. The NTS recommended a low urgency (U3, U4 or U5) to 27% of patients who eventually showed to have an ACS or other life threatening event (LTE).10 The NTS’ specificity was also poor with 0.43 (95% CI 0.40 to 0.45); the NTS recommended a high urgency to 57% of the patients who eventually showed not to have an ACS or LTE. Given this poor safety and efficiency of the NTS, there is an urgent need for a better prediction model for patients with acute chest discomfort calling OHS-PC. In addition, there is a need for exploring sex-specificity of such a prediction rule as there is an ongoing debate on whether females differ from males in reporting symptoms of ACS.16 17

The aim of this study was to develop, and internal-external cross validate a symptom-based prediction rule for diagnosing ACS which is considerate to sex categories in male and female patients who call the OHS-PC for acute chest discomfort.


We performed a cross-sectional study among 2192 patients who called one out of seven participating OHS-PC in the Netherlands because of acute chest discomfort (pain, pressure, tightness or discomfort) during the period 2014–2017.18 These OHS-PC serve a total population of 1.5 million people and cover around 300 000 calls a year. We first selected calls based on the International Code for Primary Care (ICPC; a WHO world-wide code system for primary care) with ICPC-codes K01, K02, K03, K24, K74, K75, K76, K77, K93, L04, P74, R02, R98 and calls with keywords thoracic pain, chest pain, myocardial infarction, heart attack and their common abbreviations (figure 1). We included a broad variety of symptoms to capture the entire domain of patients that could be suspected of ACS. We listed all available calls of these patients and assigned random numbers with the Random Number Generator (RAND) function in Microsoft Excel to retrieve a random sample. Calls were excluded before relistening when the patients’ age was below 18 years or when the patient did not live in the surrounding area of the OHS-PC (in which case we could not retrieve the final diagnosis from the patient’s own GP). Calls were excluded during relistening when it did not concern a triage call (eg, intercollegial consultation) or when the recording was of poor quality (figure 1).

Figure 1

Flowchart of study population. ACS, acute coronary syndrome; ICPC, International Code for Primary Care; OHS-PC, out-of-hours services for primary care.

Candidate predictors

Research team members (LTCMW, DCAE) and medical students listened to the call recordings, blinded for the outcome, to collect data about symptoms, medical history and urgency allocation. Patient (age, sex) and call characteristics (call time, call duration) were collected from the OHS-PC electronic medical files of the patients. As candidate predictors we included age and sex, the NTS triage criteria (see online supplemental appendix table 2), the ACS predictors from the modified Grijseels prediction model (male sex, radiation, nausea, sweating and history of CAD), the ‘CAD predictors’ from MHS and INTERCHEST prediction models (age, pain feels like pressure, CAD history or cardiovascular (CV) risk factors, patient assumes cardiac origin of pain), and—based on a recent own study in OHS-PC—the predictor ‘calling at night between 00:00 and 09:00’.7–9 19


The primary outcome was the diagnosis ACS. The final diagnoses were retrieved from the patient’s GP and based on the GP’s electronic medical files which include ED and cardiologist discharge letters and notes from the OHS-PC contact. The diagnosis ACS was nearly always made by a cardiologist (96.0%) and included information on levels of (high-sensitivity) troponin and electrocardiographic results. We used medical information up to 30 days following the contact with the OHS-PC to allow us to include diagnoses of ACS that were initially missed because the patient was not referred to the cardiologist the same day of the OHS-PC contact. However, in none of the patients in the study we had evidence of a missed diagnosis of ACS.

Sample size calculation

We relied on the minimal sample size criteria for prediction model development proposed by Riley et al using the ‘pmsampsize’ package in R.20 Based on an ACS prevalence of overall 11% and a Cox-Snell R2 of 0.075 (a conservative value based on a model with age and sex) and a total number of 2192 observations we were allowed to assess 19 candidate predictors.21 Based on sample size calculations we concluded that development of separate models for males and females would require a significantly larger sample, therefore we analysed sex as a statistical interaction term.20

Statistical analyses

We developed three diagnostic models using multivariable logistic regression analysis. First, we developed a base model with only age and sex as predictors, where age was modelled using a restricted cubic spline function (four knots) and an interaction with sex. This resulted in a base model with seven predictor parameters (excluding the intercept). Second, we fitted a full model with an additional 12 preselected binary predictors (having chest pain, acute chest pain shorter than 12 hours, shortness of breath, sweating, retrosternal located chest pain, radiation, pressing heavy pain, stabbing pain, history of cardiovascular disease (CVD), history of CAD, someone else calling, calling at night). Third, we applied backward elimination, with a cut-off p value <0.20 for including predictors (a higher cut-off value to lower the chances of overfitting).21

We applied internal-external cross validation (IECV) for model validation using the seven different OHS-PC locations (online supplemental appendix table 3 for patient characteristics of different OHS-PC locations).22 We evaluated the IECV performance in terms of the area under the receiver operating characteristic curve (c-statistic), the calibration slope and calibration in the large. The IECV estimates of performance were combined by using random-effect model (DerSimonian-Laird estimator). Based on the IECV we also constructed flexible calibration curves and decision curves. In the decision curve analyses we compared the final model with the currently used NTS triage model in OHS-PC in the Netherlands.23 Finally, we created an illustrative table of diagnostic test accuracy for various model-based risk thresholds of the final model, following the example in Wynants et al.23 IECV estimates for risk threshold specific sensitivity and specificity, and we applied a bivariate model commonly used for diagnostic test accuracy meta-analysis.24

Missing data

For missing data we carried out multiple imputation using the Multivariate Imputation via Chained Equation package in R, with 30 imputation rounds and 30 iterations.25 We pooled the results following Rubin’s rules.26 Predictors with over 50% missing were excluded from consideration in the models (online supplemental appendix table 4 for details about the missing data). Characteristics were compared between patients with and without information on the medical outcome—because some GPs refused diagnosis retrieval from their files—to allow for assessment of differences in characteristics between these patient groups (online supplemental appendix table 5). There were no clinically meaningful differences in symptoms and patient or call characteristics between the 2192 patients with information on the outcome, and the 1012 patients about whom knowledge of the medical outcome related to the OHS-PC contact because of acute chest discomfort was missing.

All analyses were done in R V.4.0.3. (10 October 2020) with the Regression Modelling Strategies (‘rms’) package in R.27 We reported our study in accordance to the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis criteria (online supplemental file).28

Patient and public involvement

No patients were involved in defining the research question or the outcome measures. Neither they were involved in developing plans for design. However, they participated in the discussion on implications and the implementation strategy. In addition, they were asked to advise on interpretation and writing up of the results. Results will be shared and discussed in more detail with representatives of the Dutch national patient community of cardiovascular diseases (‘Harteraad’).


Among the 2192 callers with acute chest discomfort (mean age 59.1 (SD 19.5) years and 55.3% females) 251 (11.5%) had a final diagnosis of ACS; 101 (8.3%) females and 150 (15.3%) males (table 1). Patients with ACS were older than those without (mean age 69.7 (SD 13.4) vs 57.7 (SD 19.8) years) and females with ACS were on average older than men with ACS (73.8 (SD 13.5) years vs 67.0 (SD 12.6) years).

Table 1

Characteristics of 2192 patients who called OHS-PC with acute chest discomfort between 2014 and 2017, divided between females and males with and without ACS

Over two-thirds of all callers (68.3%) received a high urgency allocation (seen within 1 hour; U1 or U2) and among the 251 patients who showed to have an ACS, 88.4% received a high urgency allocation. Calls of patients who had an ACS were shorter than calls in those without ACS (median call duration 6:34 (SD 3:38) vs 7:42 (SD 3:48) min).

Medical history and symptoms

Females and males with ACS had more often a history of CVD or CV risk factors than those without (females with ACS 81.4% vs females without ACS 61.1%, males with ACS 78.5% vs males without ACS 64.4%) (table 1). The majority of both females and males had chest pain (94.5%) and this was similar among those with and without ACS. Overall, presented symptoms among males and females calling the OHS-PC for chest discomfort were quite similar. Symptoms associated with ACS in both sexes were pressing/heavy chest pain (females with ACS 81.7% vs females without ACS 62.5%, males 81.2% vs 57.7%, respectively), retrosternal located chest pain (females 54.5% vs 40.0%, males 53.1% vs 38.7%), radiation of pain (females 86.0% vs 67.8%, males 65.4% vs 56.2%) and sweating (females 52.9% vs 42.0%, males 51.9% vs 35.8%).


In total 251 patients were diagnosed with ACS and 65 with other LTEs, and of clinical relevance is that both critical events occurred significantly more in males than females (15.3% vs 8.3%, p<0.001 for ACS, and 3.8% vs 2.3%, p=0.04 for other LTEs, respectively). Of the 101 females with ACS, 22.8% had an ST-segment elevation myocardial infarction (STEMI), 46.5% a non-ST-segment elevation myocardial infarction (NSTEMI), 19.8% unstable angina pectoris (UAP) and 10.9% non-classified ACS. In 150 males with ACS, 33.3% had an STEMI, 36.7% an NSTEMI, 26.0% UAP and 4.0% non-classified ACS (table 2).

Table 2

Diagnoses of 2192 males and females who contacted the OHS-PC for acute chest discomfort between 2014 and 2017, by sex

Twenty-eight (2.3%) females and 37 males (3.8%) had another life-threatening event than ACS (eg, pulmonary embolism, thoracic aortic dissection, acute abdominal aneurysm). All other patients (85.6%) had non-urgent medical conditions such as non-urgent cardiovascular disease (18.9%), musculoskeletal pain (17.9%), non-cardiac chest pain (not further specified) (16.9%), psychogenic disorder (11.4%), gastrointestinal disorders (7.2%), respiratory disorders (5.3%) and other non-urgent diagnoses (8.0%).

Model development, performance and validation

The base model with sex and age had an apparent c-statistic of 0.72 (95% CI 0.70 to 0.75), and an internal-external validation based c-statistic of 0.72 (95% CI 0.68 to 0.75) (online supplemental appendix table 6). The basic model shows that the risk of ACS increases with age for both sexes, with a notable peak risk for men at an age near 60 years and a more gradual increase in risk of ACS for women (figure 2). The full model including all candidate predictors had an apparent c-statistic of 0.79 (95% CI 0.76 to 0.81) and an internal-external validation based c-statistic of 0.77 (95% CI 0.74 to 0.80) (online supplemental appendix table 7). The full model had optimal calibration (flexible line close to the 45 degree reference line) up to a predicted probability of ACS of approximately 0.2 (online supplemental appendix figure 1). Risks higher than 0.2 tended to be overestimated by the model, however, since any plausible risk threshold will be lower than 0.2 in the primary care setting, we find the calibration in the relevant range to be satisfactory. After backward elimination, the backward model had an apparent c-statistic of 0.79 (95% CI 0.76 to 0.81), and the internal-external validation c-statistic was 0.77 (95% CI 0.74 to 0.79) (Table 3) It had very similar calibration to the full model (Table 3, figure 3).

Table 3

Final model for predicting the diagnosis acute coronary syndrome

Figure 2

Base model with age and sex for predicting diagnosis acute coronary syndrome. ACS, acute coronary syndrome.

Figure 3

Callibration of the final model with internal external validation.

Decision curve analyses and risk thresholds

Both the full and backward model showed a high net benefit as compared with the currently used NTS model for telephone triage in OHS-PC (figure 4). There was no difference in net benefit between the full model and backward model across plausible risk thresholds. Based on this analysis we decided to choose the backward as the final triage tool model because; (1) with fewer predictors the prediction of ACS remained similar accurate and (2) no valuable time is lost during telephone triage by asking the patient about symptoms that do not contribute to a better prediction. The final model included besides age and sex, the five following predictors; (i) acute onset of chest pain lasting <12 hours, (ii) a pressing/heavy character, (iii) radiation of pain, (iv) sweating, (v) calling at night between 00.00 and 09.00. Finally, we evaluated the diagnostic performance of the final prediction model across risk thresholds that may be chosen to apply in clinical practice (table 4, figure 5).

Figure 4

Decision curve analyses comparing the full and final models versus the currently used model and versus treat all patients. NTS, Netherlands Triage Standard.

Table 4

Diagnostic accuracy for a range of risk thresholds of the final model

Figure 5

Runway plot of diagnostic accuracy measures of the final model.


This is the first study that developed and internal-external validated a symptom-based prediction rule for telephone triage of ACS in male and female patients who contact OHS-PC for acute chest discomfort (pain, pressure, tightness or discomfort). ACS was present in 8.3% of the females and 15.3% of the males. The prediction rule is applicable for triage in the OHS-PC setting and consists of sex and age as statistical interaction terms, and five other symptom-based predictors. It had a good discriminative ability (adjusted c-statistic 0.77 (95% CI 0.74 to 0.79)) and was well calibrated up to an ACS risk of 0.2.

Strengths and limitations

The major strength of this study is that we analysed the original and very first conversations of patients with acute chest discomfort with primary healthcare providers and assessed these talks without knowledge of the diagnosis; the assessment of symptoms was therefore not affected by hindsight bias caused by knowledge of the final diagnosis.29 Furthermore, we could analyse a large sample (N=2192) of patients which allowed us to evaluate up to nineteen candidate predictors. We assessed the risk of selection due to missing outcome data, and our data suggest that this missingness was unlikely to bias our findings. Because we used data from seven different OHS-PC our results will be well generalisable to other OHS-PC in the Netherlands, but we anticipate the model might be applicable to similar OHS-PC setting in, for example, the UK and Scandinavian countries. Our results may also be generalisable to some EMS settings, because the prior chance of having an ACS among those calling for acute chest discomfort is rather similar in the EMS setting as in the OHS-PC setting.14 30

A limitation of the study is that a full external validation of the final model is not done yet. However, at the time of executing the study hardly any primary care research data were available to perform such validation. Importantly, the internal-external validation showed very good calibration up to an ACS risk of 0.2. Although we performed extensive internal-external validation making use of the datasets of nine sites with substantial differences in case mix, we will strive for formal external validation before it can be widely applied in everyday primary care practice. Another limitation is that the effects of the predictors were assumed to be similar for male and female patients while that might not be optimal for the predictions. However, development of separate models for males and females would require a significantly larger sample size than was available. Importantly, a differential non-linear effect of age was incorporated using a spline function and interaction with sex was incorporated, and the final internal-external validated model did have good overall performance.

Comparison with other studies

Our prediction model had a higher discriminating ability for ACS than the NTS (c-statistic of 0.58) and modified Grijseels prediction rule (c-statistic of 0.66).7 This may largely be explained by the addition of age, the strongest predictor of ACS. This is in line with the notion that the prevalence of ACS increases with age.7 9 31 Importantly, in our study among people aged below 40, only one (0.4%) male patient had an ACS (UAP). For males to the age of 55 we found a peak risk of ACS of around 20% and remaining at this level with further age increase onwards. For females we found a gradual increase of risk with age with a maximum ACS risk of around 18% for those aged over 80 years. Similar to the modified Grijseels prediction rule our prediction model includes sweating and radiation of pain, however, the modified Grijseels rule combined nausea and sweating to a single predictor (ie, nausea or sweating).7 8 Age and sex were predictors in our model, but also in the MHS and INTERCHEST prediction models.32 33 Also the INTERCHEST rule included pressing heavy chest pain as predictor.32 A new predictor is calling at night (between 00.00 and 09.00).19 Previous studies in the ED setting also showed circadian variability with an early morning peak for ACS patients.34 Finally, symptoms associated with ACS were rather similar between females and males, which is in line with recent sex-stratified studies of patients with chest discomfort who called the OHS-PC, but is in contrast with the prevailing opinion.16 17 35

We performed decision curve analyses to investigate what would be the optimal threshold of ACS predicted probability to initiate treatment, where ‘treatment’ in prehospital setting refers to urgent referral for hospital admission (U1, ambulance within 15 min). However, there is a principal difference between diagnostic probability and categories of urgencies. Risk prediction provides a continuous value of the probability of disease, while urgency level categorisation is based on the interpretation of how risk probability can be translated in urgency, and time within a patient should be seen and treatment delivered. Context is needed to determine optimal thresholds, which concerns what percentage of missing ACS is considered as acceptable by healthcare providers, patients and policymakers. This percentage is expected to be very low because a missed ACS can result in permanent cardiac impairment, heart failure, life-threatening arrhythmias in the early phase and death.36 Furthermore missing an ACS is the most common reason for malpractice claims worldwide.37 A survey performed among 1029 ED doctors in the USA, New Zealand and Australia showed that they considered an average missing rate between 0.1% and 1% (range 0%–10%) as acceptable.38 When we apply a maximum of 1% missing with our prediction rules, the threshold has to be set at a predicted risk of ACS of 0.05 (negative predictive value of 0.99, table 4), which means based on our data that the majority of patients needs urgent referral. This would result in over-crowded EMS and EDs, and with the available resources being limited, this may result in exceeding target triage times, which could compromise patient safety in another way.39 A possible alternative to consider may be applying different ‘treatments’ per thresholds, that is, dispatching an ambulance (U1) for the high predicted risk patients, and GP visit within 1 hour (U2) for the low predicted risk patients. During GP visit more clinical parameters (blood pressure, heart rate, overall clinical impression) can be gathered to improve ACS risk prediction, and in the future, there might be room for applying point-of-care high-sensitivity troponin testing, as these are nowadays only available in the ED setting.40 In order to determine the ideal threshold, external validation will be needed combined with clinical and management considerations. The development of this diagnostic model is the necessary first step towards an implementation study in which this model is adapted to urgency levels that can be applied by triage nurses during telephone triage at the OHS-PC. The diagnostic model needs to be ‘translated’ in simple yes/no questions that can be incorporated in the existing NTS and a personalised risk prediction for age and gender is generated. Some older questions will then be substituted. We are aiming to do so in an implementation study applying action research.

Implications for clinical practice and future research

This symptom-based prediction model for ACS has good discrimination and calibration and could readily be applied for telephone triage of patients with acute chest discomfort in primary care, notably the OHS-PC setting. The results of the decision curve analysis showed a large net benefit over a range of plausible risk threshold as compared with the currently used NTS model in the OHS-PC in the Netherlands. For future research, full external validation in other OHS-PC or EMS populations could further optimise and update the model. Furthermore, sex-specific prediction models could be developed for ACS, but given the overlap in symptoms between men and women, this would not result in major changes in predictors.


The final prediction model for ACS has good discrimination and calibration and shows promise for replacing the existing telephone triage rules for patients with acute chest discomfort in general practice and OHS-PC. However, future research with an external validation is needed to provide insights into how the prediction model can be applied in practice.

Data availability statement

Data are available upon reasonable request. Data can be made available for researchers whose proposed use of the data has been approved at request of the corresponding author, with a signed data access agreement.

Ethics statements

Patient consent for publication

Ethics approval

This study involves human participants and was approved by the Medical Ethics Committee Utrecht, the Netherlands (reference number WAG/mb/16/003208) and complied with the Declaration of Helsinki. A waiver of informed consent was given because our study had minimal risk to subjects and could otherwise not be carried out logistically. Personal and research data were handled and stored according to the European General Data Protection Regulation.


The authors would like to thank the OHS-PC foundation ‘Primair Huisartsenposten’ and all employees of the participating locations for or their cooperation in this study, notably for providing data and technical support.


Supplementary materials

  • Supplementary Data

    This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.


  • Contributors FHR and DLMZ are the lead investigators who conceived the research idea and methodology. Funding acquisition was done by FHR, DLMZ and RAMJD. LTCMW and DCAE conducted data acquisition. LTCMW performed the analyses and wrote the first draft of the manuscript. She was supervised by FHR and DLMZ, who critically revised the manuscript, and by MvS who was involved with the analyses. DCAE, EdG, EJMA, HMdR, AWH and RAMJD contributed to and approved of the final version of the manuscript. The guarantor (LTCMW) accepts full responsibility for the finished work and/or the conduct of the study, had access to the data, and controlled the decision to publish.

  • Funding This study was funded by an unrestricted grant from (1) the Department of General Practice of the University Medical Centre Utrecht, (2) a personal promotion grant of D L Zwart, MD, PhD, (3) the foundation ’The Netherlands Triage Standard’ and (4) the foundation ‘Stoffels-Hornstra’. This study contributes to the 2020B004 IMPRESS study which is funded by the Dutch Heart Foundation/Dutch Cardiovascular Alliance (DHF/DCVA).

  • Competing interests None declared.

  • Patient and public involvement Patients and/or the public were involved in the design, or conduct, or reporting, or dissemination plans of this research. Refer to the Methods section for further details.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.