Objectives Following appropriate validation, clinical prediction rules (CPRs) should undergo impact analysis to evaluate their effect on patient care. The aim of this systematic review is to narratively review and critically appraise CPR impact analysis studies relevant to primary care.
Setting Primary care.
Participants Adults and children.
Intervention Studies that implemented the CPR compared to usual care were included.
Study design Randomised controlled trial (RCT), controlled before–after, and interrupted time series.
Primary outcome Physician behaviour and/or patient outcomes.
Results A total of 18 studies, incorporating 14 unique CPRs, were included. The main study design was RCT (n=13). Overall, 10 studies reported an improvement in primary outcome with CPR implementation. Of 6 musculoskeletal studies, 5 were effective in altering targeted physician behaviour in ordering imaging for patients presenting with ankle, knee and neck musculoskeletal injuries. Of 6 cardiovascular studies, 4 implemented cardiovascular risk scores, and 3 reported no impact on physician behaviour outcomes, such as prescribing and referral, or patient outcomes, such as reduction in serum lipid levels. 2 studies examined CPRs in decision-making for patients presenting with chest pain and reduced inappropriate admissions. Of 5 respiratory studies, 2 were effective in reducing antibiotic prescribing for sore throat following CPR implementation. Overall, study methodological quality was often unclear due to incomplete reporting.
Conclusions Despite increasing interest in developing and validating CPRs relevant to primary care, relatively few have gone through impact analysis. To date, research has focused on a small number of CPRs across few clinical domains only.
- risk prediction
- clinical prediction rule
- impact analysis
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.
Strengths and limitations of this study
Clinical prediction rules (CPRs) are increasingly developed and advocated for use in clinical practice. However, little is known regarding the effectiveness of these tools versus usual care for relevant clinical outcomes.
This is the first systematic review of CPRs relevant to primary care that have gone through impact analysis.
The main limitation of this review is that the electronic search was limited to 30 prespecified journals, which may mean that some relevant studies were not retrieved. This search was supplemented with key author searches and reviewing other resources known to publish CPRs.
Nevertheless, this is the first study to examine, in detail, impact analysis studies of CPRs relevant to primary care.
Clinical prediction rules (CPRs) are clinical tools that quantify the individual contributions that various components of the history, physical examination and investigations contribute towards diagnosis, prognosis or likely response to treatment in a patient.1 These tools simplify, standardise and attempt to increase the accuracy and consistency of clinicians’ diagnostic and prognostic assessments and management decisions.1 ,2 Well-recognised examples of CPRs include the Framingham cardiovascular risk score, the Ottawa ankle rule and the Centor score for sore throat.
Developing and validating a CPR requires reference to specific methodological standards.1 ,3 ,4 Conventionally, these tools go through three distinct stages prior to full implementation in a clinical setting.1 ,3 ,4 The first stage is derivation, where the independent and combined effects of explanatory variables such as symptoms, signs and/or investigations, are established. The next stage is validation, where the final derived CPR is evaluated first in a similar clinical setting (internal validation), followed by different clinical settings (external validation). If in following these stages predictive accuracy is established, then the final stage of evaluation is to test the impact of using the CPR in clinical practice, ideally in a randomised controlled trial (RCT) for relevant clinical outcomes.1 ,5 ,6 Impact analysis aims to investigate if the implementation of a CPR in clinical practice is better than usual care for the patient, process of care and/or cost outcomes.7 ,8
Our research group recently published two studies detailing the development and content of an international register of CPRs relevant to primary care.2 ,9 With increasing interest in CPRs, large numbers have been derived but fewer have been validated or tested in an impact analysis study.2 If CPRs are to truly improve the quality of patient care then evaluation of these tools is crucial.
The aim of this systematic review is to present a narrative and critical analysis of CPRs relevant to primary care which have gone through impact analysis.
The methods used for identifying CPRs from the literature and in developing a register of these tools relevant to primary care have been published in detail previously.2 ,9 These methods are summarised below.
An electronic search string for PubMed was developed to retrieve CPRs relevant to primary care from 30 preselected medical journals (see online supplementary appendix 1 for search string and journals included).9 No restriction was placed on language. Original electronic searches were conducted from 1980 to 2009, which for the purposes of this review were updated to the end of 2013.2 In addition, secondary sources of CPRs were searched including the Journal of the American Medical Association (JAMA) Rational Clinical Examination series, a handbook of CPRs and personal resources.2 ,9 ,10 Author searches for key experts in the field were also conducted for additional relevant articles. Furthermore, reference lists of each relevant impact analysis study were searched to identify possible additional studies.
Studies were eligible for inclusion if they met the following criteria.
(1) Population: Relevant to primary care defined as ‘normally the point of first medical contact within the healthcare system, providing open and unlimited access to its users, dealing with all health problems regardless of the age, sex or any characteristic of the person concerned’.11 Although studies may not have been conducted in a primary care setting, they were eligible for inclusion providing they were relevant to primary care (eg, implementation of the Canadian head CT rule with the aim of reducing imaging for patient presenting with minor head injury). This inclusion criterion was designed to be broad to acknowledge variation in the same-day diagnostic tests that are available across different countries and the international variation in the role of primary care clinicians. Studies set in the emergency department were considered relevant to primary care if following application of the CPR the patient could be discharged home following application of the CPR.
(2) Intervention: CPR defined as ‘a clinical tool that quantifies the individual contributions that various components of the history, physical examination and investigations make towards the diagnosis, prognosis or likely response to treatment in a patient’.1 Diagnostic, prognostic and management CPRs were included and screening questionnaires (ie, applied to apparently healthy people who may be at increased risk of a disease or condition) were excluded. A requirement for inclusion was that the CPR comprised the entire intervention. Studies where the CPR was implemented as part of a broader guideline, protocol or decision aid were excluded. Studies that used a CPR to determine eligibility for trial inclusion but were not part of the intervention were also excluded.
The following study designs were included: (cluster) RCT, controlled before–after or interrupted time series studies. Uncontrolled study designs were excluded as the aim of this review was to examine the effectiveness of CPR implementation, rather than the performance of the CPR which would be captured in validation studies using observational study designs.
(3) Comparison: Usual care.
(4) Primary Outcome: Physician behaviour, for example, ordering of diagnostic tests; process of care, for example, number of inpatient bed days; and/or patient outcomes, for example, duration of symptoms.1
All articles were initially screened for inclusion according to title and abstract by one reviewer. Potentially relevant articles were then reviewed by a second reviewer with any disagreements resolved by a third independent reviewer. For each relevant article, the following data was extracted: (1) name of CPR; (2) type of CPR: prediction rule, decision rule or both; (3) clinical domain: using International Classification of Primary Care—second edition (ICPC-2)12; (4) clinical setting; (5) study population; (6) primary outcome of interest; (7) predictive accuracy of the CPR (measured by sensitivity/specificity (95% CIs) where reported, otherwise, the model's c-statistic was recorded) and (8) impact on primary outcome of interest.
Critical analysis of CPR impact analysis
Each article was critically appraised, using a published framework for impact analysis of CPRs.7 Developed in 2011 by an expert panel, this four-phase framework provides guidance for impact analysis studies (see figure 1). The phases are as follows: (1) Exploratory phase; evaluate the level of evidence and predictive accuracy of the CPR; (2) Preparation for impact analysis; consider potential barriers, assess acceptability of the CPR to clinicians and local stakeholders and conduct a pilot study; (3) Experimental phase; evaluation of the CPR with monitoring of the use of the CPR in a clinical setting; (4) Long-term implementation phase; examine if a CPR with reported positive impact on relevant clinical outcomes is implemented long-term and how this was achieved.7
Summary of effect on process and outcome of care
Meta-analysis was not possible due to heterogeneity in CPRs and outcomes of interest, so a narrative analysis was conducted. In this section, wherever appropriate and where data was available, crude ORs and absolute risk reductions (ARR) were calculated.
Methodological quality assessment
The methodological quality of each impact analysis study was independently evaluated by two reviewers (MU and BC), and by a third reviewer if consensus was not reached (EW). For each study design, an appropriate quality assessment checklist was used. RCTs and cluster RCTs were assessed through the Cochrane risk of bias tool.13 Controlled before–after studies, and interrupted time series studies, were evaluated through Cochrane criteria for these study designs.14
Overview of studies
A flow diagram of the search strategy is presented in figure 2. The PubMed search (1980–2013) and supplementary sources searches retrieved a total of 86 158 studies, of which 1111 CPR studies were identified following review of title and abstract. A total of 18 studies met the inclusion criteria for the systematic review.
Description of included impact analysis studies
A total of 14 unique CPRs were tested in 18 impact analysis studies (table 1 and online supplementary appendix 2). According to ICPC-2, these studies were classified into four broad clinical domains, namely, musculoskeletal, most commonly the Ottawa ankle rule15–20 (n=6); cardiovascular21–27 (n=6); respiratory28–32 (n=5); and neurological33 (n=1). The majority of studies were conducted in North America (Canada n=10, the USA n=4), with the remainder in the UK (n=2) and France (n=2). Most studies were set in the emergency department (ER) (n=9) and primary care (n=7). The remainder were carried out in the outpatient department (n=2).
Regarding study design, there were four cluster RCTs,15 ,18 ,27 ,33 eight RCTs,22 ,24–26 ,28–32 one pilot RCT,23 three controlled before–after studies,16 ,17 ,19 and two interrupted time series.20 ,21 In a total of 16 studies, the intervention was the impact of the CPR alone,15–29 ,32 ,33 and two studies used different trial arms to test the CPR alone versus CPR and protocol versus usual care.30 ,31 Two studies integrated the CPR into a computerised clinical decision support system (CDSS).20 ,32 Two studies used real-time CPR reminders at the point of test ordering.18 ,33
Critical analysis of CPR impact analysis studies
Preparation for impact analysis: level of evidence of CPR, consideration of potential barriers and assessment of CPR acceptability
Fifteen of 18 studies implemented a CPR that was externally validated, while three studies tested a CPR that had been derived or internally validated only.21 ,30 ,31 Ten studies reported the CPR's sensitivity from validation studies in identifying the target outcome which ranged from 85% to 100%.15–20 ,28 ,29 ,31 ,33 Five studies identified and addressed potential barriers for implementation before impact analysis, most frequently through barriers analysis.18 ,20 ,22 ,32 ,33 Six studies assessed the acceptability of the CPR to clinicians prior to the implementation phase of the study, usually through training sessions and engagement with local stakeholders.15 ,19 ,22 ,27 ,32 ,33 Seven studies reported that a pilot or simulation phase was conducted or there was a previous impact analysis on the same CPR by the same authors.16–19 ,21 ,32 ,33
Impact analysis phase: adherence with CPR use and reasons for non-adherence
Twelve studies tracked the use of the CPR during implementation, usually with standardised data collection forms or computerised tools.15–18 ,20 ,22 ,25 ,26 ,29 ,31–34 Overall, adherence with CPR use varied between studies ranging from 57.5% to 100%, with reported reasons for non-adherence including fear of missing the diagnosis, preference for own clinical judgement and patient request.20 ,33 Of the 12 studies that tracked CPR adherence, seven reported adherence of ≥80%, four reported adherence of 60–80%, and one study reported adherence of ≤60%. Nine of 12 studies reported a positive impact on primary study outcome following CPR intervention, but there was no clear link between level of adherence and successful CPR implementation. Clinicians’ acceptability of CPR use during the intervention phase was evaluated in four studies,16 ,17 ,20 ,33 of which two assessed the reported rate of comfort using a five-point Likert scale.16 ,17
Postimplementation phase: maintaining use of CPR
Of 10 studies with a positive impact on primary outcome, 4 evaluated the effect of the CPR in a post-intervention phase ranging from 5 to 12 months.15 ,16 ,18 ,20 To maintain CPR use, two studies used a passive strategy of posters, one retained computerised clinical decision support, and one did not employ any particular strategy. In all four studies, the effect of CPR use was maintained.
Importance of study design in assessing the impact of CPRs
There were five uncontrolled before–after studies retrieved during the initial search which were excluded from data analysis based on their uncontrolled study design (see online supplementary appendix 3).35–39 These studies tested the impact of the Ottawa ankle rule (n=2), the Canadian C-spine rule (n=1), the CT head rule (n=1), and the Glasgow Blatchford bleeding score (n=1). All five studies demonstrated a positive impact on primary outcome, usually physician behaviour in ordering imaging.
Effect on the process and outcome of care
Overall, 10 studies reported that CPR implementation resulted in a positive impact on primary outcome, while eight studies reported no impact versus usual care. There were no clinically important adverse outcomes reported. Studies are presented according to clinical domain. Table 2 presents a summary of the estimated effect sizes for the impact analysis studies.
Musculoskeletal (cluster RCTs n=2, controlled before–after studies n=3, interrupted time series n=1)
All six musculoskeletal studies focused on the implementation of CPRs for deciding on further imaging for patients presenting with ankle, knee or neck injury.15–20 All the included musculoskeletal CPRs had reported sensitivities of 100% in their validation studies, and they all focused on physician behaviour in deciding to order imaging. Of these six studies, five reported a positive effect on reducing imaging with crude ORs ranging from 0.03 to 0.9615–18 ,20 (see table 2). All studies adopted an educational approach to encourage CPR use among clinicians through use of educational meetings, posters and pocket cards. Of note, three studies tested the impact of the Ottawa ankle rule; two controlled before–after trials in Canada, and one cluster RCT in France.15 ,16 ,19
Cardiovascular (cluster RCT n=1, RCTs n=4, interrupted time series n=1)
Of six cardiovascular studies, two implemented chest pain CPRs to assess the impact on physician decision-making regarding emergency admission for patients with suspected myocardial infarction.21 ,22 One of these studies reported a 30% relative reduction in patients admitted inappropriately.21 The remaining four studies implemented cardiovascular risk scores in general practice. Three of these studies reported no impact on physician behaviour such as prescribing and referral to dieticians, or on patient outcomes such as reduction in lipid levels.23 ,24 ,27 However, in one large-scale RCT (n=3053) that published its findings in two separate articles, both patient lipid levels and physician antihypertensive prescribing were improved.25 ,26
Respiratory (RCTs n=5)
Of five respiratory studies, four focused on physician behaviour in terms of antibiotic prescribing for sore throat in general practice.28–30 ,32 Of these four studies, only one reported significantly reduced antibiotic prescription rates in the intervention group (age-adjusted relative risk 0.74, 95% CIs 0.60 to 0.92) versus usual care.32 The primary outcome in the fifth study was reported symptom severity in patients presenting with sore throat, and antibiotic prescribing was included as a secondary outcome.31 This study found that use of the CPR alone or CPR in combination with a rapid antigen detection test improved patient-reported symptom severity and duration, and reduced antibiotic use by 29% (adjusted risk ratio 0.71, 95% CI 0.50 to 0.95).31
Neurological (cluster RCT n=1)
One study implemented the Canadian CT head rule, which guides the ordering of brain imaging in patients presenting following minor head injury.33 Despite this CPR having 100% sensitivity in validation studies, it did not reduce imaging rates. In process evaluation, clinicians’ reported unease with certain components of the rule and fear of missing a high-stakes diagnosis as reasons for not adopting the CPR.33
Methodological quality assessment of included studies
Studies were heterogeneous with regard to risk of bias. For the RCT designs (n=13), five studies were considered low risk of bias for random sequence generation, and five were considered low risk in relation to allocation concealment (figure 3A). The remaining RCT studies had an unclear risk in these domains. Owing to the nature of many of the interventions, it was not always possible to blind participants and research personnel, therefore, performance bias was judged to be unclear or high in over half these studies. In the non-randomised study designs, the risk of selection bias was high in all studies while the risk of blinding and contamination was low in all studies (see figure 3B). Overall, six studies tested the impact of a CPR in which the authors were involved in developing.17 ,18 ,22 ,29 ,31 ,33 The impact that this may have in terms of bias is unclear.
Summary of main findings
This review indicates that despite the increasing research interest in developing and validating CPRs, relatively few of these tools relevant to primary care have gone through impact analysis. Implementation has been restricted to a few clinical domains mainly musculoskeletal, cardiovascular and respiratory, and certain CPRs have undergone multiple evaluations, for example, the Ottawa ankle rule. Of 18 studies meeting inclusion criteria, 10 demonstrated an improvement in primary outcome with CPR use when compared to usual care. Approximately half these successful studies focused on changing physician behaviour in ordering imaging for patients presenting with ankle, knee and neck musculoskeletal injuries.15–18 ,20 Four studies with a positive impact on the study's primary outcome successfully implemented post-RCT measures to maintain the impact through both passive (posters) and active strategies (retention of computerised CDSS).15 ,16 ,18 ,20
Studies which aimed to alter physician behaviour regarding prescribing were less successful with three of six such studies successful in reducing prescription rates.23 ,31 ,32 Studies that reduced antibiotic prescription rates invested significant time before CPR implementation in assessing acceptability to clinicians, and also integrated the CPR into the clinical work flow through computerised clinical decision support or point-of-care reminders.31 ,32 The importance of this type of impact analysis preparation in adequately addressing barriers to implementation, and in integrating the CPR into the clinical workflow has been highlighted.5 ,7 ,40 In this review, 12 studies considered barriers to implementation and/or gauged the acceptability of the CPR to clinicians prior to impact analysis. However, only four studies integrated the CPR into clinical work flow using either computerised CDSS or point-of-care reminders.18 ,20 ,32 ,33
The perceived seriousness of the target condition may also affect implementation of CPR. For instance, the impact of the Canadian CT head rule was evaluated in the diagnostic pathway of intracranial bleeding following minor head injury.33 This CPR has 100% sensitivity, and though implemented by an experienced CPR research group, this CPR did not impact on CT imaging rates.33 In a parallel process evaluation, clinicians’ reported unease with certain components of the rule, and fear of missing a high-stakes diagnosis as reasons for not adopting the CPR.33
Overall, adherence with CPR use during implementation varied considerably between studies ranging from 57% to 100%. Reasons for non-adherence, established through process evaluation, related to fear of missing the diagnosis, preference for own clinical judgement, and patient request for further investigation or management.
Comparison with existing literature
Previous CPR reviews relevant to inpatient and paediatric settings reported issues around the variability of methodological quality in conducting CPR studies and a paucity of impact analysis studies.3 ,6 ,41 The issue of methodological quality has recently been addressed with the publication of two standardised reporting guidelines for CPR derivation and validation studies, and systematic reviews of CPRs.42 ,43 These publications will have an important role to play in standardising CPR research and in promoting robust validation of CPRs which should then be prioritised for evaluation in future impact analysis studies.
CPRs, which demonstrate improvements in the process of care and/or patient outcomes, should then be considered for inclusion in relevant clinical guidelines to facilitate dissemination into clinical practice. A recent survey which examined the use of CPRs in clinical practice by GPs in the UK reported that GPs most often used cardiovascular, depression, fracture and atrial fibrillation CPRs.44 CPR use was dictated by perceived clinical utility, familiarity and local policy requirements. In a supplementary review of clinical guidelines, very little interguideline consistency was found to guide clinicians in terms of which, if any, of these tools to use in practice.44 Prioritising the evaluation of a few adequately validated CPRs with proven predictive accuracy in relevant clinical settings would add significantly to this evidence base and facilitate, if appropriate, the inclusion of certain CPRs into future clinical guidelines.
Implications for clinical practice and research
CPR research is a relatively new methodological discipline and a challenging area of research.2 In the conduct of this review, several uncontrolled before–after impact analysis studies were retrieved. While these studies have a role in contributing to the overall evidence base, they are not a substitute for carefully conducted RCTs in determining the effectiveness of CPRs on clinically relevant outcomes.6 In this review, the majority of included RCTs focused on physician behaviour or process of care as the primary outcome. This is not surprising considering how challenging it is to demonstrate differences in patient outcomes, requiring much larger sample sizes which significantly increase running costs.10 In addition, contextual issues which exist between countries, due to differences in healthcare delivery, healthcare systems and incentives, render process of care outcomes difficult to generalise.5 ,6
Certainly, CPR impact needs to be considered early in the development phase of any new CPR. For instance, Irish research shows high levels of GP referrals to symptomatic breast units.45 Recent research efforts have focused on the development of a breast cancer CPR for use in primary care to aid these referral decisions.46 However, although this CPR underwent methodologically robust development and demonstrates good predictive accuracy it is unlikely its use will impact on referral rates. This is due largely to the existence of a low-risk threshold for referral driven by a combination of factors including patient expectation, media interest and fears of medicolegal ramifications for clinicians if a diagnosis is missed. So when considering an impact analysis RCT in this clinical domain, these contextual issues would need to be addressed in tandem with validation and impact analysis studies.
Certain clinical domains have seen a proliferation of CPR research, particularly musculoskeletal and cardiovascular conditions. The publication of several carefully conducted impact analysis trials for CPRs relating to knee, ankle and neck injuries is largely due to one Canadian research group, while historically, the availability of large UK population data sets facilitated the development of cardiovascular prognostic CPRs.15–18 ,47 ,48 In this review, five impact analysis studies (two were uncontrolled before–after studies detailed in online supplementary appendix 3) focused on the impact of the Ottawa ankle rule in emergency room settings, three of which were conducted in the same country.15 ,16 ,19 ,35 ,36 Ideally, CPR development and impact analysis should be aligned with clinical need rather than developing or testing the effectiveness of CPRs when accurate tools already exist.44
The relatively small number of impact analysis studies retrieved means it is not possible to make firm conclusions about the overall effectiveness of these tools in primary care. However, certain CPRs, such as the Ottawa ankle and knee rules, are appropriate for use in clinical practice, and have a role in reducing unnecessary imaging rates. Future research should focus on conducting RCTs of broadly validated CPRs with consideration of contextual and local implementation factors.7 Pertinent issues include how best to integrate the CPR into clinical workflow, and the potential benefits of embedding CPRs as part of computerised clinical decision support.
Although this review was conducted systematically, and multiple resources searched to retrieve relevant articles, electronic searches were limited to 30 preselected journals, and as a result it is possible that relevant studies were not retrieved. In addition, this search was last updated in December 2013. However, to the best of our knowledge, this review is the first to analyse, in detail, CPR impact analysis studies relevant to primary care. The broad definition of primary care used for this review led to the inclusion of impact analysis studies conducted in the emergency room setting. This was necessary to account for the variation in primary care services and access internationally. Studies that implemented CPRs as part of a broader guideline, protocol or decision aid were excluded. Finally, due to the heterogeneous nature of the included studies, meta-analysis was not possible.
Impact analysis of CPRs in primary care has, to date, focused on a small number of CPRs in a limited number of clinical domains. Future research should focus on prioritising well-validated and accurate CPRs for impact analysis to determine if these tools impact on the process of clinical care and patient outcomes.
EM and MJMU are co-first authors and contributed equally.
Contributors EW wrote the manuscript, acted as a reviewer during the systematic review and contributed to the results. MJMU contributed to writing the manuscript, acted as a reviewer for the systematic review, contributed to the results, and did methodological quality assessment. BC acted as a reviewer for the systematic review, contributed to the results and completed the methodological quality assessment of included articles. AZ acted as a reviewer for the systematic review and contributed to the results. CK ran the original electronic searches, acted as a reviewer, and contributed to the results and discussion. RG acted as a reviewer, and contributed to the results and discussion. SMS acted as a reviewer, and contributed to the results and discussion. TF conceived the idea, oversaw the project and acted as a reviewer. All authors read and approved the final manuscript.
Funding This research was funded by the Health Research Board (HRB) of Ireland through the HRB Centre for Primary Care Research, grant number HRC/2007/1.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement No additional data are available.