Article Text


Healthcare provider perceptions of clinical prediction rules
  1. Safiya Richardson1,
  2. Sundas Khan1,
  3. Lauren McCullagh1,
  4. Myriam Kline2,
  5. Devin Mann3,
  6. Thomas McGinn1
    1. 1Department of Medicine, Hofstra North Shore—LIJ School of Medicine, Manhasset, New York, USA
    2. 2Biostatistics Division, Feinstein Institute for Medical Research, Manhasset, New York, USA
    3. 3Department of Medicine, Boston University, Boston, Massachusetts, USA
    1. Correspondence to Dr Sundas Khan; SKhan31{at}


    Objectives To examine internal medicine and emergency medicine healthcare provider perceptions of usefulness of specific clinical prediction rules.

    Setting The study took place in two academic medical centres. A web-based survey was distributed and completed by participants between 1 January and 31 May 2013.

    Participants Medical doctors, doctors of osteopathy or nurse practitioners employed in the internal medicine or emergency medicine departments at either institution.

    Primary and secondary outcome measures The primary outcome was to identify the clinical prediction rules perceived as most useful by healthcare providers specialising in internal medicine and emergency medicine. Secondary outcomes included comparing usefulness scores of specific clinical prediction rules based on provider specialty, and evaluating associations between usefulness scores and perceived characteristics of these clinical prediction rules.

    Results Of the 401 healthcare providers asked to participate, a total of 263 (66%), completed the survey. The CHADS2 score was chosen by most internal medicine providers (72%), and Pulmonary Embolism Rule-Out Criteria (PERC) score by most emergency medicine providers (45%), as one of the top three most useful from a list of 24 clinical prediction rules. Emergency medicine providers rated their top three significantly more positively, compared with internal medicine providers, as having a better fit into their workflow (p=0.004), helping more with decision-making (p=0.037), better fitting into their thought process when diagnosing patients (p=0.001) and overall, on a 10-point scale, more useful (p=0.009). For all providers, the perceived qualities of useful at point of care, helps with decision making, saves time diagnosing, fits into thought process, and should be the standard of clinical care correlated highly (≥0.65) with overall 10-point usefulness scores.

    Conclusions Healthcare providers describe clear preferences for certain clinical prediction rules, based on medical specialty.

    Statistics from

    Strengths and limitations of this study

    • This is the first study to examine healthcare provider perceptions of usefulness of CPRs in the hospital setting.

    • Providers consistently rated CHADS2, the Thrombolysis in Myocardial Infarction (TIMI) Score (NSTEMI), Wells Score for Pulmonary Embolism, Alcohol Abuse CAGE and the Ottawa Ankle Rule highly. These CPRs would be ideal candidates for integration into an electronic health record (EHR).

    • Emergency medicine providers consistently rated CPRs more positively and may serve as early adapters to CPRs integrated into electronic health records.

    • For all providers, the qualities of CPRs being useful at point of care and that these help with decision-making, save time diagnosing, fit well with one's thought process, and should be the standard of clinical care correlated highly (≥0.65) with usefulness scores.

    • A significant limitation of the results of this study is that mean ratings for CPR characteristics reflect only the opinion of healthcare providers who selected the CPR as one of the top three most useful.


    Evidence-based medicine was announced as a fundamental paradigm shift in medicine in the early 1990s and predicted to de-emphasise intuition, clinical experience and pathophysiological rationale in favour of hard scientific evidence.1 Decades later, the accessible body of clinical research has grown exponentially, but translation into common clinical practice has been protracted and inconsistent.

    The seamless integration of clinical prediction rules (CPRs) into the point of care will aid in transferring evidence-based medicine into daily clinical practice. CPRs can be defined as validated tools that quantify the individual contributions that components of history, physical and laboratory results make towards a diagnosis, prognosis or treatment response.2 A few commonly used CPRs include the CENTOR criteria, which predicts the likelihood of Streptococcal pharyngitis;3 the CAGE score, which serves as a screening test for alcoholism;4 and the CHADS2 score, which predicts the risk of stroke in patients with atrial fibrillation.5

    CPRs integrated into electronic clinical decision support tools have demonstrated the ability to shape healthcare provider behaviour towards more evidence-based clinical practice.6 However, provider adoption continues to be a significant barrier to widespread use of clinical decision support as a whole, which is reported at 10–20%.7 Efficiency, usefulness, information content, user interface and workflow have been reported by clinicians to be the keys to effective decision support.7 These are likely to be large determinants of clinician adoption rates.

    In light of the growing interest in integrated clinical decision support, and CPRs in particular, this study sought to help address the biggest challenge of implementation, poor provider adoption. The study focuses on provider perceptions of usefulness of CPRs in an effort to illuminate preferences, attitudes and thoughts that might be relevant to all types of clinical decision support. We examine healthcare provider perceptions of usefulness based on specialty and level of training with the ultimate goal of discovering which CPRs might be better adopted by these providers.


    A web-based survey platform was distributed to 401 healthcare providers between 1 January and 31 May 2013 in two academic medical centres, Hofstra North Shore—LIJ School of Medicine and Boston University, in the USA.

    The survey content and structure were informed by qualitative interviews with physicians, a literature review and feedback received after pilot testing. The survey was piloted for approximately 1 month and after minor modifications, for instruction clarity and reduced length, distributed via email to attending physicians, nurse practitioners and residents training in the fields of internal medicine (IM) and emergency medicine (EM).

    Providers were included in this study if they were credentialed as medical doctors, doctors of osteopathy or nurse practitioners, and were currently employed in either the IM or EM departments at each institution. Providers were excluded if they were currently involved in the study.

    Participants were recruited, consented and asked to complete the survey via email. Additionally, providers were approached during grand rounds and resident afternoon conferences to encourage them to complete the survey. Laptops with the survey preloaded were placed at meetings to encourage completion. In addition, providers were sent reminder emails twice a month throughout the study period.

    The survey consisted of three distinct sections. In the first section, participants were asked for demographic information, including hospital affiliation, professional degree, current position (attending vs resident), percentage of time devoted to clinical responsibilities, primarily outpatient versus inpatient practice, years of practice, medical specialty, race, gender and age. Demographic information, including race and gender, was assessed to determine the extent to which findings could be generalised to other medical communities.

    In the second section, providers were asked to pick from a list of 24 CPRs: National Emergency X-Radiography Utilization Study (NEXUS) C-Spine Rule,8 Canadian C-Spine Rule,9 Ottawa Knee Rule,10 Walsh,11 Lee Index,12 The Thrombolysis in Myocardial Infarction (TIMI) Risk Score (NSTEMI),13 CHADS2,5 4T Score for Heparin-Induced Thrombocytopenia (HIT),14 Ottawa Ankle Rule,15 Pulmonary Embolism Rule-Out Criteria (PERC),16 Wells Score for deep venous thrombosis (DVT).17 Wells Score for Pulmonary Embolism (PE),18 Alcohol Abuse CAGE,4 Model for End-Stage Liver Disease (MELD) Score,19 San Francisco Rule for Syncope,20 Modified Early Warning System (MEWS),21 CURB 65,22 Ranson's Criteria,23 Pittsburgh Knee Rule,24 Predicting Tuberculosis (TB) in Patients,25 Pneumonia Severity Index (PSI)/Pneumonia Patient Outcomes Research Team (PORT) Score,26 Acute Physiology and Chronic Health Evaluation (APACHE II),27 Mortality in Emergency Department Sepsis (MEDS)28 and Ventilator Associated Pneumonia (VAP).29 They were asked to select all of the CPRs that were familiar to them. Of those CPRs, participants were then asked to select three that they found most useful.

    The last section of the survey applied only to those three CPRs. They were asked questions about their perception of the utility and favourability of the CPRs. Statements such as “The 4T score for Heparin-Induced Thrombocytopenia is easy to use” were rated on a Likert scale from 1 (strongly disagree) to 5 (strongly agree). The last question in this section asked the provider to rate the CPR on a 10-point scale in terms of overall usefulness.

    Statistical methods

    Descriptive statistics, such as means and SDs for continuous variables, and frequencies and proportions for categorical variables, were used to describe the respondent characteristics.

    The χ2 test or Fisher's exact test, as appropriate, was used to explore the association between each of the categorical questionnaire items and the key variables of interest (eg, IM vs EM). The Mann-Whitney test was used to compare the target groups on the ordinal and continuous variables. Finally, the Spearman correlation was used to measure the correlation between selected ordinal variables and the usefulness of the CPR.


    Of the 401 healthcare providers distributed the web-based survey, 22 individuals declined participation, 111 respondents agreed to participate but did not finish the survey, 1 individual completed the survey but left the agreement field blank, and 4 individuals left the agreement field blank and did not finish the survey. A total of 263 individuals, 66% of those asked to participate, agreed to respond, provided written informed consent and completed the survey. No stipend was provided.

    Demographic characteristics

    The IM and EM groups were compared on a number of demographic characteristics (table 1). There were significant differences between the two groups in terms of the institution they represented. Whereas a greater proportion of IM respondents were from Hofstra North Shore-LIJ School of Medicine (77% vs 63%), a greater proportion of the EM respondents were from Boston University (37% vs 22%; p<0.01). Significant differences were also noted when comparing the IM and EM groups on position occupied. Attending physicians in EM were over-represented compared with IM attending physicians (54.2% vs 28%, respectively; p<0.001).

    Table 1

    Demographics of survey participants

    Participants were diverse in terms of age, race and total years of practice. There was a male predominance (61%), which paralleled that seen in national US physician data where only about one-third of medical doctors are women.30 Compared with national US physician data, our sample included slightly less Caucasians, 62% vs 75%; and less African-Americans, 3.3% vs 6%; and more Asian, 27% vs 12.8% physicians. The majority (75%) of the physicians were between 25 and 39 years of age, and had nine or fewer years of practice.

    Most familiar and most useful CPRs

    Participants were asked to select an unlimited number of CPRs that they were familiar with and of those choose three they felt were the most useful (table 2). The Alcohol Abuse CAGE,4 CHADS2,5 TIMI Score (NSTEMI)13 and Wells Score for PE18 were in the top five most frequently chosen as familiar and useful. Ranson's criteria23 was one of the top five most selected as familiar but not as useful and vice versa for the MELD score.19 The CHADS25 score was chosen as most useful by most participants (63%).

    Table 2

    All 24 CPRs, frequency of selection as familiar and top 3 most useful, ordered by mean 10-point usefulness score

    When the list of CPRs most frequently selected as most useful is evaluated by specialty, the rankings diverge. EM providers were more likely to choose CPRs commonly used in emergency departments like PERC,16 NEXUS C-Spine Rule,8 Ottawa Ankle Rule15 and Canadian C-Spine.9 IM providers were more likely to choose CPRs commonly used on inpatient services like CHADS2,5 TIMI score (NSTEMI),13 Alcohol CAGE4 and MELD.19 Of note, both lists for CPRs rated as most useful included the Wells Score for PE.18

    EM versus IM healthcare provider perceptions of CPRs

    Providers were then asked specific questions about each of the three CPRs they rated as most useful. EM providers, compared with IM providers, rated their CPRs significantly more positively as having a better fit into their workflow (p=0.004), helping more with decision-making (p=0.037) and better fitting into their thought process when diagnosing patients (p=0.001) (table 3). There was a trend observed, although not meeting statistical significance, where EM providers consistently reported higher Likert scores for positive CPR qualities, such as easy to use, and IM providers consistently reported higher Likert scores for negative CPR qualities, such as limits independent decision. Lastly, compared with IM providers, EM providers rated their CPRs on a 10-point scale as overall significantly more useful (p=0.009).

    Table 3

    Mean ratings of all chosen CPRs by EM versus IM

    Specific CPR overall usefulness score by provider type

    The overall usefulness score was considered to be the ultimate indicator of strength of provider preference for the CPR. Scores for each CPR's usefulness were compared across provider specialty, resident versus attending position, and primary outpatient versus inpatient practice.

    The only CPR with a significant difference between usefulness scores between specialties was the Ottawa Ankle Rule15 and the Wells Score for PE,18 both preferred by providers in EM. Of note, many of the 24 CPRs could not be compared by specialty because these were not selected by any EM providers as one of the top three most useful, including the 4T Score for HIT,14 APACHE II,27 Lee Index,12 MELD,19 MEWS,21 Predicting TB,25 Ranson's,23 Ventilator Associated Pneumonia29 and MEDS.28

    Two differences were observed between usefulness scores of providers working in mostly or all inpatient versus outpatient settings. Inpatient providers rated the 4T Score for HIT14 as significantly more useful, while providers working in mostly or all outpatient settings rated the Walsh score11 as significantly more useful. There were no differences between resident versus attending ratings of overall usefulness for any CPR.

    CPR characteristics and overall usefulness score

    Ratings for perceived qualities of each CPR were analysed in terms of their correlation with usefulness score (table 4). For all providers, EM and IM, the perceived qualities of being useful at point of care and helps with decision-making, saves time in diagnosing, fits into one's thought process, and should be the standard of clinical care correlated highly (≥0.65) with usefulness scores.

    Table 4

    Correlations between CPR characteristics and the overall usefulness rating


    Perceived utility of clinical decision support tools and clinical guidelines have been previously studied;31 ,32 however, this is the first study to examine healthcare provider perception of usefulness of CPRs in the hospital setting. Providers surveyed in this study reported clear preferences for certain CPRs. Participants consistently rated CHADS2,5 TIMI Score (NSTEMI),13 Wells Score for PE,18 Alcohol Abuse CAGE5 and the Ottawa Ankle Rule15 highly. These CPRs would be ideal candidates for integration into an electronic health record (EHR).

    Interestingly, EM providers consistently rated their chosen CPRs more positively. We found as well that qualities like ease of use, saves time, helps with decision-making, and should be standard of clinical care had a strong relationship to providers’ perception of utility. These qualities should be considered as requirements for a CPR considered for integration into an electronic health record.

    Improved clinical care as well as decreased costs and decreased waste are potential results of provider preferred integrated CPRs. Although the USA spends nearly double the average, $3923, of all of the Organisation for Economic Co-operation and Development (OECD) countries33 on healthcare, American patients receive about 55% of recommended clinical care.34 Overtreatment and failures in execution of care processes are partially responsible for waste in healthcare spending, estimated as exceeding 20%.35 Meta-analysis of the effect of clinical decision support has shown that providers with decision support were more likely to provide preventive care services and order appropriate treatments.7


    A significant limitation of the results of this study is that mean ratings for CPR characteristics reflect only the opinion of healthcare providers who selected the CPR as one of the top three most useful. However, the structure of the survey also ensures that CPR characteristic ratings were made only by providers who were likely to use the CPR in daily practice.

    Additionally, participants were recruited during academic conferences, including grand rounds as well as afternoon conferences for residents. This may have increased the number of participants who attend academic conferences, and who are more familiar with CPRs.

    Implications for clinical practice and research

    Meaningful clinical decision support requires not just understanding healthcare provider perceptions, but also choosing tools that are strongly evidence-based and have been tested for their effectiveness. Future trials should focus on evaluating the clinical impact of healthcare provider preferred CPRs.


    Healthcare providers describe clear preferences for certain characteristics and disease-specific CPRs. EM providers consistently rated CPRs more positively and may serve as early adapters for CPRs integrated into EHRs. Understanding provider perceptions may help to address limiting factors in meaningful integration of clinical decision support into our electronic health systems.


    The authors would like to acknowledge the contributions of Megan Knaus, research coordinator affiliated with Hofstra North Shore—LIJ who helped in survey development and Dr Ambili Ramachandran, Internal Medicine resident affiliated with Boston University, who led data collection at that site. The abstract from this work was previously presented as a poster at the EDM Forum Symposium in San Diego California on 7 June 2014.


    View Abstract
    • Supplementary Data

      This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.


    • Collaborators Ambili Ramachandran, Megan Knaus.

    • Contributors TM and DM were involved in study concept and design, critical revision of the manuscript for important intellectual content and study supervision. SR was involved in acquisition of data. SR, TM, LM, SK and MK were involved in analysis and interpretation of data and drafting of the manuscript. SR, SK, LM and MK were involved in statistical analysis. LM and SK were involved in administrative, technical and material support.

    • Competing interests None declared.

    • Ethics approval North Shore—LIJ Health System IRB.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Data sharing statement Additional data can be accessed by emailing MK, Associate Research Statistician, Biostatistics Unit, Feinstein Institute for Medical Research, North Shore-LIJ Health System.

    Request permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.