You are here

  • Home
  • Archive
  • Volume 2, Issue 5
  • Physicians’ response to computerised alerts for psychotropic drugs in older persons: a multilevel analysis of the associated alert, patient and physician characteristics

Article Text

Article menu
XML
Health informatics
Research
Physicians’ response to computerised alerts for psychotropic drugs in older persons: a multilevel analysis of the associated alert, patient and physician characteristics
  1. Robyn Tamblyn1,2,3,
  2. Kristen Reidel3,
  3. Vaishali Patel3
  1. 1Department of Medicine, McGill University, Montreal, Quebec
  2. 2Department of Epidemiology, Biostatistics and Occupational Health, McGill University, Montreal, Quebec
  3. 3Clinical and Health Informatics Research Group, McGill University, Montreal, Quebec
  1. Correspondence to Dr Robyn Tamblyn; robyn.tamblyn{at}mcgill.ca

Abstract

Objective Computerised drug alerts are expected to reduce patients’ risk of adverse drug events. However, physicians over-ride most drug alerts, because they believe that the benefit exceeds the risk. The purpose of this study was to determine the drug alert, patient and physician characteristics associated with the: (1) occurrence of psychotropic drug alerts for elderly patients and the (2) response to these alerts by their primary care physicians.

Setting Primary care, Quebec, Canada.

Design Prospective cohort study.

Participants Sixty-one physicians using an electronic prescribing and drug alert decision-support system in their practice, and 3413 elderly patients using psychotropic drugs.

Primary and secondary measures Psychotropic drug class, alert severity, patient risk for fall injuries and physician experience, practice volume and computer use were evaluated in relationship to the likelihood of having: (1) a psychotropic drug alert, (2) the prescription revised in response to an alert. Cluster-adjusted alternating logistic regression was used to assess multilevel predictors of alert occurrence and response.

Results In total 13 080 psychotropic drug alerts were generated in 8931 visits. Alerts were more likely to be generated for male patients at higher risk of fall-related injury and for physicians who established the highest alert threshold. In 9.9% of alerts seen, the prescription was revised. The highest revision rate was for antipsychotic alerts (22.6%). Physicians were more likely to revise prescriptions for severe alerts (OR 2.03; 95%CI 1.39 to 2.98), if patients had cognitive impairment (OR 1.95; 95%CI 1.13 to 3.36), and if they made more visits to their physician (OR 1.05 per 5 visits; 95%CI 1 to 1.09).

Conclusions Physicians view and respond to a small proportion of alerts, mainly for higher-risk patients. To reduce the risk of psychotropic drug-related fall injuries, a new generation of evidence-based drug alerts should be developed.

  • Epidemiology
  • Geriatric Medicine

This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/2.0/ and http://creativecommons.org/licenses/by-nc/2.0/legalcode.

https://doi.org/10.1136/bmjopen-2012-001384

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Article summary

    Article focus

    • Computerised decision support is expected to reduce prescribing errors that may increase the risk of adverse events but most alerts are ignored.

    • We assessed whether psychotropic drug alerts were more likely to be generated for higher-risk patients, and for physicians in relationship to their demographic and practice characteristics. We also assessed whether these characteristics were associated with the likelihood that a prescription would be revised in relationship to an alert.

    Key messages

    • Commercial drug decision-support systems generate many alerts for psychotropic drugs, 97% of which are of moderate or serious severity (not severe), and the majority of these alerts will be ignored by primary care physicians.

    • Patients at greater risk of fall-related injuries are more likely to have psychotropic drug alerts, as are male physicians, and physicians who screen out the majority of alerts.

    • Physicians are more likely to change a prescription in response to an alert if the patient is a frequent visitor at the clinic and if they have cognitive impairment.

    Strengths and limitations of this study

    • This is the first study to use empirically generated audit trails within a computerised medical record to simultaneously evaluate the drug, patient and physician characteristics that are associated with the likelihood of having a psychotropic drug alert generated by a commercial drug alert system, as well as the physicians’ response to an alert, in a large cohort of primary care physicians and their elderly patients.

    • We were unable to assess all possible alert-related predictors of physician response (eg, type of alert) as the majority of physicians elected to view only the severe alerts.

    Introduction

    Psychotropic medication use is associated with an increased risk of adverse drug events such as fall-related injuries in older persons.1–4 Fall-related injuries among older adults are a significant source of morbidity, mortality, and healthcare costs. In the USA, about a third of adults 65 years and older experience a fall annually, resulting in US$19 billions of healthcare service use.5 ,6 Half of the adverse drug events associated with fall-related injuries in older persons are considered preventable and a majority of errors that contribute to preventable adverse drug events occur at the time of prescribing.7

    Improving the management of psychotropic medications has been identified as a potentially cost-effective means by which to reduce fall-related injuries in older persons.7 ,8 Computerised prescribing systems integrated with clinical decision support systems could serve as a means to more effectively prescribe psychotropic medications because these systems have been found to prevent medication errors by providing physicians with alerts at the point of prescribing.9–11 However, evidence is mounting that physicians override the majority of computerised drug alerts they receive from such systems—reported at upwards of 90% in some studies4—which may limit the utility of such systems in improving medication management of psychotropic medications in older adults.

    Physicians’ high rates of over-riding computerised drug alerts may be associated with ‘alert fatigue,’ which has been theorised to occur when a high volume of drug alerts of limited clinical relevance are generated.12–14 Interventions have been developed to address ‘alert fatigue’ by modifying how alerts are presented and viewed, including reducing the volume of clinically irrelevant alerts through an expert panel review15 or by providing alerts on-demand to physicians.16 However, these interventions have met with varying levels of success. Furthermore, evaluations of physicians’ response to alerts suggest that a complex set of factors may shape physicians’ usage of these systems including features of the computerised drug alert system and alerts15 ,17 ,18 the complexity of patients’ health needs19 and physician characteristics and practice environment.19

    Identifying the multilevel determinants of response to alerts may help refine the content and delivery of alerts in a manner that is better attuned to physicians’ practice and patients’ needs.

    We conducted a study to identify factors associated with physicians’ response to alerts for older adults on psychotropic medications. We also sought to characterise the key steps leading to a physician's response to alerts, including factors associated with the generation and viewing of the alert, as these steps may be linked to whether physicians subsequently respond to an alert.

    Methods

    Study design and population

    We conducted a prospective cohort study with 61 primary care physicians, and all of their patients aged 65 years of age or older who were prescribed psychotropic drugs (n=3413). All computerised drug alerts generated for these patients between 1 January and 31 December 2007 that involved their psychotropic medications were retrieved. This study was approved by the McGill University Institutional Review Board.

    Electronic prescribing system

    All physicians used a bilingual (English/French) electronic health record and drug management system (MOXXI) that is, integrated with the provincial health insurance agency (Régie de l'assurance maladie du Québec, RAMQ). The RAMQ provides daily downloads of all new medical services and prescriptions for all consenting patients in the practice from its population-based health insurance databases, enabling new health problems, procedures, emergency department visits, hospitalisations and medications to be automatically updated in the patient's health record. The MOXXI system includes a health problem list, an electronic prescriber, information on all medications dispensed including the drug, strength, quantity, duration and prescribing physician, and automated alerts for potential prescribing problems.20

    The automated prescribing alerts and drug knowledge database, maintained by Vigilance Santé (Montreal, Canada), generates alerts using information from the patient's current medication list and new electronic prescriptions for all major types of prescribing problems, including excess dose, drug–allergy, drug–drug, drug–disease, drug–age contraindications and therapeutic duplication (www.vigilance.ca). The drug knowledge base classifies all types of alerts by the level of clinical importance on a three-level severity scale, with level 1 alerts defined as ‘severe alerts—avoid completely’, level 2 alerts defined as ‘serious alerts, avoid if possible’ and level 3 alerts as ‘moderate alerts, use with caution’. Alerts are presented in an interruptive manner, forcing physicians to either change the prescription to remove the alert, or enter a reason for ignoring the alert before being able to print the prescription (see the online appendix for screenshots). Physicians may view a detailed description of the interactions or contraindications identified in the alert by clicking in a detail panel. In addition, physicians have the option to suppress and not view serious and/or moderate alerts (eg, level 2 and/or 3) as well as individual drug alerts that he or she disagrees with at the time the alert is generated. In addition, all alerts, current and previously suppressed, can be viewed by the physician in a non-interruptive manner by accessing the alert history section of the application.

    Data sources

    Audit trails in the MOXXI database were used to measure physicians’ usage of the system, including date and time of use, screens viewed and functions used, prescriptions written, alerts generated, alerts seen by the physician (physician alert level preference) and the physician's response to alerts he or she viewed. Physicians’ characteristics and e-prescribing experience was assessed at the completion of training using questionnaire and standardised e-prescribing skill assessment.11

    Patient and practice characteristics were assessed using data from Quebec's universal public health insurance program (RAMQ). For each consenting patient and physician, data were available from the beneficiary, prescription and medical services claims databases to measure beneficiary age, sex and neighbourhood measures of socio-economic status as well as all medical visits, hospitalisations, prescriptions and health problems and injuries recorded in the year prior to enrolment and throughout the study period.21 The accuracy of these data have been validated in prior studies.22–28

    Outcomes

    Alert generated

    Using data from the MOXXI database, we determined, at each patient visit, whether one or more psychotropic drug alerts was generated. An alert could be generated on the basis of existing active medication (the patient's profile of dispensed medication) and/or a newly prescribed psychotropic medication. To evaluate patient and physician characteristics associated with the generation of alerts, each visit was classified as a visit with or without a psychotropic drug alert.

    Alert medication revised

    For each psychoactive drug alert that was generated at a visit, we first assessed whether the physician saw the alert as a function of the physician's ‘alert level setting’. Among alerts that were seen, we assessed whether the physician made an effort to modify, stop or change the dose of one or more of the drugs implicated in the alert, or alternately over-rode the alert in which case they were required to record the reason for doing so.

    Multilevel predictors

    Physician-level predictors

    Physician demographic characteristics examined included gender and practice experience, defined as the number of years since graduation from medical school. Both were measured at enrolment as these characteristics are associated with prescribing behaviour and may be predictive of alert generation and response. Practice characteristics such as practice volume and the proportion of elderly in the practice are also associated with prescribing behaviour.29 Daily practice volume was calculated based on medical service billing claims made by each physician during 2006, calculated by counting the number of visits the physician billed for that year and dividing by the number of workdays with one or more billings. The percentage of patients 65 years and older was measured as the number of patients 65 years or older that the physician billed a service for in 2006, divided by the total number of patients the physician billed a service for in 2006. Experience and skills related to electronic prescribing was also measured as a factor in alert response as it may influence the speed and confidence in modifying prescriptions in response to an alert. The electronic prescription rate was measured by calculating the number of electronic prescriptions (e-RXs) per 100 visits, and the electronic prescription speed was defined as the time (in minutes) to write three standardised prescriptions electronically. Lastly, the physician-selected alert-level setting was measured as a possible determinant of alert response. Physicians could choose to view severe alerts only (level 1 alerts), view severe and serious alerts (levels 1 and 2 alerts), or view all alerts (levels 1, 2 and 3 alerts) as ‘interruptive alerts’ during the prescribing process.

    Patient-level predictors

    Five patient attributes were measured as factors that may influence the likelihood of alerts as well as physician response to alerts. Patient age was measured as of 1 January 2007, and patient gender was identified. In addition, patient risk factors for fall-related injuries were measured from ICD9 diagnostic and procedure codes in their medical service claims from the years 2005 and 2006, using a previously validated set of codes for each risk factor.21 ,24 ,30 ,31 Risk factors measured included the presence of a prior fall-related injury, gait and balance problems, lower-extremity weakness, and cognitive impairment. We also assessed each patient's medication complexity by counting the number of active medications (prescribed or dispensed, including psychotropic medications) at each visit. Lastly, we measured visit frequency as a reflection of the opportunity to modify treatment, defined as the number of ambulatory care visits made in the year prior to the visit date.

    Alert-level predictors

    The attributes of each alert was characterised by type of alert, severity and therapeutic class. Type of alert was classified into one of nine mutually exclusive categories: drug–disease contraindication, cumulative side effects, drug–age contraindication, therapy duplication, drug interaction, excess dose, renal dose adjustment, drug or potential cross allergy and other. Alert severity was classified as moderate, serious or severe using the classification established by the vendor (www.vigilance.ca). Therapeutic class of drug(s) causing the alert included antidepressants, antipsychotics, anticonvulsants, benzodiazepines, opioids and other psychoactive classes.

    Analysis

    To estimate the relationship between the alert, patient and physician predictors and (1) alert generation and (2) response to alerts we used alternating logistic regression. Alternating logistic regression models provide a method of deriving unbiased estimates of association as it permits non-independence of observations with more than one level of clustering to be managed within the context of a dichotomous outcome. For the outcome, alert generated (yes/no), the unit of analysis was visit and the outcome was modelled with two levels of clustering: visits within patients, and patients within physician. For the outcome, alert revised (yes/no), the unit of analysis was alert and the outcome was modelled with two levels of clustering; alerts within patient and patients within physician. For the generation of alerts outcome, sample size was sufficient to include all variables in the multivariate model using alternating logistic regression. However, for the revision of alerts outcome the sample size of alerts that were actually seen by physicians was not sufficient to include all variables in the multivariate model and have it converge. Therefore, only variables found to have a bivariate association with a level of statistical significance of p=0.20 or less were included in the final model predicting revision of alerts.

    Results

    Overall, 44% of physicians were female. On average, physicians had 23.4 years of practice experience, 21% (SD 11.7%) of patients in the physicians’ practice population were 65 years of age and older, 21 patients were seen per day and 53.7 electronic prescriptions were written per 100 visits (table 1). Physicians varied in computer-based prescribing proficiency with a range in the time required to write three standardised prescriptions of 2.8–10.4 min. Most physicians (82%) set the drug alert preference to view only severe alerts. Eligible patients had a mean age of 75.5 years, 9.4 active drugs and 12.2 visits. In total 67.5% were female, and up to 24.1% had at least one risk factor for fall-related injuries, the most common being gait and balance problems (table 1).

    View this table:
    Table 1

    Physician and patient characteristics

    In 2007, 13 080 psychotropic drug alerts were generated in the 8931 visits made by study patients; an average of 1.5 psychotropic drug alerts per visit. The majority of alerts were drug–disease contraindications (42.9%) followed by cumulative side effects (23.8%; table 2). Most alerts were of serious (52%) or moderate (45.1%) clinical importance. Antidepressants (45%) and benzodiazepines (30.8%) were most common drugs implicated. Only a minority of the alerts generated (6.2%) were seen because the majority of physicians had set their preferences to see only severe alerts. As a result, alert types most likely to be seen were for excess dose (69.8%), drug or potential cross allergies (29%) and drug–age contraindications (18.3%). Among those that were seen, 9.9% were revised, with the highest revision rate being for excess dose (14.8%), renal dose adjustment (16.7%), severe alerts (16.2%) and for antipsychotics (22.6%).

    View this table:
    Table 2

    Characteristics of alerts by stage in alert generation, viewing and revision process

    Factors associated with alert occurrence

    Alerts were more likely to be generated at a visit for patients who had a prior fall-related injury (OR 1.44; 95% CI 1.03 to 2.01; table 3). Female patients after adjusting for greater visit frequency were less likely than males to have a psychotropic drug alert generated during the visit (OR 0.85; 95% CI 0.75 to 0.98). Female physicians (OR 0.61; 95% CI 0.42 to 0.91), and physicians who saw more patients per day (OR per 5 patient increase: 0.89, 95% CI 0.79 to 0.99) were less likely to have psychotropic drug alerts, whereas physicians who set their alert preferences to view only the severe alerts were 99% (95% CI 1.04 to 3.81) more likely to have a psychotropic drug alert generated for their patients.

    View this table:
    Table 3

    Patient and physician level predictors of alert generation

    Factors associated with alert response

    As only 80 alerts were revised, we were unable to include all potential predictors of alert response in a multivariate alternating logistic regression model. Therefore, we included only predictors with a p value ≤0.20 in the bivariate association (table 4). Severe alerts were twice as likely to be revised when compared to serious or moderate alerts (OR 2.03; 95% CI 1.39 to 2.98). Alerts for renal dose adjustment were the most likely type of alert to be revised (OR 1.64; 95% CI 1.09 to 2.46), with a significantly higher likelihood of revision than drug–age contraindications (reference group) and drug interactions (OR 0.49; 95% CI 0.22 to 1.08). Drug alerts for patients with cognitive impairment were 95% more likely to be revised than those with no cognitive problems (OR 1.95; 95% CI 1.13 to 3.36), and the likelihood that an alert would be revised increased for patients who made more visits to their physician (OR 1.05 per 5 visit increase; 95% CI 1.004 to 1.09). Physician sex, practice experience, practice characteristics, computer skill and alert level setting were not associated with the likelihood of responding to an alert.

    View this table:
    Table 4

    Factors associated with revising a prescription in response to an alert

    Discussion

    This is the first study to employ empirically generated audit trails with a computerised medical record to investigate the alert, patient and physician characteristics associated with both the generation of psychotropic alerts from a commercially available drug alert system and physician response in primary care. We found that psychotropic drug alerts were common, with 1.5 alerts occurring per visit, that the majority (97%) were moderate or serious alerts (not severe alerts) and that physicians saw a minority (6.1%) of these alerts because they (82%) set their threshold for viewing alerts to see only the severe alerts. Patients at higher risk of fall-related injuries were more likely to have alerts for their psychotropic drugs, and were also more likely to have medications revised in response to an alert. Alerts were more likely to be generated for patients’ psychotropic medications if their physician was male and had established an alert threshold to view only the severe alerts. Additionally, the severity and type of the alert were associated with a physician's response to an alert.

    Our findings are coherent with prior research,12–17 ,19 ,32–34 as well as a recent Delphi assessment of important attributes of drug alert systems to improve clinical utility.35 Alerts are considered clinically useful when graded by severity, when patient risk is considered, and when the probability of an adverse event is significant. Physicians in this study responded in a similar manner; preferentially revising severe alerts and in patients who were at higher risk of an adverse medication-related event. Both Isaac and Grizzle noted a similar pattern.17 ,34 Physicians were more likely to respond to the most severe alerts, and for patients who were being newly exposed to a drug or drug combination, where the risk would be higher than those who had already demonstrated that they could tolerate the medication. These findings as a group suggest that alerts will be most useful if graded by clinical importance, and if patient-specific estimates of risk are calculated and presented, so that only the most clinically relevant alerts could be presented.

    However, one of the greatest barriers to developing more clinically useful alert systems is the paucity of empirical evidence of the risk associated with the vast majority of alerts. Most current alert systems, both commercial and home-grown, base the grading of alerts as well as the probability of harm on lowest level of evidence—expert opinion,36–38 although there are notable recent attempts39 to grade alerts by the strength of the evidence. To make substantial gains in improving drug safety, and reducing the risk of adverse drug events with computerised decision support, real-world evidence needs to be generated on the risks associated with drug alerts, and this evidence needs to be incorporated into a new generation of alert systems. For example, with the increasing use of electronic medical records, an opportunity exists to use clinical and administrative data to monitor adverse drug events and model the actual risks of adverse events associated with prescribed medications. Moreover, priority should be placed on drugs and drug combinations that have the highest population attributable risk, which is a function of the prevalence of use, magnitude of the risk and severity of adverse effects. This would likely include the most prevalent classes of medication—cardiovascular drugs, antibiotics, psychotropic medications, anti-inflammatories and analgesics, as well as those with established high risks such as anticoagulants.

    This is the first study to document that physicians who are most likely to generate alerts are also more likely to establish the highest threshold for viewing alerts. This finding, while new, fits with prior observations on the high frequency of overriding computerised prescribing alerts by physicians.12 ,15 ,16 ,19 Alert fatigue created by too many drug alerts is a common complaint,17 ,34 and methods of over-riding, or suppressing alerts are highly valued, particularly when alerts are presented in an interruptive manner. However, if physicians with a high volume of drug alerts are more likely to have a higher rate of adverse drug events in their patient population, then facilitating alert over-rides defeats the purpose of implementing computerised decision support in the first place. It has been assumed that reducing the sheer number of alerts to the most serious problems will solve the problem of alert fatigue and alert over-rides. But this assumption may not be correct. A growing body of literature in educational psychology suggests that physicians who have the highest prevalence of quality and safety problems in practice may also be most likely to over-rate the quality of their performance.40 If these factors influence alert response, then the physicians who are most likely to generate severe drug alerts may be least likely to respond to them. This possibility needs to be assessed in future research, as more comprehensive feedback, such as comparative benchmarking for quality of drug management as well as patients outcomes, may need to be included with drug alerts to alter physician's self-assessment of performance so that safety and quality problems will be addressed.41

    Our results need to be considered in light of several limitations. First, the number of alerts that were actually seen by physicians and revised was small, and we had insufficient power to identify factors that may have increased the likelihood of responding to an alert by 50% or less. Our sample is limited to primary care physicians and psychotropic drug alerts. The generalisability of these findings to alerts related to other classes of drugs, to other alerting systems, and to other types of physicians or health professionals is not known and would be an important avenue for future research. Lastly, we studied physicians in one jurisdiction, and were unable to assess contextual factors that may modify physician response to alerts such as jurisdictional differences in malpractice litigation risk, funding for the quality of performance rather than for services alone, or the existence of support systems for quality improvement.

    In conclusion, to reduce the risk of psychotropic drug-related fall injuries, a new generation of drug alerts should be developed that incorporates empirical estimates of the risk of injury related to psychotropic drugs, as well as patient characteristics. Eventually, these empirical risk estimates could be calculated using clinical and administrative data by systems capable of modelling such data in real time, providing earlier warnings about new drugs associated with increased risks of fall injuries. In addition, the need to incorporate more comprehensive feedback on the quality and outcomes of drug prescribing for physicians that generate a high proportion of alerts should be evaluated in subsequent research.

    Acknowledgments

    RT was supported by the Canadian Institutes of Health Research Emerging Team Grant. VP was supported by the CIHR Postdoctoral Fellowship Award. KR was supported by the Frederick Banting and Charles Best Canada Graduate Scholarship.

    References

    1. ↵
      1. French DD,
      2. Campbell R,
      3. Spehar A,
      4. et al
      . Drugs and falls in community-dwelling older people: a national veterans study. Clin Ther 2006;28:619–30.
      OpenUrlCrossRefPubMed
    2. ↵
      1. Leipzig RM,
      2. Cumming RG,
      3. Tinetti ME
      . Drugs and falls in older people: a systematic review and meta-analysis: I. Psychotropic drugs. J Am Geriatr Soc 1999;47:30–9.
      OpenUrlCrossRefPubMedWeb of Science
    3. ↵
      1. Hartikainen S,
      2. Lonnroos E,
      3. Louhivuori K
      . Medication as a risk factor for falls: critical systematic review. J Gerontol A Biol Sci Med Sci 2007;62:1172–81.
      OpenUrlAbstract/FREE Full Text
    4. ↵
      1. Eslami S,
      2. Abu-Hanna A,
      3. de Keizer NF
      . Evaluation of outpatient computerized physician medication order entry systems: a systematic review. J Am Med Inform Assoc 2007;14:400–6.
      OpenUrlAbstract/FREE Full Text
    5. ↵
      1. Ganz DA,
      2. Bao Y,
      3. Shekelle PG,
      4. et al
      . Will my patient fall? JAMA 2007;297:77–86.
      OpenUrlCrossRefPubMedWeb of Science
    6. ↵
      1. Rubenstein LZ,
      2. Josephson KR
      . The epidemiology of falls and syncope. Clin Geriatr Med 2002;18:141–58.
      OpenUrlCrossRefPubMedWeb of Science
    7. ↵
      1. Gurwitz JH,
      2. Field TS,
      3. Harrold LR,
      4. et al
      . Incidence and preventability of adverse drug events among older persons in the ambulatory setting. JAMA 2003;289:1107–16.
      OpenUrlCrossRefPubMedWeb of Science
    8. ↵
      1. Frick KD,
      2. Kung JY,
      3. Parrish JM,
      4. et al
      . Evaluating the cost-effectiveness of fall prevention programs that reduce fall-related hip fractures in older adults. J Am Geriatr Soc 2010;58:136–41.
      OpenUrlCrossRefPubMedWeb of Science
    9. ↵
      1. Kaushal R,
      2. Shojania KG,
      3. Bates DW
      . Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med 2003;163:1409–16.
      OpenUrlCrossRefPubMedWeb of Science
    10. ↵
      1. Poissant L,
      2. Taylor L,
      3. Huang A,
      4. et al
      . Assessing the accuracy of an inter-institutional automated patient-specific health problem list. BMC Med Inform Decis Mak 2010;10:10.
      OpenUrlCrossRefPubMed
    11. ↵
      1. Tamblyn R,
      2. Huang A,
      3. Kawasumi Y,
      4. et al
      . The development and evaluation of an integrated electronic prescribing and drug managment system for primary care. J Am Med Inform Assoc 2006;13:148–59.
      OpenUrlAbstract/FREE Full Text
    12. ↵
      1. van der Sijs H,
      2. Aarts J,
      3. Vulto A,
      4. et al
      . Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc 2006;13:138–47.
      OpenUrlAbstract/FREE Full Text
    13. ↵
      1. Ko Y,
      2. Abarca J,
      3. Malone DC,
      4. et al
      . Practitioners’ views on computerized drug-drug interaction alerts in the VA system. J Am Med Inform Assoc 2007;14:56–64.
      OpenUrlAbstract/FREE Full Text
    14. ↵
      1. Glassman PA,
      2. Belperio P,
      3. Simon B,
      4. et al
      . Exposure to automated drug alerts over time: effects on clinicians’ knowledge and perceptions. Med Care 2006;44:250–6.
      OpenUrlCrossRefPubMedWeb of Science
    15. ↵
      1. Shah NR,
      2. Seger AC,
      3. Seger DL,
      4. et al
      . Improving acceptance of computerized prescribing alerts in ambulatory care. J Am Med Inform Assoc 2006;13:5–11.
      OpenUrlAbstract/FREE Full Text
    16. ↵
      1. Tamblyn R,
      2. Huang A,
      3. Taylor L,
      4. et al
      . A randomized trial of the effectiveness of on-demand versus computer-triggered drug decision support in primary care. J Am Med Inform Assoc 2008;15:430–8.
      OpenUrlAbstract/FREE Full Text
    17. ↵
      1. Isaac T,
      2. Weissman JS,
      3. Davis RB,
      4. et al
      . Overrides of medication alerts in ambulatory care. Arch Intern Med 2009;169:305–11.
      OpenUrlCrossRefPubMedWeb of Science
    18. ↵
      1. Taylor LK,
      2. Tamblyn R
      . Reasons for physician non-adherence to electronic drug alerts. Medinfo 2004;11(Pt 2):1101–5.
      OpenUrl
    19. ↵
      1. Weingart SN,
      2. Toth M,
      3. Sands DZ,
      4. et al
      . Physicians’ decisions to override computerized drug alerts in primary care. Arch Intern Med 2003;163:2625–31.
      OpenUrlCrossRefPubMedWeb of Science
    20. ↵
      1. Tamblyn R,
      2. Huang A,
      3. Kawasumi Y,
      4. et al
      . The development and evaluation of an integrated electronic prescribing and drug management system for primary care. J Am Med Inform Assoc 2006;13:148–59.
      OpenUrlAbstract/FREE Full Text
    21. ↵
      1. Buckeridge D,
      2. Huang A,
      3. Hanley J,
      4. et al
      . Risk of injury associated with opioid use in older adults. J Am Geriatr Soc 2010;58:1664–70.
      OpenUrlCrossRefPubMedWeb of Science
    22. ↵
      1. Tamblyn RM,
      2. Lavoie G,
      3. Petrella L,
      4. et al
      . The use of prescription claims databases in pharmacoepidemiological research: the accuracy and comprehensiveness of the prescription claims database in Quebec. J Clin Epidemiol 1995;48:999–1009.
      OpenUrlCrossRefPubMedWeb of Science
    23. ↵
      1. Levy AR,
      2. Tamblyn RM,
      3. Fitchett D,
      4. et al
      . Coding accuracy of hospital discharge data for elderly survivors of myocardial infarction. Can J Cardiol 1999;15:1277–82.
      OpenUrlPubMedWeb of Science
    24. ↵
      1. Tamblyn RM,
      2. Reid T,
      3. Mayo N,
      4. et al
      . Using medical services claims to assess injuries in the elderly: the sensitivity of diagnostic and procedure codes for injury ascertainment. J Clin Epidemiol 2000;53:183–94.
      OpenUrlCrossRefPubMedWeb of Science
    25. ↵
      1. Wilchesky M,
      2. Tamblyn RM,
      3. Huang A
      . Validation of diagnostic codes within medical services claims. J Clin Epidemiol 2004;57:131–41.
      OpenUrlCrossRefPubMedWeb of Science
    26. ↵
      1. Cadieux G,
      2. Tamblyn R
      . Accuracy of physician billing claims for identifying acute respiratory infections in primary care. Health Serv Res 2008;43:2223–38.
      OpenUrlCrossRefPubMedWeb of Science
    27. ↵
      1. Kawasumi Y,
      2. Abrahamowicz M,
      3. Ernst P,
      4. et al
      . Development and validation of a predictive algorithm to identify adult asthmatics from medical services and pharmacy claims databases. Health Serv Res 2011;46:939–63.
      OpenUrlPubMedWeb of Science
    28. ↵
      1. Cadieux G,
      2. Buckeridge DL,
      3. Jacques A,
      4. et al
      . Accuracy of syndrome definitions based on diagnoses in physician claims. BMC Public Health 2011;11:17.
      OpenUrlCrossRefPubMed
    29. ↵
      1. Tamblyn R,
      2. McLeod P,
      3. Hanley JA,
      4. et al
      . Physician and practice characteristics associated with the early utilization of new prescription drugs. Med Care 2003;41:895–908.
      OpenUrlCrossRefPubMedWeb of Science
    30. ↵
      1. Deyo RA,
      2. Cherkin DC,
      3. Ciol MA
      . Adapting a clinical comorbidity index for use with ICD-9-CM administrative databases. J Clin Epidemiol 1992;45:613–19.
      OpenUrlCrossRefPubMedWeb of Science
    31. ↵
      1. Romano PS,
      2. Roos LL,
      3. Jollis JG
      . Adapting a clinical comorbidity index for use with ICD-9-CM administrative data: differing perspectives. J Clin Epidemiol 1993;46:1075–9.
      OpenUrlCrossRefPubMedWeb of Science
    32. ↵
      1. Kucher N,
      2. Puck M,
      3. Blaser J,
      4. et al
      . Physician compliance with advanced electronic alerts for preventing venous thromboembolism among hospitalized medical patients. J Thromb Haemost 2009;7:1291–6.
      OpenUrlCrossRefPubMed
    33. ↵
      1. Scott GPT,
      2. Shah P,
      3. Wyatt JC,
      4. et al
      . Making electronic prescribing alerts more effective: scenario-based experimental study in junior doctors. J Am Med Inform Assoc 2011;18:789–98.
      OpenUrlCrossRefPubMedWeb of Science
    34. ↵
      1. Grizzle AJ,
      2. Mahmood MH,
      3. Ko Y,
      4. et al
      . Reasons provided by prescribers when overriding drug-drug interaction alerts. Am J Manag Care 2007;13:573–8.
      OpenUrlPubMedWeb of Science
    35. ↵
      1. Riedmann D,
      2. Jung M,
      3. Hackl WO,
      4. et al
      . How to improve the delivery of medication alerts within computerized physician order entry systems: an international Delphi study. J Am Med Inform Assoc 2011;18:760–6.
      OpenUrlCrossRefPubMedWeb of Science
    36. ↵
      1. Smithburger PL,
      2. Kane-Gill SL,
      3. Benedict NJ,
      4. et al
      . Grading the severity of drug-drug interactions in the intensive care unit: a comparison between clinician assessment and proprietary database severity rankings. Ann Pharmacother 2010;44:1718–24.
      OpenUrlCrossRefPubMedWeb of Science
    37. ↵
      1. Malone DC,
      2. Abarca J,
      3. Hansten PD,
      4. et al
      . Identification of serious drug-drug interactions: results of the partnership to prevent drug-drug interactions. Am J Geriatr Pharmacother 2005;3:65–76.
      OpenUrl
    38. ↵
      1. Paterno MD,
      2. Maviglia SM,
      3. Gorman PN,
      4. et al
      . Tiering drug-drug interaction alerts by severity increases compliance rates. J Am Med Inform Assoc 2009;16:40–6.
      OpenUrlAbstract/FREE Full Text
    39. ↵
      1. van Roon EN,
      2. Flikweert S,
      3. le Comte M,
      4. et al
      . Clinical relevance of drug-drug interactions: a structured assessment procedure. Drug Saf 2005;28:1131–9.
      OpenUrlCrossRefPubMedWeb of Science
    40. ↵
      1. Eva KW,
      2. Regehr G
      . “I'll never play professional football” and other fallacies of self-assessment. J Contin Educ Health Prof 2008;28:14–19.
      OpenUrlCrossRefPubMedWeb of Science
    41. ↵
      1. Dauphinee WD,
      2. Tamblyn R,
      3. Cruess SR,
      4. et al
      . Commentary: one strategy for building public trust in changing times. Acad Med 2010;85:941–4.
      OpenUrlPubMedWeb of Science

    Supplementary materials

    • Supplementary Data

      This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

      Files in this Data Supplement:

    Footnotes

    • Contributors All authors contributed significantly to the conception and design or analysis and interpretation of the data. All authors drafted the article or revised it for important intellectual content, and have given final approval of the version to be published.

    • Funding Canadian Institutes of Health Research (CIHR) New Emerging Team; CIHR postdoctoral fellowship funded VP, and a CIHR MSc award funded KR.

    • Competing interests None.

    • Ethics approval Ethical Review Board, McGill University.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Data sharing statement There are no additional data available.