Article Text

Validating ICD-10 codes for adverse drug events in hospitalised older adults: protocol for a cross-sectional study
  1. Mitchell Nohner,
  2. Bryanna De Lima,
  3. Katie Drago
  1. General Internal Medicine and Geriatrics, Oregon Health & Science University, Portland, Oregon, USA
  1. Correspondence to Dr Katie Drago; drago{at}


Introduction Adverse drug events (ADEs) among hospitalised older adults are common yet often preventable. Efforts to recognise ADEs using pharmacist review and electronic health record adaptations have had mixed results. Our health system developed and implemented a geriatric prescribing context designed to offer age-friendly dose and frequency defaults for hospitalised patients 75 years and older. The impact of this context on ADEs remains unknown. To measure its impact, our team created a list of ADE-related International Classification of Diseases (ICD) codes specific to 10 commonly used medications at our institution. This protocol paper presents the process of designing a screening tool for ADEs, validating the tool with manual chart reviews and measuring the impact of the context on ADEs.

Methods and analysis This retrospective cross-sectional study will assess our list of ICD-10 codes against manual chart review to determine its accuracy. An electronic health record report for patients aged 75 years and older admitted to the hospital for a minimum of two nights was generated to identify 100 test positives and 100 test negatives. Test positives need at least one code from each level of our ICD-10 code list. The first level of codes identifies any possible ADEs while the second level is more symptom based. Test negatives must not have any code from the list. Two physicians blinded to test status will complete a structured chart review to determine if a patient had an ADE during their hospitalisation. Acceptable inter-rater reliability will need to be met before proceeding with independent chart review. Positive predictive value and negative predictive value will be calculated once all the chart reviews are completed.

Ethics and dissemination The Oregon Health & Science University Institutional Review Board approved this study (#21385). The results of the study will be disseminated in peer-reviewed journals and conference presentations.

  • Quality in health care
  • Adverse events

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See:

Statistics from

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.


  • A strength of this study is the modification of the Institute for Healthcare Improvement Trigger Tool to conduct confirmatory chart reviews for specific adverse drug events potentially caused by 10 commonly used medications among hospitalised older adults.

  • One strength is the adverse drug event list was carefully curated by two physicians to ensure common and high risk adverse events are captured.

  • A limitation is that the list of adverse drug events was tailored to the medications of interest, and will not identify all possible in-hospital adverse drug events for the older adult population.

  • The study design did not allow for estimating sensitivity and specificity or identifying all adverse drug events in the population.

  • As this is a single-site study, findings may not be generalisable to other populations.


Hospitalised older adults are at uniquely elevated risk of adverse drug events (ADEs) compared with their younger counterparts.1 One recent meta-analysis by Jennings et al calculated a pooled adverse drug reaction prevalence of 16% among patients 65 years and older in the hospital.2 Age-indiscriminate prescribing of medications is one, but not the only, root cause for these events. Efforts, however, to adapt the electronic health record (EHR) in order to reduce ADEs have had mixed results.3–8 Our health system developed and embedded the Geriatric Prescribing Context (GPC) into the EHR in July 2017.9 The GPC is an automated set of age-friendly default doses and frequencies triggered when prescribers write orders for patients 75 years and older admitted to the hospital or emergency department. There are no hard stops, alerts or extra steps prescribers have to take to write medication orders. The GPC offers age-sensitive defaults for 51 high-risk medications but does not impact orders from procedural units, the outpatient setting or from an order set. Initial investigation demonstrated 10%–30% improvement in the number of medication orders using the age-friendly defaults in the top 10 most commonly used inpatient medications adjusted by the GPC.9 Little is yet known about the effect of the GPC on ADEs as this is a novel adaptation.

The gold standard to identify ADEs is through chart review but this process is time-consuming and labour-intensive. International Classification of Diseases (ICD) codes documenting ADEs can be culled from the EHR easily and quickly and have been identified as a possible proxy for chart review. However, studies have found low sensitivity, specificity and predictive values for multiple different sets of ICD codes.10–13 Existing lists of ADE-related ICD codes are not specific to older adults, the medications most commonly prescribed to hospitalised older adults or the most frequently experienced dose-related side effects.


In an effort to understand the GPC’s impact on ADEs among hospitalised older adults, our team created a novel list of ADE-related ICD-10 codes for 10 commonly used medications in the hospital. This list needs to be validated with chart reviews before being used to assess the impact of the GPC on inpatient ADEs.

The aims of our study are to:

  1. Design an effective tool (ICD-10 codes) to highlight common ADEs related to 10 commonly used inpatient medications specific to older adults.

  2. Validate the tool against a manual chart review and calculate positive (PPV) and negative predictive values (NPV).

  3. Use the validated ICD-10 list of ADE-related codes to measure the GPC’s impact on ADEs, if any.

Methods and analysis

The conduct of our study will follow the Strengthening the Reporting of Observational Studies in Epidemiology14 guidelines and our reporting will follow the Standard Protocol Items: Recommendations for Interventional Trials.15

Study setting

This will be a retrospective cross-sectional study measuring the number of ADEs in the 12 months before and after the implementation of the GPC in July 2017. The study population will be from three Oregon Health & Science University (OHSU)-affiliated hospitals throughout Oregon. Two of the participating sites are community hospitals and one is a tertiary care, academic hospital.

Eligibility criteria

All patients 75 years or older admitted to one of the three participating hospitals prescribed 1 of 10 commonly used inpatient medications during their stay will be eligible for this study. The inclusion age of 75 years or older was chosen as that was the minimum age for the GPC to be active within the EHR. Patients between 65 and 74 years were thought to have more heterogeneous geriatric prescribing needs. The medications included in this study are the 10 most commonly prescribed medications of 51 total medications that had automatic dose adjustments from the GPC. They are primarily pain and psychiatry related as these are the most commonly prescribed inpatient medications for older adults. These medications include: acetaminophen, oral diphenhydramine, intravenous fentanyl, intravenous haloperidol, intravenous hydralazine, intravenous hydromorphone, oxycodone, intravenous prochlorperazine, quetiapine and trazodone. All study participants will need an inpatient stay of 2 nights or longer as the focus of this work is to assess inpatient ADEs only. While older adults are certainly at high risk of outpatient ADEs, this is not the focus of the protocol or the GPC. Patients in rehabilitation and inpatient psychiatry units will be excluded.


Two separate lists of ICD-10 codes were generated to capture ADEs for the selected medications. The first set of ICD-10 codes was aimed at the broad identification of ADEs. With guidance from inpatient coding experts, a list of 32 ICD-10 codes for poisoning and adverse effects related to the study drugs was created. These codes were more general in nature, for example, ‘Adverse effects of other opioids…’, to help identify any possible ADE. The second set of ICD-10 codes was more specifically symptom based. This list was refined from Hohl et al’s systematic review identifying codes for ADEs.16 Using Lexi-comp, a medical resident cross-referenced that list with the common adverse events for the study’s medications, only including the codes for ADEs with an incidence rate of 5% or more.17 For the rare but more serious ADEs, a group of geriatric inpatient providers identified specific ‘sentinel event’ ADEs for several of the medications, and the corresponding ICD-10 codes were added to the list above. For example, respiratory depression related to oxycodone may occur in less than 5% of patients receiving that medication, and as such would not have been included per the Lexi-comp review outlined above. However, given the higher risk of injury to the patient from this ADE, the ICD-10 code for such a reaction was included. Thus, a final list of 73 symptom-based ICD-10 codes was created that aimed to cover both common and high-risk sentinel event drug reactions. Through the combination of the broad ICD-10 code list and the symptom-triggered code list, the goal was to build an ICD-10 master list capable of identifying the majority of ADEs. The full list of ICD-10 codes can be found in the online supplemental appendix 1.


Our primary outcome is the presence or absence of an ADE. The Institute for Healthcare Improvement (IHI) Global Trigger Tool for Measuring Adverse Events was created to provide a structured method of conducting retrospective chart reviews to identify possible ADEs.18 This tool relies on triggers (laboratory values, medication administration, nursing notes, etc) to alert the reviewer that an ADE may have occurred and to instigate further investigation of such an event. The use of trigger tools focuses and expedites the chart review process. Trigger tool-based chart review has been evaluated in the past, and found to have poor sensitivity and very good specificity when compared with the gold-standard expert chart review.19 Starting with the IHI Global Trigger Tool, our team cross-referenced the standard triggers with our list of ADEs specific to the medications and population of interest.18 20 Of the initial 53 triggers, 37 were excluded due to irrelevance to this project (eg, triggers related to surgical, perinatal and emergency department ADEs). The remaining triggers were then bolstered with the addition of 24 new custom triggers to assist in capturing ADEs pertinent to the study’s medications (for full trigger tool, see online supplemental appendix 2). The two reviewers then employed this modified trigger tool to complete structured chart reviews. If a trigger was present, the reviewer then determined if an ADE truly occurred by assessing for patient harm (and if so, identifying the causative drug and stratifying the harm via an adapted National Coordinating Council for Medication Error Reporting and Prevention Index for Categorizing Error).21


The sample for validation was randomly selected from the EHR from patients admitted between July 2017 and August 2019. The sample for determining the GPC’s impact will count all ADEs in the 12 months before and after the GPC. These data will be aggregated to eliminate seasonal variability in ADEs related to those times of year when new providers tend to join in clinical care. The study will be conducted from September 2021 to June 2023. The creation and validation of the ICD-10 code list will take place between September 2021 and June 2022. The impact on the GPC will be assessed starting in June 2022 with the goal of completing the analysis by the end of June 2023.

Sample size

A research assistant calculated the sample size needed using PPV, NPV and a Shiny app.22 For 80% power and 80% minimally acceptable NPV, 57 test positives and 50 test negatives were needed. Our team decided to use 100 test positives and 100 test negatives to be conservative.

Data collection and management

Once we finalised the list of ICD-10 codes and the modified Trigger Tool, our next step was to validate the codes through manual chart review. A patient with an ICD-10 code from both levels in the online supplemental appendix 1 was considered a test positive and a patient with no ICD-10 code from either level was considered a test negative. The research assistant created two separate reports in the EHR to identify test positives and test negatives that met the eligibility criteria. Patients were randomised and the top 100 from each report were combined into one list. This list was randomised again to be divided up among two reviewers. Reviewers were blinded to ADE status. The gold standard for confirmation was chart review conducted by experienced physician experts in the care of older adults using the modified version of the IHI’s Trigger Tool. Reviewers documented the trigger codes, category of harm and responsible medication. Inter-rater reliability will need to be met before reviewers can proceed with independent review. A minimum Cohen’s kappa value of 0.60 is needed to ensure moderate agreement between reviewers.23

Statistical methods

PPV and NPV will be calculated at the completion of the chart reviews. PPV is calculated by dividing the true positives by the total number of test positives or those with at least one ICD-10 code from each level. NPV is calculated by dividing the true negatives by the total number of test negatives or those without any ICD-10 codes from the lists. The research assistant will match the test positive and test negative status based on ICD-10 codes with the ADEs identified through chart review. The goal will be to have at least 80% PPV and 80% NPV with our ICD-10 screening tool. If this is attained, our team will proceed to assess the impact of the GPC on ADEs. This process will use the report from the EHR with the chosen ICD-10 codes for 12 months before and after the GPC implementation. We will use control charts to track the data to assess any shifts over time. We will do a hypothesis test at the time of implementation to determine if there was a significant change. However, if we do not reach 80% PPV and 80% NPV, we will need to complete chart reviews to confirm ADEs prior to adding data to the control charts. If NPV is high but PPV is low, then we will need to complete chart reviews on presumed positives. If PPV is high but NPV is low, then we will complete chart reviews on presumed negatives.

Patient and public involvement

Neither patients nor the public will be involved in the design, conduct, reporting or dissemination plans of our research.


This study protocol provides a potential screening tool to identify ADEs related to the most commonly used inpatient medications in a highly vulnerable group—hospitalised older adults. Our team used existing ICD-10 code lists and the IHI Global Trigger Tool to develop a targeted screening tool to understand the impact of a single intervention, the GPC. While the GPC intervention is currently limited to our institution, we believe this screening tool could be widely used across health systems to quickly identify ADEs among older adults in the inpatient setting. ADEs among older adults are a significant driver for morbidity, mortality and increased healthcare-related costs. A demographic-specific, validated ICD-10 screening tool has potential value to others working to reduce the impact of ADEs among hospitalised older adults.

Previous studies using ICD codes alone to detect ADEs have shown suboptimal results with low sensitivity and specificity.10 12 The validation plan for this tool deliberately takes a conservative approach to sampling with the goal of maximising predictive values to detect ADEs but there remains a possibility that predictive values for this tool will also be low. In that event, the research team will add a second layer of confirmatory chart review for all positive screens. We believe that this alternative approach is still reasonable for use beyond this study as it limits the amount of manual chart review. The use of this demographically targeted and validated screening tool to identify ADEs could be an acceptable substitute to save time and effort for busy providers, even with the possibility of suboptimal predictive values.

Ethics and dissemination

The study protocol has been approved by the OHSU Institutional Review Board (#21385). Results will be disseminated via scientific journals and conference presentations, and individual data made available to the interested parties.

Ethics statements

Patient consent for publication


Supplementary materials

  • Supplementary Data

    This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.


  • Contributors KD and MN created the initial idea of the study in consultation with BDL. All authors designed the study protocol. BDL and KD prepared the initial draft and MN critically revised the manuscript. BDL, KD and MN contributed to the final draft. All authors read and approved the final manuscript.

  • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

  • Competing interests None declared.

  • Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.