Article Text
PDF
Abstract
Objectives Administrative data are increasingly being used for surveillance and monitoring of mental health and substance use disorders (MHSUD) across Canada. However, the validity of the diagnostic codes specific to MHSUD is unknown in emergency departments (EDs). Our objective was to determine the concordance, and individual-level and hospital-level factors associated with concordance, between diagnosis codes assigned in ED and at discharge from hospital for MHSUD-related conditions.
Design Population-based retrospective cohort study.
Setting EDs and hospitals within Vancouver Coastal Health Authority (VCH), British Columbia, Canada.
Participants 16 926 individuals who were admitted into a VCH hospital following an ED visit from 1 April 2009 to 31 March 2017, contributing to 48 116 pairs of ED and hospital discharge diagnoses.
Primary and secondary outcome measures We examined concordance in identifying MHSUD between the primary discharge diagnosis codes based on the International Statistical Classification of Diseases, 9th and 10th Revisions (Canada) assigned in the ED and those assigned in the hospital among all ED visits resulting in a hospital admission. We calculated the percent overall agreement, positive agreement, negative agreement and Cohen’s kappa coefficient. We performed multiple regression analyses to identify factors independently associated with discordance.
Results We found a high level of concordance for broad categories of MH conditions (overall agreement=0.89, positive agreement=0.74 and kappa=0.67), and a fair level of concordance for SUDs (overall agreement=0.89, positive agreement=0.31 and kappa=0.27). SUDs were less likely to be indicated as the primary cause in ED as opposed to in hospital (3.8% vs 11.7%). In multiple regression analyses, ED visits occurring during holidays, weekends and overnight (21:00–8:59 hours) were associated with increased odds of discordance in identifying MH conditions (adjusted OR 1.47, 95% CI 1.11 to 1.93; 1.27, 95% CI 1.16 to 1.40; 1.30, 95% CI 1.19 to 1.42, respectively).
Conclusions ED data could be used to improve surveillance and monitoring of MHSUD. Future efforts are needed to improve screening for individuals with MHSUD and subsequently connect them to treatment and follow-up care.
- mental health
- substance use
- surveillance and monitoring
- diagnostic codes
- validation study
- acute health care
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
- mental health
- substance use
- surveillance and monitoring
- diagnostic codes
- validation study
- acute health care
Strengths and limitations of this study
Concordance between emergency department (ED) and hospital diagnostic codes was assessed within a population-based cohort identified from the linkage of two comprehensive health administrative datasets of acute care.
Among the few studies that examine the validity of diagnostic codes used for mental health and substance use disorders in acute care, informing their use in surveillance and monitoring.
Analysis was limited to ED visits admitted to hospital and may not be representative of diagnostic accuracy of ED visits discharged directly from EDs.
Data capture was limited to primary diagnostic codes assigned in EDs; thus, improved detection of substance use disorders is possible in settings where more diagnostic codes are available.
Introduction
Mental health conditions and substance use disorders (SUDs) are the leading cause of the global burden of diseases, posing substantial health and economic impacts on individuals and society.1 Individuals with a mental health and/or SUD are at increased risk of emergency department (ED) use and hospitalisation,2 and often access the ED as first contact point for medical care.3 While health administrative data are increasingly being used for surveillance and performance monitoring across Canada,3–6 ED data are often not included in these efforts,4 6 partially due to the lack of consistency in ED data collection and reporting across the country.7
Vancouver Coastal Health (VCH) is one of BC’s five regional health authorities, serving approximately one million residents. This includes the Downtown Eastside (DTES) neighbourhood, one which has historically featured a high prevalence of substance use, mental illness, infectious diseases and homelessness.8 In 2015, prior to the declaration of a public health emergency in opioid overdose, VCH made integrated, comprehensive care delivery among the cornerstones of its DTES Second Generation Strategy (DTES-2GS) to acknowledge the unique needs of this population, which often presents to care with multiple concurrent disorders and social problems.9 Reducing SUD-related acute care visits is a central aim in both the overdose response and the DTES-2GS initiative. Thus, diagnostic information collected in these visits is of central importance to surveillance, monitoring and evaluation.10 11
While diagnostic codes assigned in hospital are subject to nationally coordinated cleaning and validation,12 13 the validity of the diagnostic codes specific to mental health and SUD in the ED are unknown. One systematic review identified 39 studies examining the accuracy of mental health diagnoses in administrative data, yet none of these studies assessed diagnostic accuracy in EDs.14 Our objectives were to determine the concordance, and individual-level and hospital-level factors associated with concordance, between the International Statistical Classification of Diseases and Related Health Problems, 9th and 10th Revisions, Canada (ICD-9-CA and ICD-10-CA) diagnosis codes assigned in the ED and at discharge from hospital for any mental health and substance use-related conditions observed in VCH between 2007 and 2017.
Methods
Study population and data sources
We obtained our data from a cohort of individuals residing in or having a record of receiving community-based services in Vancouver’s DTES neighbourhood between 1 April 2009 and 31 March 2017. The cohort was defined using the health administrative databases held by VCH, including the CommunityMart database (capturing community-based health service referrals), the EDMart database (capturing ED visits) and the Discharge Abstract Database (DAD) (capturing hospitalisations). The DAD contains data on inpatient acute care, day care and rehabilitation care at 11 hospitals under the purview of VCH, and the EDMart contains data on ED visits at 11 hospitals and acute care clinics under the purview of VCH. Available detail on each ED is included in online supplementary appendix 1.
Supplemental material
Data linkage was performed by the VCH data stewards, based on unique personal health numbers recorded in each database. We extracted data on individuals with at least one ED or hospital record during study follow-up to identify any indication of mental health conditions or SUD. All records of ED visits resulting in hospital admission were used to assess concordance in diagnostic codes within the two databases. Linkage of ED visits to the resulting hospital admission was ascertained, based on the following data fields and criteria: (if an ED visit had the same ‘ContinuumID’ as the hospital admission OR if the hospital admission occurred within 1 day of the ED visit) AND (the ED visit had a flag indicating admission to an hospital OR the hospital admission indicated an entry code of ‘E’ (ED)). The ‘ContinuumID’ was a unique ID used within a VCH facility to track patient movement across different health systems at each visit, and therefore might not capture the ED visit that resulted in admission at another hospital. Therefore, we supplemented this by capturing hospital admission within the same day of ED visit. These linkages were further ascertained using available data fields in EDMart (‘AdmittedFlag’) and in DAD (‘Entrycode’) to confirm ED visits, which resulted in hospital admission. While the ED data contain records since April 2007, only those collected after 1 April 2012 were subject to national-level data quality control.15 16 In contrast, hospitalisation data are standardised nationally with VCH DAD data dated back to 1 April 2007, with key fields (including diagnosis codes) subjected to error checks and validation.12
Measures
We considered the primary diagnosis field in ED (coded using ICD-10-CA from 1 April 2012 onward, and a combination of ICD-9-CA and ICD-10-CA prior to this date), and any of the up to 25 primary and secondary diagnosis fields (coded by ICD-10-CA exclusively) in hospital. We defined mental health conditions and SUD using the ICD-9-CA from 290 to 319, or their ICD-10-CA equivalents from F00–F99, consistent with the Canadian surveillance system.4 6 In addition, we considered more specific codes to identify alcohol use disorder (AUD), opioid use disorder (OUD), other SUD, mood disorders, depression, psychoses, neurotic/stress/somatoform disorders, anxiety disorders and personality disorders (specific codes provided in online supplementary appendix 1).
We defined an a priori list of patient-related factors and ED visit-related factors, which might be associated with diagnostic discordance as informed by the literature17 and given data availability. This included gender (male, female), age (<25, 25–34, 35–44, 45–54, 55–64 and ≥65), homelessness status (yes, no), event urgency using the Canadian Triage Acuity Scale,18 length of ED visit (<6 hours, ≥6 hours; defined as the difference between date/time of triage or registration in ED and date/time the patient left the nursing unit), and the time (9:00–20:59 hours and 21:00–8:59 hours) and day (weekday, weekend and holiday) of ED visit. Additionally, we considered the fiscal year of ED visit (2007–2011, 2012–2014 and 2015–2017) and a hospital indicator variable to examine the potential differences in diagnostic discordance over time, and across hospitals, respectively. Finally, we considered whether the patient ever resided in the DTES (yes, no) given the features of high disease prevalence and high vulnerability of the DTES neighbourhood.8
Statistical analysis
We examined the proportion of acute care visits with an indication of mental health conditions or SUD identified by hospital records only, by both ED and hospital records, and by ED records only. Among all ED visits resulting in a hospital admission, we examined the concordance between the primary cause assigned in ED and in hospital. We calculated the per cent overall agreement, positive agreement, negative agreement and the Cohen’s kappa coefficient—a statistic which measures agreement for categorical items, taking into account the possibility of the agreement occurring by chance of ED–hospital discharge diagnosis pairs.19 Landis and Koch’s classifications were used to interpret the kappa statistics produced: 0 as no agreement, 0.01–0.2 as slight, 0.21–0.40 as fair, 0.41–0.60 as moderate, 0.61–0.80 as high and 0.81–1.00 as almost perfect agreement.20 In addition, treating hospital discharge diagnosis as the reference standard, we calculated sensitivity, specificity, positive predictive value and negative predictive value of the primary cause of visit assigned in an ED. We repeated these analyses for each specific type of mental health and substance use disorder (MHSUD). Further, we examined the concordance between the primary cause assigned in ED against any cause assigned in hospital.
Finally, among ED visits transferred to hospital with at least one diagnosis code in either hospital or ED indicating an MHSUD, we performed a multiple regression analysis to identify patient-related factors and ED visit-related factors independently associated with discordance. We used a generalised linear mixed effects model with a logit link and binomial distribution to account for intra-individual correlation between repeated acute care visits.21 We repeated the analysis for any SUDs and any mental health conditions. All analyses were performed using SAS statistical software V.9.4.
Patients and public involvement
Patients and the public were not involved in the design, conduct and reporting of this research.
Results
Among 56 875 DTES cohort participants, 43 017 (75.6%) had at least one ED or hospital record between 1 April 2007 and 31 March 2017, and were included in our analyses. Among them, 16 996 (39.5%) had an indication of a MHSUD (table 1). It is notable that ED data increased the attribution of MHSUDs within the study cohort by 25.5% compared with using hospital records only. The proportions of missed case identification without ED records were comparable for AUD, OUD and other SUDs (21.9%–27.8%), and higher for most mental health conditions (27.9%–65.3%), except for personality disorders (8.8%) (table 1).
Identification of mental health and substance use disorders using hospital (any diagnosis codes) and emergency department (ED) records among DTES cohort participants (n=56 875)*
A total of 16 926 individuals were admitted into a VCH hospital following an ED visit, contributing to 48 116 pairs of ED and hospital discharge diagnoses (table 2), and 15.4% of ED visits resulted in hospitalisation during the study period. We found a high level of overall agreement between ED and hospital primary diagnoses in classifying whether a visit was related to a mental health condition or SUD (overall agreement=0.89, positive agreement=0.82 and kappa=0.74). Compared with the primary diagnosis code assigned at hospital discharge, the primary diagnosis code assigned at ED discharge was less likely to classify a visit as SUD-related (11.8% vs 3.8%), resulting in a fair level of agreement between ED and hospital in identifying any SUD-related visits (positive agreement=0.31, kappa=0.27). In contrast, ED and hospital classified a more comparable proportion of visits as mental health-related (21.7% vs 22.0%) with a high level of agreement (overall agreement=0.89, positive agreement=0.74 and kappa=0.67).
Concordance between the primary diagnosis code assigned in the ED and the primary diagnosis code assigned at hospital discharge following an ED visit for any mental health conditions or SUDs (n=16 926, 48 116 pairs)
The primary diagnoses in ED had consistently high specificity (0.93–1.00) in classifying mental health and SUD. However, sensitivity varied widely (0.09–0.78). The positive predictive values ranged from 0.41 to 0.67 for SUD and 0.16 to 0.67 for mental health conditions (table 2). These values increased to 0.80–0.94 and 0.30–0.75, respectively, if any of the up to 25 diagnosis codes in the hospital were considered (see online supplementary appendix 2 for further details).
In a multiple regression analysis, we found younger age and shorter length of ED visit were associated with increased odds of discordance between ED and hospital settings in identifying SUDs (table 3). In contrast, they were associated with decreased odds of discordance in identifying mental health conditions. Otherwise, ED visits occurring during holidays, weekends and overnight (21:00–8:59 hours) were associated with increased odds of discordance in identifying mental health conditions (adjusted OR (aOR) 1.47, 95% CI 1.11 to 1.93; 1.27, 95% CI 1.16 to 1.40 and 1.30, 95% CI 1.19 to 1.42, respectively). However, we did not observe such associations for discordance in identifying SUDs. Lastly, ED visits occurring in more recent years were associated with reduced odds of discordance in identifying SUDs compared with visits prior to 1 April 2012 (aOR, 0.53, 95% CI 0.44 to 0.63; 0.74, 95% CI 0.60 to 0.91; respectively, for fiscal years 2012–2014 and 2015–2017).
Factors associated with discordance between the primary diagnosis code assigned in the emergency department (ED) and the primary diagnosis code assigned at hospital discharge following an ED visit*
Discussion
Using hospital and ED administrative records from VCH, we found a high proportion (39.5%) of individuals with an indication of a mental health or SUD, 25.5% of which would not have been identified without the use of ED data. We found a high level of overall agreement between ED and hospital in classifying whether a visit was related to any MHSUD. The concordance was higher for determining any mental health-related visit as opposed to SUDs. An individuals’ age and several ED visit-specific factors were found to be independently associated with the concordance between ED and hospital, with the direction of associations differing in identifying mental health conditions and SUDs.
It is important to emphasise the fact that 25.5% of individuals with mental health or SUD could not have been identified using only hospital records. The value of these data for disease surveillance and the monitoring and evaluation of changes in policy and practice are clear. However, it is notable that SUD overall, and illicit drug use, in particular, were largely underdetected in EDs as compared with hospitals. While there are many competing priorities in ED settings, improving screening for SUDs can help connect individuals to treatment and follow-up care and reduce the likelihood of subsequent readmission.22
Otherwise, we found a comparable proportion of contacts in EDs and hospitals, which were identified to be mental health-related. This was consistent with a systematic review of mental health diagnoses accuracy in administrative data that showed comparable accuracy for diagnoses made in inpatient settings compared with other settings.14 Moreover, we found a high level of agreement between hospital and ED diagnoses in determining any mental health conditions in general, with a high concordance for psychoses, moderate concordance for mood disorders, but low concordance for anxiety disorders and personality disorders. Our findings were consistent with a systematic review that concluded administrative data were generally predictive of true diagnosis of psychotic categories, but were less satisfactory in identifying anxiety disorders.14
Given the inherent complexity in diagnosing mental health conditions, it is not surprising that there is inadequate reliability in the diagnosis of specific types of mental health conditions. Diagnostic accuracy is further limited in ED given only one primary diagnosis code was available. Notably, we found ED visits occurring during holidays and weekends, as well as those during overnight shifts were associated with higher odds of mental health diagnostic discordance between ED and hospital. Various studies have investigated the ‘weekend effect’ in healthcare, suggesting poorer health outcomes among individuals admitted to hospitals during the weekend as opposed to weekdays. Two Canadian studies of ED admission found weekend admissions were associated with significantly higher in-hospital mortality rates.23 24 To our knowledge, this study is the first to identify an independent association between ED visit timing and mental health diagnosis accuracy. Several factors might explain these associations, including the intensity of care and medical staff, the volume of ED visits at different times of the day and week, as well as potential impairment of physical and cognitive abilities of medical staff due to sleep deprivation especially during overnight shifts.25 26 Some adjustments to staffing models at the ED might be made to improve diagnosis accuracy. Future studies should further investigate whether increased diagnostic discordance associated with visit timing will result in adverse patient outcomes.
Several limitations are worth noting. First, we assessed the validity of the ED diagnostic codes in a limited and selective subset of all ED visits; those subsequently admitted to hospital no doubt had symptoms of higher severity, and such cases may not be representative of diagnostic accuracy of ED visits discharged directly from ED. Further, diagnostic codes assigned in hospital are not an ideal ‘gold standard’ as they have been reported to have a high specificity but moderate sensitivity in identifying mental health conditions and SUD.27 Nonetheless, in the absence of an external ‘gold standard’, an analysis of concordance between ED and hospital diagnosis codes can help illuminate data quality.28 Third, only primary diagnosis codes were available in VCH ED records. In settings where more diagnosis codes are available, there may be improved detection of SUDs. Fourth, we did not consider diagnosis codes related to intentional self-harm, which was a common reason for hospitalisation due to a mental health disorder. However, among the ED visits that resulted in hospitalisation, only 0.24% had a hospital diagnosis code indication of intentional self-harm. Furthermore, our sensitivity analyses which used all (up to 25) diagnoses codes in hospital captured 90% of the self-harm-related hospital admissions as those admissions are likely to have a diagnosis code between 290 and 319 (or F00–F99) in addition to the self-harm diagnoses codes. Finally, caution should be exercised while generalising our findings to other settings, noting that our study population represents a highly socioeconomically disadvantaged patient population with a high prevalence of MHSUD. The study team had limited access to information on the clinical context of EDs, such as the availability of psychiatrists and other specialised services, which may help to explain variation in concordance across settings and may limit generalisability to health administrative data used for these purposes in other jurisdictions.
In conclusion, we found a high level of diagnostic concordance between ED and hospital for broad categories of mental health conditions, and a fair level of concordance for SUD. A large proportion of individuals with an indication of a mental health condition or SUD would be missed without ED records. Future efforts are needed to improve screening for individuals with MHSUDs and connect them to treatment and follow-up care.
Acknowledgments
We thank the Vancouver Coastal Health Authority staff involved in data access and procurement, including: Michael Li, Joleen Wright, Christopher Mah, Janine Johnstone, Susan Sirett and Kacy Fehr.
References
Footnotes
Contributors LW, DP and BN designed the study, conceptualised the analysis and wrote the first draft of the manuscript. FH and LW performed data cleaning and conducted the analyses. FH, LAP and DP provided critical input on and revised the manuscript. BN and CM are principal investigators of the parent study. RM and RB provided input on the manuscript on behalf of VCH. All authors read and approved the final manuscript.
Funding This work was supported by the Vancouver Coastal Health Authority and the Canadian Institutes of Health Research grant number GIR-145128.
Competing interests None declared.
Ethics approval Ethical approval was obtained from the Providence Health Care Research Institute (H16-00516) and the Simon Fraser University Office of Research Ethics (2016s0670).
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement Data are available on reasonable request.
Patient consent for publication Not required.