The annual direct cost of injuries across the Australian health sector represents a significant public health and economic burden and is a major cause of years of life lost (YLL) and years of life lived with a disability (YLD).12 Injuries contribute substantially to morbidity and mortality in the Australian population, with community injury accounting for 5.4% of hospital discharges in 2003–043 and 6% of all causes of death in 2006.4 With a growing and aging population increasing demand for healthcare services, in addition to injuries being a leading cause of hospitalisation for children and young–middle aged adults, comes an urgent need for a strong evidence base to inform public health decisions and priority setting around the national health priority area of injury. Accurate and comprehensive information on injury causation and incidence, routinely sourced from hospital separations and mortality data, underpins burden of disease estimates and is imperative for injury research, policy and practice.25

Injury causation information is routinely collected in hospital records in Australia, with causes of injuries coded for all cases where a patient is admitted to hospital and treated for an injury. Cause of injury information which is documented in the medical record by clinical staff is translated by clinical coders into coded form using the International Statistical Classification of Diseases and Related Health Problems (ICD), which is the classification system used to describe morbidity and mortality in most countries throughout the world6; the 10th revision of ICD was published by the World Health Organization in 1992.7 A clinical modification of ICD-10 was developed in Australia, ICD-10-AM,8 which has been adopted by other countries such as New Zealand, Ireland, Germany, Romania, Slovenia and Saudi Arabia. The United States, along with a number of other countries, still uses the clinical modification of the previous ICD edition, ICD-9-CM, to code hospital data. Since its introduction in 1998, ICD-10-AM has been modified every two years by the National Centre for Classification in Health (NCCH) with the assistance of clinicians, clinical coders and health-related organisations to ensure the classification is current and appropriate for Australian clinical practice. This process of revision provides valuable periodic opportunities for users of the ICD-10-AM coded data to assess the extent to which the system provides them with necessary information, and to propose enhancements to the system. Two notable revisions to previous editions of ICD-10-AM include the enhanced coverage of sports-related injury and perpetrators of assault.

There are a number of factors which may reduce the accuracy of ICD coded external cause data.9 The quality and detail of documentation provided by clinicians in patient medical records directly affects the quality of coded data.10 Poor documentation within the source record from which the coding is being performed has been shown to decrease data quality by contributing to an overuse of non-specific “dump” codes.1112 Several studies examining the use of activity and place codes in Australia have found a high use of “dump” codes.1314 Detailed information regarding the causal and contributory factors of an injury is required to facilitate injury prevention research, and therefore the classification system used to code this data needs to provide a high level of detail (specificity).15 Criticisms of ICD-10 and earlier versions of the ICD coding system have asserted that external cause codes have a rigid structure that provides incomplete coverage or insufficient detail to identify certain important injury factors within hospital data.16

There has been very limited research examining agreement on the assignment of external cause of injury codes in hospital data, with all but one study conducted on the earlier version of the ICD (9-CM).111217^–20 A systematic review conducted by the current authors21 found that coding agreement using ICD-9-CM ranged from around 64% when examining exact code agreement,111219 to around 85% when examining agreement to the three character level.17 For ICD-10-AM coded data, agreement was between 71% for exact code agreement to around 74% when examining agreement to the 3-character level.20 Differences in agreement were evident when examining various external cause axes (ie, “intents”) and code blocks (ie, mechanisms); for example, 87% agreement for mechanism of injury codes,12 95% and 86% agreement respectively for coding of intent in two studies,1219 an average 70% agreement in the coding of falls,1217 and between 63% and 81% agreement in code assignment for motor vehicle traffic crashes.1217

The aim of this study is to assess the extent of coder agreement in assignment of external cause of injury codes using ICD-10-AM, for injury-related hospitalisations in public hospitals in Australia.

METHODS

A retrospective on-site medical record review and recoding methodology was used to examine the extent of coder agreement for external cause of injury codes for injury-related hospital admissions.

Study sample

Sample selection followed a two stage procedure designed to ensure a dispersion of cases across locality and hospital caseloads (ie, the number of hospital discharges annually with a principal diagnosis range: S00–T79), while approximating a probability based result. The sample size for hospitals and cases was determined by a number of factors, including budget, resources and statistical power considerations. The record review and subsequent statistical analyses were multifactorial, thereby increasing the required sample size. Four state health departments (Queensland, New South Wales, Victoria and South Australia) agreed to be industry partners for this research study, so due to limited resources, data collection was restricted to these four states who contributed to the project.

Selection of hospitals

This study included a stratified random sample of 50 hospitals across four states in Australia (Queensland, New South Wales, Victoria and South Australia. All public hospitals within the four states were stratified by locality (urban, regional, remote) and injury caseload (large, >2500 injury cases/year; medium, 1000–2499 injury cases/year; small, 200–1000 injury cases/year; or very small, <200 injury cases/year) (note: private hospitals were excluded from this study as they have different admission practices, have a different case mix, and treat significantly fewer injuries per annum than are treated in public hospitals). In 2002–03 a little over 400 000 injuries were treated across 450 hospitals in the four states. Of these, 220 hospitals were excluded from the stratification process as they reported very small injury counts (<200 injuries/y), collectively amounting to less than 5% of the total injury caseload. In addition, four hospitals in remote locations reporting >200 injuries/y were excluded from the sampling process due to resource constraints. The final sampling frame resulted in 226 hospitals available for selection. To ensure that hospitals contributing the most to national hospitalisation injury data estimates were included in the sample, a sampling fraction was employed. Using a target sample size of 50 hospitals, the number of hospitals required within each stratum was calculated according to the proportional contribution of the strata to the total injury count across the four states. The resulting sampling fractions were: 0.66 for large hospitals; 0.30 for medium hospitals; and 0.03 for small. A random sample of each stratum was obtained using these weights.

Selection of cases

A random sample of cases was obtained from all hospital discharges between 1 July 2002 and 30 June 2004 with a principal diagnosis of injury (ICD-10-AM code range S00–T79), and at least one assigned external cause code (V01–Y98), from within the selected hospitals. For medium and large injury caseload hospitals 100 cases were randomly selected, and for small injury caseload hospitals 50 cases were randomly selected from each hospital, resulting in a final sample of 4850 cases. Patients in Australian hospitals do not have unique person identifiers, thus it is not possible to assess whether a patient was selected twice in the sample. However, it is very unlikely that any duplicate cases exist in the dataset as there were no instances where the auditor used the same person’s medical record for two separate discharge events within the same hospital.

Training of auditors

A data collection manual was prepared by the lead auditor (GW) in conjunction with all chief investigators which specified the aims of the project, process for data collection and items to be collected. Two other auditors participated in the study and were trained on-site by the lead auditor. The lead auditor collected data in all four states, while the two additional auditors only collected data in New South Wales and Victoria. At the commencement of the training process, both the lead auditor and the auditor being trained dual-coded approximately 20 records and compared their results to ensure they were collecting and coding information consistently in accordance with the data collection manual instructions.

Data collection process

Lists of medical record numbers for selected cases were provided to the health records departments in each hospital, and the medical records were made available to the auditor for on-site review. Data collection forms containing a unique project ID, with sections for the auditor to record their information on the front of the sheet, were provided; the original coded data was printed on the back of the sheet to ensure the original codes were not visible to the auditor while recoding the records. Using the same medical record available to the original coder, the auditor attended each hospital site and first recorded text descriptions of the injury circumstances recorded in the record. If resources were available to do so, health information managers at each hospital blanked out any coded data that was printed in the medical record. If resources did not permit this, the auditor blanked out the codes on all medical records prior to commencing the review process, to ensure codes were not visible while recoding the record. Information was extracted from all medical record documentation sources including ambulance reports, emergency department notes, clinical notes, progress notes, discharge summaries, specialist reports, allied health reports, etc. The detailed text descriptions of the injury event and circumstances were recorded for each documentation source identifying the source to which the text description belonged.

After completing the text descriptions, the auditor coded the record using ICD-10-AM, 3rd edition, and recorded the new codes on the front sheet of the data collection form. Once the front sheet was completed, the auditor consulted the second sheet of the data collection form (where the original codes were recorded) to compare new and original codes. Where the new and original codes differed, the auditor was responsible for providing comment as to their rationale for code selection in comparison to the codes assigned by the original coder. The process for the review (ie, recoding, comparison and comment on code differences) was based on the Australian Coding Benchmark Audit (ACBA) procedure, a coding audit method that involves recoding a sample of hospital-admitted patient episodes, comparing the results and reflecting on the reason for differences. The ACBA, as developed by NCCH, is utilised in several localities and hospital settings across Australia; it was modified to address only external cause codes.

Data analysis

Codes from the original dataset were compared to the recoded data to identify the level of coder agreement for external cause coding. In contrast to previous studies, the authors have deliberately avoided using the terms “correct” and “incorrect” coding or “accuracy of coding” as disagreement in coding between the original coder, and the auditor could reflect errors on the part of either the original coder or the auditor. To enable comparison of findings from this study to findings from previous studies, similar categories of codes were used with slightly modified terminology to describe “agreement” rather than “correctness”. Code agreement levels were grouped into the following distinct categories, and comparisons made across major intent, mechanism and activity blocks:

• Disagreement at the 3-character level.

• Agreement at the 3-character level, where 4th character level disagreement.

• Agreement at the 4-character level, where 5th character level disagreement.

• Complete code agreement.

Place of occurrence codes are all the same at the 3-character level (all place codes commence with the characters “Y92”), thus were grouped into the following distinct categories and compared across major place categories:

• Agreement at the 4-character level, where 5th character level disagreement.

• Complete code agreement.

As the external cause chapter has varying degrees of specificity of codes across the chapter, differences at a 3-character, 4-character and 5-character level have different implications depending on which broad block they belong to. Table 1 provides a description of the information contained in the 3rd, 4th and 5th character levels for each major ICD-10-AM external cause block, activity block and place block, to assist in the interpretation of results.

RESULTS

Of the 4850 records selected for review from 50 hospitals, 4373 cases were able to be retrieved, accounting for 90% of the original sample. Across hospitals, the median number of missing cases was 4 (range 0–51). The few hospitals where a large number of cases were unable to be reviewed were large facilities that experienced significant difficulties providing resources to retrieve the records as they were largely stored off-site in secondary storage facilities. Most hospitals (n = 36) were able to provide 90% of requested records.

Coder agreement for external cause codes

Examining the level of code agreement for external cause codes, it was found that overall:

• Disagreement at the 3-character level was 26.02% (95% CI 25.96% to 26.08%; n = 1137).

• Agreement at the 3-character level where there was disagreement at the 4th character level was 3.39% (95% CI 3.33% to 3.45%; n = 148).

• Agreement at the 4-character level where there was disagreement at the 5th character level was 2.95% (95% CI 2.89% to 3.01%; n = 129).

• Agreement for the complete external cause code was 67.61% (95% CI 67.55% to 67.67%; n = 2954).

Table 2 shows the code concordance for major external cause categories. At a broad block level there was a high degree of concordance (90%) in terms of the broad mechanism that codes belonged to (transport events, falls, etc) for most mechanisms.

Table 3 shows the levels of agreement within major external cause intent and mechanism blocks (categorised from the original external cause code). Agreement levels vary across intent blocks, with coders most likely to disagree at the 3-character level in the assignment of undetermined intent codes, medical and surgical complication codes, and sequelae codes. With regard to the assault code block, 14% of cases agreed at the 4-character level (type of assault) but disagreed at the 5-character level (perpetrator of the assault). Agreement levels also differed across mechanism code block, with complete code agreement as high as 100% for firearm-related injuries (note: n = 3 cases) and injuries due to hot or cold conditions, but as low as 60% for other threats to breathing codes.

Coder agreement for activity codes

Examining the level of code agreement for activity codes, it was found that:

• Disagreement at the 3-character level was 7.33% (95% CI 7.29% to 7.37%; n = 317).

• Agreement at the 3-character level where there was disagreement at the 4th character level was 21.91% (95% CI 21.87% to 21.95%; n = 947).

• Agreement at the 4-character level where there was disagreement at the 5th character level was 2.66% (95% CI 2.62% to 2.70%; n = 115).

• Agreement for the complete activity code was 68.09% (95% CI 68.05% to 68.13%; n = 2943).

Table 4 shows the code concordance for major activity categories. At a broad block level there was moderate to high concordance in terms of which broad activities the codes belonged to (sports, working, etc). The original coder and the auditor agreed the patient was involved in a sports/leisure activity when sustaining the injury in around 83% of cases, and over 90% of the time the coders agreed that the person was working for an income at the time of the injury. The original coder and auditor only agreed 48% of the time that the patient was doing an “other specified activity” at the time of the injury.

Table 5 shows the levels of agreement within major activity code blocks (categorised from the original activity code). Agreement levels vary across activity blocks, with coders most likely to disagree at the 3-character level in the assignment of sports and leisure codes (25.6% disagreement). Coders were most likely to agree where the activity was coded as “unspecified activity” (81.3% agreement).

Coder agreement for place of occurrence codes

Examining the level of code agreement for place codes, it was found that:

• Percentage disagreement at the 4-character level was 22.04% (95% CI 21.98% to 22.10%; n = 963).

• Percentage agreement at the 4-character level where there was disagreement at the 5th character level was 2.49% (95% CI 2.43% to 2.55%; n = 109).

• Agreement for the complete place code was 75.46% (95% CI 75.40% to 75.52%; n = 3297).

Table 6 shows the code concordance for major place categories. At a broad block level there was moderate to high concordance in terms of which broad place the codes belonged to (home, residential institution, etc). The original coder and the auditor agreed that the patient was at home at the time of the injury in around 75% of cases, and over 70% of the time the coders agreed that the person was at school/other institution/public administrative area at the time of injury. The original coder and auditor only agreed 61% of the time that the patient was at a sports area at the time of injury.

Table 7 shows the levels of agreement within major place code blocks (categorised from the original place code). Agreement levels vary across place blocks. Coders were most likely to disagree at the 4-character level in the assignment of place codes for sports area and other specified places, with almost 39% and 34% disagreement, respectively, in these blocks. Coders were most likely to agree where the place was a residential institution (92.2%) or a street or highway (83.5%).

DISCUSSION

This study, identifying the level of coder agreement in the assignment of external cause codes for injury-related hospitalisations, reported 68% agreement for the complete external cause code, 68% for activity codes and 75% for place of occurrence codes. These levels of agreement were found to vary depending on the category to which the code belonged. This study found similar levels of agreement to those found by previous studies using ICD-9-CM, with average level of complete code agreement reported in previous studies of around 64%.111219 At a 3-character level ICD-10-AM code agreement for external causes rose to 74%, slightly less than previously reported code agreement at the 3-character level of 85% using ICD-9-CM.17

This research represents one of the first reported studies examining coding agreement in the assignment of external cause codes using ICD-10-AM, with all but one prior study conducted using ICD-9-CM.111217^–20 With this earlier version of the classification largely replaced by ICD-10 in many countries throughout the world, and ICD-10-AM in use in 11 countries worldwide and evaluated for use in an additional 16 countries, this study provides important data to assess the validity of ICD-10-AM external cause coded data. This study has the largest reported sample size of all external cause recoding studies, with over 4300 cases from 50 hospitals across several states in Australia; the largest previous study reported a sample size of 1670 cases from 52 hospitals.17 As ICD-10-AM includes separate code blocks for both activity at the time of the injury and place of occurrence, this study also provides the first detailed figures regarding the level of agreement of coders in the assignment of activity and place codes.

There were however some limitations to the study that need to be considered when interpreting the results. First, while the auditors were instructed not to look at the original coded data when recoding the medical record, as the data were provided on the back of the data collection sheet the potential for the auditor to glance at the original codes was present. However, given the multicentre nature of the data collection and large amount of paperwork and coordination of datasheets to match to medical records, it was simply not feasible to provide the original codes on separate sheets of paper. Additionally, as three auditors collected and coded data for this study, there may have been inconsistencies between auditors in the assignment of codes. However, as coding is generally performed by multiple coders in a large number of hospitals across Australia, and as the auditors’ codes were not considered to be the “gold standard” for the purposes stated above, any fluctuations between the auditors in their external cause coding reflect normal variability in the coding process. Finally, to ensure that hospitals which contributed the most to the national hospitalisation injury data estimates were included in the sample, a sampling fraction and exclusion process was employed and as such, the results of this study can not be generalised to excluded hospitals. It is possible that the coding in such hospitals may be less accurate than in hospitals where coders are exposed to injury cases more regularly, hence the extent of coder agreement on a national level may be poorer than estimated in the current study.

To understand the reason for coder difference, it is important to reflect on the documentation and coding process. First, a range of personnel are responsible for documenting information in a patient’s medical record and while injury causation information is important to injury researchers, clinicians may not consider it a vital aspect to the clinical care of the patient and hence fail to document this information accurately. Divergent information may be elicited from the multiple documentation sources by different coders due to conflicting or incomplete documentation provided by individual clinical staff. Coders may differentially prioritise key cause of injury information by focusing on particular elements to the exclusion of other important elements causing the injury. ICD-10-AM index terms often do not exactly describe the cause of injury element for which the coder is searching, and coders may have different “mental catalogues” of appropriate index terms to search for, leading them to different index terms and ultimately different codes. Also, use of different coding tools (encoder/coding books) may influence the pathway to the ICD index. Coder judgement in the face of ambiguity or deficiencies, in either the information provided or the classification system, may create divergence in final decisions.

This study has implications for injury researchers and policy makers who use external cause coded hospital data to examine trends and patterns in injury causation. Given that the results show only 68% agreement for complete external cause code and 74% agreement for the 3-character code block level, as well as variability in agreement levels across different code blocks, place and activity codes, researchers need to be aware of the reliability of their specific data of interest when they wish to undertake trend analyses or case selection for specific causes of interest. Studies such as the present research enable identification of those categories where inter-rater agreement on external causes is high as well as those categories where there is relatively poor agreement, to provide some indication of the reliability of data for specific purposes.

To date classification developers, hospital management and health departments have placed most emphasis on developing standards, guidelines, and quality assurance programmes around the accuracy of diagnosis coding, with case mix funding/clinical care research/quality of care indicators all reliant on diagnostic codes. There has been very limited research examining and validating the quality of external cause coded data, a lack of definitions and standards around the assignment of external cause codes, and a corresponding limited appreciation of the potential value of these data for injury surveillance and prevention activities. With the ICD scheduled for a major revision over the next five years, and one of the main areas of work targeted being the injury and external cause of injury chapters of the ICD, research and development work around external cause coding and classification will provide a valuable evidence base to inform changes and improvements to the classification in the future.