Article Text
PDF
Abstract
Objectives To assess the accuracy of International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes in identifying subjects with melanoma.
Design A diagnostic accuracy study comparing melanoma ICD-9-CM codes (index test) with medical chart (reference standard). Case ascertainment was based on neoplastic lesion of the skin and a histological diagnosis from a primary or metastatic site positive for melanoma.
Setting Administrative databases from Umbria Region, Azienda Sanitaria Locale (ASL) Napoli 3 Sud (NA) and Friuli Venezia Giulia (FVG) Region.
Participants 112, 130 and 130 cases (subjects with melanoma) were randomly selected from Umbria, NA and FVG, respectively; 94 non-cases (subjects without melanoma) were randomly selected from each unit.
Outcome measures Sensitivity and specificity for ICD-9-CM code 172.x located in primary position.
Results The most common melanoma subtype was malignant melanoma of skin of trunk, except scrotum (ICD-9-CM code: 172.5), followed by malignant melanoma of skin of lower limb, including hip (ICD-9-CM code: 172.7). The mean age of the patients ranged from 60 to 61 years. Most of the diagnoses were performed in surgical departments.
The sensitivities were 100% (95% CI 96% to 100%) for Umbria, 99% (95% CI 94% to 100%) for NA and 98% (95% CI 93% to 100%) for FVG. The specificities were 88% (95% CI 80% to 93%) for Umbria, 77% (95% CI 69% to 85%) for NA and 79% (95% CI 71% to 86%) for FVG.
Conclusions The case definition for melanoma based on clinical or instrumental diagnosis, confirmed by histological examination, showed excellent sensitivities and good specificities in the three operative units. Administrative databases from the three operative units can be used for epidemiological and outcome research of melanoma.
- sensitivity and specificity
- administrative database
- melanoma
- Icd-9-cm
- validity
- accuracy
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Strengths and limitations of this study
This study is the first that evaluated the accuracy of the International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes for melanoma in three large computerised Italian administrative databases using the same case definition for melanoma.
The strength of this study includes the use of medical chart review as the reference standard and the use of Standards for Reporting of Diagnostic accuracy (STARD) guidelines for reporting.
The results from the present assessment cannot be generalised in other settings.
We are unsure whether the results presented for the ICD-9 code 172.x related to malignant melanoma of the skin could be also valid for the corresponding ICD-10 code C43.x.
Introduction
The burden of cancer is increasingly growing across populations and it is associated with major economic expenses and health resource use. Melanoma is probably the most aggressive form of skin cancer and when it spreads beyond the primary site in the skin it has very poor prognosis.1 Reports indicate that incidence of malignant melanoma has increased globally2 3 having an impact on the public health and economic burden of disease, particularly in Western countries.4–6
Trends in epidemiology of melanoma, and its survival rates can be assessed using cancer registries or administrative healthcare databases.7 Compared with cancer registries, administrative databases have the advantage that they can link different sources of information (such as prescription data or comorbidities) providing a comprehensive research. However, these databases need to be adequately validated by comparing their main content, that is, the diagnosis represented by the International Classification of Diseases, 9th Revision (ICD-9) or 10th Revision (ICD-10) edition, with another source, which generally is a cancer registry or the medical chart.8
In Italy, all the Regional Health Authorities maintain large healthcare information systems containing patient data from all hospital and territorial sources. These databases have the potential to address important issues in postmarketing surveillance,9 10 epidemiology,11 quality performance and health services research.12 However, there is a concern that their considerable potential as a source of reliable healthcare information has not been achieved due to lack of validation including codes related to melanoma.13 Hence, it is imperative that Regional Health Authorities systematically validate their databases for critical diseases to productively use the information they contain.13–18
The objective of the present study was to evaluate the accuracy of the ICD-9-Clinical Modification (CM) codes in correctly identifying melanoma using three large Italian administrative healthcare databases. We performed this study applying the same methodological approach as stated in our previous protocol on validation concerning breast, lung and colorectal cancer cases.8
Methods
Setting and data source
Administrative databases
From the early ’90s, local and regional Italian administrative databases have collected healthcare data about residents from public and private hospitals. These data include demographics, vital statistics, hospital admission and discharge dates, the admitting hospital department, principal and secondary discharge diagnoses as well as diagnostic procedures. Additionally, these databases comprise the records of all drug prescriptions listed in the National Drug Formulary and prescriber’s information. Since healthcare is covered almost entirely by the Italian National Health System and each resident has a unique regional identification code, it is possible to reconstruct the disease and prescription history of each resident within the administrative database. However, within the environment of the databases and new code is generated to secure the identity of the residents.
The administrative databases contain also the number of the Hospital Discharge Register with which it is possible to identify the medical charts that are stored physically in their respective hospital or local health unit. The registration number contains the codes of the hospital and department of admission and is generated in way that it becomes a single code at national level to avoid any duplicate.
The target administrative databases for the present study were from the Umbria Region (890 000 residents), ASL Napoli 3 Sud (NA) (1 170 000 residents) and the Friuli Venezia Giulia (FVG) Region (1 227 000 residents). For the purpose of the present study, the corresponding Units (Regional Health Authority of Umbria for Umbria Region, Registro Tumori Regione Campania for ASL Napoli 3 Sud and Centro di Riferimento Oncologico Aviano for FVG Region) conducted the same validation process independently within each own database.
Source population
All residents aged 18 or above of Umbria Region, ASL Napoli 3 Sud and the FVG Region represented the target population. Any resident that has been discharged from hospital with a diagnosis of melanoma was considered. Due to difficulty in obtaining the medical charts, subjects who have been hospitalised outside the regional territory of competence were excluded from analysis.
Patient and public involvement
Patients were not directly involved. This was a retrospective study based on the consultation of medical charts.
Case selection and sampling method
In each administrative database, patients with the first occurrence of melanoma between 1 January 2012 and 31 December 2014 were identified using the ICD-9-CM codes 172.x located in primary position of hospital discharges. From this cohort, prevalent cases, that is, melanoma cases (ICD-9-CM codes in any position) in the 5 years (2007–2011) before the period of interest were excluded. This cohort represented our target population from which a sample of cases was obtained using a simple random method.
In the same time frame, non-cases, that is, patients having in primary position a diagnosis of cancer (ICD-9 140–239) other than melanoma (ICD-9 172.x) were identified. From this cohort prevalent cases, that is, those with the same diagnosis (ICD-9 140–239 codes in any position) in the 5 years (2007–2011) before the period of interest were excluded. This cohort represented our target population from which a sample of non-cases (controls) was obtained using a simple random method.
Chart abstraction and case ascertainment
The corresponding medical charts of the randomly selected samples of cases and non-cases were obtained from hospitals for validation purposes. Information retrieved from each medical chart included: date of birth and gender of the patient, dates of hospital admission and discharge, and any diagnostic procedure that contributed to the diagnosis of melanoma.
Within each unit, two reviewers received training on data abstraction. Based on a sample of 20 medical charts, within each unit, the inter-rater agreement regarding data abstraction of the several items within the medical charts among the pairs of reviewers was calculated using the κ statistics. The agreement among the pairs of reviewers resulted very high (κ>0.90). Following the consensus review, data abstraction has been completed independently.
Case ascertainment of melanoma within the medical chart was based on (1) the clinically documented presence of a primary lesion of the skin and (2) the histological documentation of melanoma from a primary or metastatic site.8 To ensure consistency among reviewers, cases with uncertainty were discussed and resolved through the involvement of an oncologist (Rita Chiari).
Validation criteria
For melanoma, we considered the ICD-9-CM codes 172.x valid, when there is evidence of a neoplastic lesion of the skin and a histological diagnosis from a primary or metastatic site positive for melanoma.
Statistical analysis
As reported elsewhere,8 a random sample of 130 charts of cases was necessary to obtain an expected sensitivity of 80% with a precision of 10% and a power of 80% according to binomial exact calculation.19 For specificity calculation, we randomly selected non-cases from an oncological cohort of subjects within the databases excluding the subjects with the ICD-9 codes of melanoma. A sample of 94 charts of non-cases was deemed necessary to obtain an expected specificity of 90% with a precision of 10% and a power of 80%.8
Sensitivity and specificity with their corresponding 95% CIs were calculated by constructing 2×2 tables.
Results
The incident cases of melanoma were 113 from Umbria, 134 from NA and 403 from FVG, from which, respectively, 113, 130 and 130 cases were randomly selected and the corresponding medical charts were requested for assessment. Fourteen (11%) and one medical charts (1%) were not available from NA and Umbria, respectively. Figure 1 shows the study screening process by which incident cases were identified from the three operative units. For the non-cases, each unit randomly selected 94 medical charts. Two medical charts of non-cases from Umbria were missing.
Flow chart of incident melanoma cases identification in primary position from the three administrative databases and the corresponding charts identified and examined.
The most common ICD-9-CM subgroup was the code 172.5 (ie, malignant melanoma of skin of trunk, except scrotum) accounting for 30% in Umbria, 34% in NA and 38% in FVG, followed by the code 172.7 (ie, malignant melanoma of skin of lower limb, including hip) accounting for 19% in Umbria, 26% in NA and 21% in FVG. The mean age of the patients was 61 years in Umbria, and 60 years in the other two operative units. Most of the cases were identified in surgical departments with a percentage ranging from 75% to 86%. The instrumental tools used for diagnosis included ultrasound, full-body CT scan, whole body Positron Emission Tomography/Computed Tomography (PET/CT), CT scan of the head or MRI of the brain and lymphoscintigraphy.
Histological examinations from biopsy were 77 (69%) for Umbria, 33 (28%) for NA and 55 (42%) for FVG, while histological examinations from resection specimens after surgery were 80 (71%), 94 (81%) and 118 (91%), respectively. Table 1 displays the basic characteristics of malignant melanoma of skin cases in each unit.
Characteristics of subjects with melanoma who were identified in the three administrative healthcare databases
Clinical or instrumental diagnosis together with histological examinations based on melanoma case definition showed high sensitivities in the three operative units. The sensitivities were 100% (95% CI 96% to 100%) for Umbria, 99% (95% CI 94% to 100%) for NA and 98% (95% CI 93% to 100%) for FVG. The false positive rates were higher than the false negative rates resulting in the following specificities: 88% (95% CI 80% to 93%) for Umbria, 77% (95% CI 69% to 85%) for NA and 79% (95% CI 71% to 86%) for FVG. Figure 2 displays accuracy results with their CIs.
Sensitivity and specificity with 95% CIs for malignant melanoma ICD-9-CM codes for the three administrative databases. ICD-9-CM, International Classification of Diseases, Ninth Revision, Clinical Modification.
Misclassification of cases and controls is described in table 2. In Umbria, six false positive cases were due to histological documentation missing and seven were due to negative histology of the wide excision of previous melanoma. In NA, 15 false positive cases were due to histological documentation missing and 12 were due to negative histology for melanoma. In FVG, 7 false positive cases were due to histological examination missing and 15 were due to negative histology for melanoma (11 of which resulted positive for melanoma in situ). Overall, there were only two false negatives, one possible melanoma metastasis (in NA) and another skin cancer of unclear histology (in FVG).
Reason for incorrect identification of cases and controls
Sensitivity analysis based on the worst case scenario did not show any statistical difference when missing data were considered false negatives (non-cases) or false positives (cases). Due to the 14 medical charts of the cases, the specificity for the NA administrative database was reduced from 77% to 69% (95% CI 61% to 77%) although with no statistical difference.
Discussion
In administrative databases, the diagnosis of a disease is associated with a specific code from the ICD-9 or ICD-10 edition. Despite its limitation, the ICD code is an innovative tool designed to map health conditions to corresponding generic categories together with specific variations.20 Within three administrative databases, we have completed the validation of ICD-9 codes related to breast, lung, colorectal and cervix cancer. We limited our analysis to ICD-9 codes because in Italy they are still used in the hospital discharge data.
In the present study, we evaluated the validity of diagnoses related to melanoma recorded as administrative data, using chart review as the gold standard. Our results suggest that the ICD-9 codes 172.x are accurate to identify incident melanoma cases. The sensitivities were excellent across all the three administrative databases and specificities were good. As far as we know this is the first study that addressed the topic of validation of melanoma in Italy. In the USA, using a linked SEER tumour registry-Medicare database, Barzilai et al determined the accuracy of Medicare claims to identify patients aged 65+ diagnosed with invasive melanoma.21 The authors found that the overall sensitivity of combined part A and part B Medicare to identify incident cases of melanoma was 90%. Specificity and predictive values were not calculated.21
Recent progresses in the use of immune mediated or therapies such as targeted immunomodulatory therapies such as vemurafenib and dabrafenib have shown encouraging results in survival for metastatic patients with melanoma.22 Another immunotherapeutic agent, ipilimumab, has shown to have important properties in enhancing the immune response against melanoma.23 Trends in the epidemiology and evaluation of such innovative immunotherapies in terms of long-term outcomes can be performed using population-based studies in these validated administrative databases.
Strength and limitation
Our main strength is that to ascertain the presence of melanoma we used medical chart in which a clinical diagnosis combined with a histological documentation need to be present. Although we did not publish a specific protocol for the assessment of the accuracy of melanoma ICD-9 codes, our study was based on the protocol8 that aimed to assess the validation of codes related to breast, colorectal and lung cancer. With respect to the methodology, we state that no deviation from protocol occurred during study performance. Additionally, we followed recommended guidelines based on the criteria published by the Standards for Reporting of Diagnostic accuracy (STARD) initiative for the accurate reporting of investigations of diagnostic studies. Hence, we used a detailed and explicit eligibility criteria, as well as duplicate and independent processes for medical chart review and data abstraction.24–26
As declared in our protocol, we prioritised the estimation of sensitivity rather than positive predictive value (PPV) because PPVs can be influenced by the prevalence of disease. However, we calculated the PPVs that resulted 88% for Umbria, 77% for NA and 82% for FVG. To comply with the STARD, we provide absolute numbers for the 2×2 tables (table 3).
Cross tabulation of the index test (ICD-9-CM code 172.x) results by the results of the reference standard (medical chart)
The mean age of our sample population ranged between 60 and 61 years that is higher than the mean age (55 years) reported in the medical literature.27 Age variability can be due to thickness and histological subtype of the melanoma, but we were able to plan the acquirement of these data.27
A possible limitation of our results for future research is that validation studies of administrative databases are related to the context where they are generated, and may not be generalisable to other settings. Another limitation is that we are unsure whether the results presented for the ICD-9 code 172.x related to malignant melanoma of the skin could be also valid for the corresponding ICD-10 code C43.x.
Conclusion
Our study showed that administrative healthcare databases from Umbria, Napoli and FVG are accurate in identifying new melanoma cases using the ICD-9 code 172.x. Hence, these databases can confidently be used to monitor melanoma trends, and to assess the quality of healthcare for patients with melanoma.
References
Footnotes
Contributors AM, IA, MF and DS conceived the original idea of the study. IA, DS, AM, MF, EB, GG, FC, MO, FS and WO designed the study. MG, AG, MFV, FC, MO, PE and VC identified the cohort using administrative database with the supervision of PC, GG, WO, EB, DS, MF, AM and FS. IA, FC, MO, AG, MG, VC and MFV undertook the data abstraction with the supervision of AM, GG, WO, FS, MF, EB, and DS. IA, FS, AM and DS performed case ascertainment. IA, AM, FC, PE, VC and MO performed the analysis. DS, MF, GG, PC, AG, MG, FS, MFV, WO and EB helped in the interpretation of the data. The initial draft of the manuscript was prepared by IA, AM, EB, FS, MF and MO. DS, GG, PC, FC, AG, MG, VC, MFV, PE and WO revised critically the manuscript for important intellectual content. All the authors read and approved the final manuscript. AM, MF and EB are the guarantor of the data for the respective operative units.
Funding This study was developed within the D.I.V.O. project (Realizzazione di un Database Interregionale Validato per l’Oncologia quale strumento di valutazione di impatto e di appropriatezza delle attività di prevenzione primaria e secondaria in ambito oncologico) supported by funding from the National Centre for Disease Prevention and Control (CCM 2014), Ministry of Health, Italy.
Disclaimer The study funder was not involved in the study design or the writing of the protocol.
Competing interests None declared.
Patient consent Not required.
Ethics approval Ethical approval for the present study including the access to medical charts was obtained from the Ethics Committee of the Umbria Region Health Authority, authorisation number 2656/15 (04/11/2015).
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement No additional data are available.
Collaborators Giuliana Alessandrini; Marcello De Giorgi; Rita Chiari; Roberto Cirocchi; David Franchini; Paolo Collarile.