Literature search

Comprehensive searches of MEDLINE, EMBASE, Web of Science and the Cochrane Library, from their inception, will be performed to identify published peer-reviewed literature. We will employ a search strategy that we developed based on the combination of: (1) keywords and MeSH terms to identify records concerning breast, lung and colorectal cancers; (2) terms to identify studies likely to contain validity or accuracy measures and (3) a search strategy designed to capture studies that use healthcare administrative databases based on the combination of terms used by Benchimol et al20 and the Mini-Sentinel's program.21 ,22 The developed search strategy is available as online supplementary appendix 1. To retrieve additional articles, the authors will hand search relevant reference lists of key articles. We will also use the ‘Cited-By’ tools in PubMed and Google Scholar to find relevant articles that cited the article of interest, identified through the aforementioned search strategy. Titles and abstracts will be screened for eligibility by two independent reviewers. Discrepancies will be solved by discussion.

This review protocol is prepared according to the Preferred Reporting Items for Systematic reviews and Meta-Analysis Protocols (PRISMA-P) 2015 statement23 and the results will be presented following the PRISMA flow diagram (figure 1).24 This protocol has also been published in the PROSPERO International Prospective Register of systematic reviews with registration number CRD42015026881 (http://www.crd.york.ac.uk/PROSPERO).

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Study screening process.

Inclusion criteria

Full texts of eligible peer-reviewed articles that used administrative data for the ICD-9 or ICD-10 codes related to breast, lung and colorectal cancer diagnoses, without publication date restriction, published in English, will be obtained. For each study, the following inclusion criteria will be applied: (1) the presence of a reference standard case definition for the cancer of interest; (2) the presence of at least one test measure (eg, sensitivity, positive predictive values (PPVs), etc); (3) the data source was from an administrative database (ie, a database in which data is routinely and passively collected without a priori research question) and (4) the study database was from a representative sample of the general population.

We aim to focus on primary diagnosis of cancers, hence studies that considered algorithms to identify cancer history, cancer progression or recurrence will not be evaluated.

Studies that used EHRs to validate the disease of our interest will also be included. An EHR consists of a digital file used by healthcare providers for patient care and generally it includes clinical notes, prescription records and radiology and laboratory data.25 Similar to most administrative databases, EHRs are not established for research purposes.26 However, studies that used validated EHRs as a reference standard will be considered in our evaluation.

In addition, studies that employ databases that were not truly administrative (eg, cancer registries, epidemiology surveillance systems, etc) will be excluded.

Selection process

At the initial stage, titles and abstracts will be screened for potentially eligible studies. Subsequently, full texts of articles will be obtained and assessed to determine if they meet the inclusion and exclusion criteria. We will conduct data abstraction using standardised data collection forms that will be first tested on a sample of eligible articles. Two review authors working independently and in tandem will be involved in title and abstract screening, full-text screening and data abstraction. Any discrepancies will be resolved by consensus and where necessary, with the involvement of a third review author. Calibration exercises will be performed at each level of the process.

Data extraction

Data extraction will include the following information:

1. The details of the included study (including title, year and journal of publication, country of origin and sources of funding; the first author will be used as the study identification);

2. The disease of interest (breast, lung and colorectal cancer);

3. The target population from which the administrative data were collected;

4. The type of administrative database used (eg, hospitalisation discharge data), outpatient records (eg, physician billing claims), etc;

5. The ICD-9 or ICD-10 code used or the administrative data algorithms tested these may include current procedural terminology; prescription fills, timing of diagnosis, etc);

6. The modality of development of the algorithm (eg, using classification and regression trees, logistic regression, expert opinion, etc);

7. External validation;

8. Use of training and testing cohorts;

9. The reference standard used to determine the validity of the diagnostic code (eg, medical chart review, patient self-reports, cancer registry, etc);

10. The characteristic of the test used to determine the validity of the diagnostic code or algorithm (eg, sensitivity, specificity, PPVs and negative predictive values (NPVs), area under the receiver operating characteristic curve, likelihood ratios and κ statistics);

11. Any funding source and conflict of interest.

Quality assessment

The design and method of the included primary studies will be assessed using a checklist developed by Benchimol et al,20 based on the criteria published by the Standards for Reporting of Diagnostic accuracy initiative for the accurate reporting of studies using diagnostic studies.27 The checklist is provided in online supplementary appendix 2. The presence of potential biases within the studies will be reported in a descriptive way. Neither subgroup analysis nor publication bias assessment is anticipated.

Analysis

For each algorithm, we will abstract the performance statistics provided in the included studies. Validation statistics may include sensitivity, specificity, PPV and NPV. Sensitivity measures the degree to which an ICD-9 or ICD-10 code (eg, ICD-9 174) correctly identifies individuals who possess the characteristic of interest (ie, breast cancer) in the source used as a reference standard (eg, medical chart).16 We will calculate 95% CI when they are not reported in the articles. Where possible, we will calculate PPV and NPV will be calculated if not reported.

PPV is the number of true positives divided by the total number of cases receiving the code and expresses the likelihood that the code corresponds to a true-positive case. NPV is the number of true negatives divided by the total number of cases without the code of interest and expresses the likelihood that the absence of the code corresponds to a true-negative case.

Where possible, validation statistics will be aggregated and stratified by administrative data source (outpatient vs inpatient data), type of ICD code (ICD-9 or ICD-10), stage of disease and country of origin.