Selective Reporting Biases in Cancer Prognostic Factor Studies

Abstract

Background: Nonreported and selectively reported information and the use of different definitions may introduce biases in the literature of prognostic factors. We probed these biases in a meta-analysis of a prognostic factor for head and neck squamous cell cancer (HNSCC) mortality that has drawn wide attention—the status of the tumor suppressor protein TP53. Methods: We compared results of meta-analyses that included published data plus unpublished data retrieved from investigators; published data; and only published data indexed with “survival” or “mortality” in MEDLINE/EMBASE, with or without standardized definitions. We also evaluated whether previously published meta-analyses on mortality predictors for various malignancies addressed issues of retrieval and standardized information. All statistical tests were two-sided. Results: For the 18 studies with 1364 patients that included published and indexed data, we obtained a highly statistically significant association between TP53 status and mortality. When we used the definitions preferred by each publication, the association was stronger (risk ratio [RR] = 1.38, 95% confidence interval [CI] = 1.13 to 1.67; P = .001) than when we standardized definitions (RR = 1.27, 95% CI = 1.06 to 1.53; P = .011). The addition of 13 studies with 1028 subjects that included published but not indexed data reduced the observed association (RR = 1.23, 95% CI = 1.03 to 1.47; P = .02). Finally, when we obtained data from investigators (11 studies with 996 patients) and analyzed it with all other data, statistical significance was lost (RR = 1.16, 95% CI = 0.99 to 1.35; P = .06). Among 18 published meta-analyses of 37 cancer prognostic factors, 13 (72%) did not use standardized definitions and 16 (89%) did not retrieve additional information. Conclusions: Selective reporting may spuriously inflate the importance of postulated prognostic factors for various malignancies. We recommend that meta-analyses thereof should maximize retrieval of information and standardize definitions.

An enormous amount of data is produced on prognostic factors of outcomes for cancer and other diseases ( 1 ) , and the pace is accelerating as a result of discovery-driven high-throughput research ( 2 ) . Summarizing and making sense of this literature through meta-analyses is a daunting task ( 3 , 4 ) . Although meta-analyses of prognostic factors are being undertaken and published at an increasing rate ( 4 ) , there are several unanswered issues about the validity of the literature on prognostic factors and about the problems that underlie prognostic evidence. In contrast to randomized trials, for which the process of conducting systematic reviews is standardized and major biases are well recognized ( 5 ) , data on prognostic factors poses poorly understood challenges for those conducting meta-analyses. For example, information on a specific prognostic relationship may be presented as a key indexed finding in one study, appear in the “small print” (i.e., is incidentally mentioned) in another study, not be presented at all in yet another study, or be mentioned but not presented with data. Moreover, investigators define outcomes, predictors, and analyses in various nonstandardized ways ( 6 ) , and this may introduce biases depending on which information is synthesized.

The purpose of this study was to assemble empirical evidence on the importance of selective reporting biases for prognostic evidence in malignant diseases. First, we focused on a prognostic factor for head and neck cancer that has received extensive attention in the biomedical literature—the status of the tumor suppressor protein TP53. We evaluated whether the indexed, published, and unpublished data gave different results and whether the use of standardized definitions instead of those preferred by each publication influenced the final inferences. Second, we examined whether issues of retrieval of information and standardization of definitions and analyses are adequately addressed across published meta-analyses of prognostic factors for cancer mortality.

S TUDIES AND M ETHODS

Meta-Analysis Design and Search for Data

The tumor suppressor protein TP53 and its gene have been widely studied as regulators of carcinogenesis and cancer outcomes ( 7 ) . A PubMed search showed 31 899 entries for “p53” or “TP53” as of April 25, 2004. We performed a meta-analysis of the available evidence on whether TP53 status (as measured with various immunohistochemical or molecular techniques) is a predictor of mortality in patients with head and neck squamous cell cancer (HNSCC), a cancer for which TP53 status has been frequently analyzed. We examined whether meta-analysis results would differ depending on the level of inclusion and standardization of eligible data. The following three levels of information search were considered.

First, we tried to identify studies with any allusion to TP53 status and HNSCC that were indexed with “mortality” OR “survival” in MEDLINE and EMBASE (last update April 2004, search terms for the malignancy and prognostic factor available on request from the authors). We classified the identified studies as the “published and indexed data.” We then removed the “mortality” OR “survival” limiting terms to obtain studies classified as all the “published data.” Finally, when a report suggested that mortality data had been collected, but no usable data were available in the publication, we communicated with the primary investigators. When there was no response within 2 months, a second communication attempt was made. We classified the additional recovered information as the “retrieved” data. When studies overlapped, only the largest available study was retained.

Definitions and Standardizations

We used a priori defined standardized outcomes and definitions for TP53 status to avoid subjective selection of outcomes and definitions across studies as much as possible ( 6 ) . The level of TP53, measured by immunohistochemistry, is associated only modestly with TP53 mutations detected by reverse transcription–polymerase chain reaction (RT-PCR) in exons 4–9 ( 8 ) . When a study provided data for both methods, we used the immunohistochemistry information. For immunohistochemistry, we defined a TP53-positive status as nuclear staining in at least 10% of tumor cells or at least moderate staining in qualitative scales. This cutoff point is the same as the one that we used in a previous meta-analysis of TP53 status ( 9 ) . If different definitions of a TP53-positive status were used, we accepted the cutoff closest to 10%. In sensitivity analyses, we used RT-PCR data instead of immunohistochemistry data, when both were available.

The main outcome was all-cause mortality. To avoid bias that may arise, if investigators select the follow-up period to report according to the results at each follow-up period, we standardized definitions to include 24 months of follow-up in all studies (because most studies had at least this much follow-up) and categorized patients as dead within 24 months or as surviving for at least 24 months. Cox models that allow estimation of a hazard ratio for the whole follow-up are not routinely presented in this TP53 literature. The very few patients censored before 2 years were counted as alive. In sensitivity analyses, these patients were excluded.

As a secondary outcome, we also recorded published information on the presence of lymph node metastasis at the time of diagnosis, which is the strongest known predictor of outcome in HNSCC ( 10 ) . Lymph node metastasis was defined as the involvement of at least one lymph node.

Data Extraction

Two authors (PK and KL) extracted data independently and reached a consensus on the classification of all data. For each report, we recorded author name, journal and year of publication, country of origin, sample size, staging, demographics, tumor location, antibodies and cutoff points for immunohistochemistry analyses, exons analyzed with RT-PCR, definition of a TP53-positive status, prospective versus retrospective design, and use of blinding during the analysis. We created 2 × 2 contingency tables for 2-year survival compared with death according to TP53 status and for the presence of lymph node metastasis compared with its absence, according to TP53 status. For indexed studies, we also recorded the mortality data as defined by each published report.

Analysis

Risk ratios (RRs) for 2-year mortality associated with TP53 status were combined for the various levels of information examined (published and indexed, all published, and all published and retrieved) ( 11 ) . For indexed studies, we also estimated risk ratios for mortality according to the definitions preferred by each report. Between-study heterogeneity was assessed with the Q statistic ( 12 ) . Fixed effects models, such as the Mantel–Haenszel model ( 12 ) , assume that differences between studies are due to chance. Random effects models, such as the DerSimonian and Laird model ( 12 ) , allow that results may differ genuinely between studies. Unless stated otherwise, random effects estimates are reported. We also performed subgroup analyses for blinding (theoretically, blinded studies are less likely to be biased), type of design (prospective, retrospective, or unclear), geographic area (North America, Europe, or Asia), type of measurement, sample size, and source of data. Heterogeneity between subgroups was quantified with the I² statistic ( 13 ) , which takes values from 0% to 100%. The larger the value, the larger the heterogeneity; values of 75% or higher indicate very large heterogeneity.

For each group of studies, we examined whether results differed between small and larger studies. This result may be a hint for publication bias or other biases ( 14 ) . We assessed inverted funnel plots that show the natural logarithm of the risk ratio on the horizontal axis and the inverse standard error on the vertical axis ( 15 ) , their regression equivalent ( 14 ) , and the Begg–Mazumdar correlation test (considered statistically significant for P <.10) ( 16 ) . We also evaluated whether adjusted estimates were available from the primary studies for data synthesis and synthesized the available data on the relationship between TP53 status and lymph node status. Analyses were conducted with the SPSS package of programs, version 11.0 (SPSS, Chicago, IL), and Meta-Analyst (Joseph Lau, Boston, MA). All P values are from two-sided statistical tests.

Evaluation of Published Meta-Analyses of Prognostic Factors

Selective reporting biases may arise for any prognostic factor. To assess the extent to which these problems are appreciated and properly handled in published meta-analyses on cancer prognostic factors, we identified relevant English-language meta-analyses in MEDLINE with a search algorithm based on “prognosis” AND “meta-analysis” AND “cancer.” We accepted meta-analyses that examined potential prognostic factors for any malignancy and examined their association with mortality. For each eligible meta-analysis, three independent investigators recorded the author, year, journal of publication, malignancy and prognostic factors addressed, and whether the summary results were statistically significant ( P <.05) for each prognostic factor, as reported by the meta-analysis authors. We also collected information on limiting terms posed to the literature search, efforts made to retrieve additional information (unpublished data or data not presented in sufficient enough detail for quantitative synthesis), mentions to the amount of data not amenable to quantitative synthesis, efforts made to use standardized and consistent definitions for mortality and for the prognostic factor across studies, performance of tests for potential publication bias, and use of adjusted and/or unadjusted effects for data synthesis.

Contributions

The original idea for biases in prognostic factor meta-analyses was generated by J.P.A.I., and the protocol was developed by J.P.A.I. and P.A.K. and commented on by K.T.L. P.A.K. and K.T.L. performed the data extraction on the TP53 meta-analysis, and all three authors performed data extraction on the published meta-analyses. P.A.K. and J.P.A.I. performed the statistical analyses, and all three authors interpreted the findings. P.A.K. and J.P.A.I. drafted the final manuscript, and K.T.L. revised it critically.

R ESULTS

Eligible and Available Data for TP53 Meta-Analysis

We examined the full text of 116 reports addressing TP53 status in HNSCC. Of those, 20 were excluded because they overlapped with another study. Another 17 studies with 1342 patients had apparently collected no clinical data on either lymph node involvement or mortality. Of the 79 potentially eligible studies ( 17 – 95 ) with some clinical information and with 5854 patients ( Table 1 ), 64 with 4824 patients clearly alluded to mortality information. For 22 of 64 studies, even though we contacted their primary investigators, we could not retrieve any additional data. Seventeen of the primary investigators did not reply at all; and five responded and stated that they were not able to retrieve the raw data. Thus, only 42 studies with 3388 patients could eventually be analyzed, including 18 studies with 1364 patients that had readily available published data and survival or mortality as an indexed term; 13 with 1028 patients that had readily available published data that were not appropriately indexed; and 11 studies with 996 patients that had data retrieved from the investigators ( Fig. 1 and Table 2 ).

The 22 studies with eventually unavailable mortality data were not statistically significantly smaller on average than the 42 studies with usable data (mean number of patients = 65.2 and 80.7, respectively; Mann–Whitney P = .90) and were not statistically significantly more likely to use immunohistochemistry (18 of 22 studies versus 31 of 42 studies; chi-square P = .47) to measure TP53. Published and indexed, published–not indexed, and retrieved studies with available mortality data did not differ statistically significantly in any characteristics ( P >.05 for all) ( Table 2 ).

Of the 22 studies with nonretrievable analyzable information on survival, TP53-negative status was claimed to be associated with worse 5-year survival in two studies with 104 patients. One study with 57 patients showed a non–statistically significant trend in the same direction, and four studies with 511 patients showed at a non–statistically significant trend in the opposite direction. Two studies with 88 patients made no comment, and 13 studies with 692 patients stated that there was no statistically significant difference in survival without further details.

In 10 of the 18 studies with readily available and indexed data, investigators used definitions in the mortality analyses that differed from those in this meta-analysis. All 10 studies used a follow-up of other than 2 years (i.e., 5-year survival); in one study, the TP53 definition also differed from the one we used (i.e., the authors used PCR instead of immunohistochemistry data).

Data Synthesis

When only the 18 published and indexed data were considered, positive TP53 status was highly statistically significantly associated with mortality when we used the definitions preferred by each publication (risk ratio [RR] = 1.38, 95% confidence interval [CI] = 1.13 to 1.67, P = .001; and statistically significant between-study heterogeneity, P = .02). The strength of the relationship between TP53 status and mortality decreased when we used standardized, prespecified definitions of TP53 status and used 2-year mortality data (RR = 1.27, 95% CI = 1.06 to 1.53, P = .01; and non–statistically significant between-study heterogeneity, P = .13). When we considered published but not indexed survival data from 13 studies, the strength of the relationship between TP53 status and mortality was reduced even more (RR = 1.23, 95% CI = 1.03 to 1.47, P = .02; and statistically significant between-study heterogeneity, P <.001), because published but not indexed survival data did not show any clear association between TP53 status and mortality (RR = 1.13, 95% CI = 0.81 to 1.59, P = .47; and statistically significant between-study heterogeneity, P <.001). The data retrieved from the investigators for 11 studies actually showed a statistically significant trend for an association in the opposite direction (RR = 0.97, 95% CI = 0.72 to 1.29, P = .81; and statistically significant between-study heterogeneity P = .05). Finally, when all available data were considered, positive TP53 status was no longer associated with worse mortality (RR = 1.16, 95% CI = 0.99 to 1.35, P = .06; statistically significant between-study heterogeneity P <.001) ( Table 3 and Fig. 2 ).

Sensitivity Analyses

Sensitivity analyses that excluded living patients censored before 2 years of follow-up showed even less evidence for any prognostic association between TP53 status and mortality (for analyses of published and indexed data, RR = 1.14, 95% CI = 0.93 to 1.40; for analyses of all published data, RR = 1.15, 95% CI = 0.97 to 1.38; and for analyses that included all data retrieved from investigators, RR = 1.11, 95% CI = 0.95 to 1.29). Sensitivity analyses of studies that used data from RT-PCR instead of data from immunohistochemistry, when both were available, provided estimates similar to the main analyses (for the respective datasets, RR = 1.38, 95% CI = 1.12 to 1.71; RR = 1.30, 95% CI = 1.09 to 1.56; and RR = 1.21, 95% CI = 1.03 to 1.42).

Subgroup Analyses

Subgroup analyses showed a statistically significant association in studies that did not state whether they were blinded, but no association in blinded studies ( I² = 56%). Although the available RT-PCR data showed a statistically significant association ( I² = 57% compared with immunohistochemistry-derived estimates), this association might be spurious because all four additional RT-PCR studies with 250 patients, for which detailed data could not be retrieved and included in the quantitative synthesis, claimed that there was no association. The overall estimates were similar whether we performed subgroup analyses according to primary tumor location ( I² = 37%) or immunohistochemistry cutoff ( I² = 18%). Absolutely no heterogeneity was found between subgroups defined by geographic location or study design ( I² = 0% for both analyses). Prospective studies showed no association (RR = 1.01, 95% CI = 0.71 to 1.43, P = .95), whereas retrospective studies showed a borderline statistically significant association (RR = 1.22, 95% CI = 1.00 to 1.49, P = .05) ( Table 3 ).

Bias, Adjusted Analyses, and Lymph Node Status Analyses

The estimates provided by larger, more precise indexed studies were more conservative than those provided by smaller indexed studies, as reflected in the asymmetric funnel plot of the data ( Fig. 3, A , P = .09 for the regression equivalent test; and P = .13 for the correlation test, correlation coefficient = .26). The asymmetry decreased when all published data were considered ( Fig. 3, B , P = .13; and P = .56, correlation coefficient = −.07, respectively) and disappeared when all retrieved data were also included ( Fig. 3, C , P = .98; and P = .35, correlation coefficient = −.10, respectively).

Some information on adjusted analyses for the association between TP53 status and mortality was given in 18 of the 42 analyzed studies and in six of the 22 studies with nonretrievable analyzable information. However, 13 studies provided only a P value or a statement of whether or not the association was statistically significant. In the 11 studies that provided adjusted estimates of the association between TP53 status and mortality, the adjusting variables were never the same across studies. Lymph node stage was the most commonly used adjusting parameter, and it was used in only five studies.

Positive TP53 status was also statistically significantly associated with the presence of lymph node metastasis when we analyzed the 39 studies with published data (RR = 1.17, 95% CI = 1.08 to 1.27, P <.001; and statistically significant between-study heterogeneity). Subgroup analyses are listed in Table 4 . We found a statistically significant difference between the estimates provided by large, more precise studies and those provided by smaller studies (for the regression analysis, P = .01; for the correlation test P = .04. correlation coefficient = .24; Fig. 3, D ).

Published Meta-Analyses of Prognostic Factors for Various Cancers

Among 593 entries obtained by the initial search, our screening strategy identified 18 English-language meta-analyses ( 96 – 113 ) that targeted potential predictors of mortality in various malignant diseases ( Table 5 ). Most analyzed prognostic factors (28 [76%] of the 37 factors) were eventually found to be statistically significantly associated with mortality. Although only two (5%) of the 37 meta-analyses explicitly used “survival” as a limiting term in the search algorithm, 16 of the 18 stated that the presentation of survival data in the text was considered as an eligibility criterion. Language was used as a limiting term in 12 (67%) of the 18 meta-analyses. Only two (13%) of the 18 meta-analyses stated an effort to retrieve data from the primary investigators, and only one of them actually presented the number of patients for whom data were retrieved. One of these two meta-analyses also presented the number of patients for whom informative data existed, but these data could not be retrieved. Another nine meta-analyses reported on the number of studies with eligible but not evaluable data; however, defining studies with eligible but not evaluable data was limited to studies that reported survival data in a nonusable form, and studies that clearly had collected follow-up information but did not present survival data at all in their publications were not considered.

Only five (28%) of the 18 meta-analyses used a standardized follow-up time, and only one (6%) attempted to use a standardized definition for the expression of the prognostic factor to the extent possible. None, however, converted the data from all studies to exactly the same definition. Five (28%) of the 18 papers considered the possibility of publication bias. The applied test was statistically significant in three of them; another study claimed a symmetric funnel plot, whereas the data showed the contrary. Five meta-analyses used adjusted estimates from the primary studies, and 13 meta-analyses apparently used unadjusted estimates. No meta-analysis performed separate analyses for both adjusted and unadjusted estimates.

D ISCUSSION

Selective reporting has the potential to threaten the validity of the literature on postulated prognostic factors. In our case study, readily accessible published data would have been misleading because it indicated that TP53 status is a strong prognostic factor for outcome of HNSCC. When we made no effort to standardize data across studies but rather relied only on the definitions used in each publication, we found that the association was particularly strong, reaching a P value of .001. When we standardized the definitions of TP53 status and outcomes and retrieved additional information that was mentioned in only a cursory fashion or was not published at all, the statistical significance of the association was abrogated. We should caution that the confidence intervals of the risk ratio for readily indexed, nonindexed, and unpublished retrievable information overlapped. However, we believe that readily available information on prognostic factors may be the tip of the iceberg and that superficial perusal of the literature may lead to erroneous conclusions.

An overview of meta-analyses on prognostic factors revealed that the use of nonstandardized information is almost ubiquitous in meta-analyses and typically only readily presented data are used. Thus, most meta-analyses of prognostic factors published to date appear to be susceptible to the biases that we observed for the association between TP53 and HNSCC.

Potential publication bias is a problem across biomedical research ( 5 , 14 , 15 ) . Bias diagnostics may suggest the existence of this problem, when large studies differ in their results from smaller studies, but these diagnostics are neither very sensitive nor specific ( 114 ) . Moreover, given the plethora of candidate predictors and outcomes, extensive prognostic analyses may be generated by a study team ( 115 ) , but only a fraction of those analyses may be published and even fewer of those analyses may be reported in adequate detail. Investigators may select definitions of outcomes and prognostic factors that yield most impressive, statistically significant results ( 6 ) . Selective reporting and presentation bias are not uncommon even in randomized trials ( 116 ) , but they may be even more prominent in the prognostic factor literature. Prognostic factor studies are increasing rapidly across various biomedical fields, and thousands of articles are published for various predictors of the outcome of malignant diseases. In this TP53 meta-analysis, an exhaustive search showed that half of the studies that definitely had collected survival data failed to provide information that would be sufficient for any additional analysis. Differential measurement error through lack of blinded measurements and the flexible use of definitions for outcome measurements and cutoff points for interpretation of prognostic markers can introduce additional bias and create spurious findings ( 1 , 117 ) . In fact, many prognostic studies target outcomes other than mortality, and these outcomes can be susceptible to selective choice of definitions. Even mortality, the most definitive clinical endpoint possible, may occasionally be manipulated (e.g., with cause-specific deaths, including different variants of nonlethal disease progression, or with variable censoring methods).

In the absence of a single large study, these deficiencies may be overcome by prospective registration of data on specific prognostic factors and by meta-analyses of prospectively collected individual-level data ( 118 ) . Some fields are already moving toward standardized reporting and archiving. Standardization is particularly important for discovery-driven research, where hundreds or thousands of potential molecular predictors may be measured in minimal time ( 119 ) . Yet comprehensive registration often will not be feasible. Because prognostic factors are easy to probe in clinical samples without any requirements for rigorous study design, many investigators will continue to generate data, and much of the data will remain unpublished or will be selectively presented.

Our study had several limitations. It is almost certain that some pertinent information could not be retrieved, and it is not possible to know the effect of including these missing data. Moreover, despite our efforts to standardize data, complete standardization was not feasible. It was not possible, for example, to synthesize standardized information on hazard ratios or to find data with the same TP53 cutoff across all studies. These limitations point to the unavoidable problems that other meta-analyses of prognostic factors are likely to face, even with the best of intentions and efforts.

Given these unavoidable biases, meta-analyses of the prognostic literature offer an opportunity to scrutinize the possible extent of bias and uncertainty. This type of investigation is even more important than arriving at summary estimates ( 120 ) . Our results indicate that the conduct and reporting of prognostic meta-analyses need to be improved ( 121 ) . Otherwise, meta-analyses may spuriously shrink the confidence intervals of biased findings. Searches should be broad, including as many studies, because much of the relevant information from the analyses may be buried in the small print or barely alluded to in the published papers. Efforts to retrieve additional unpublished information are strongly recommended. It would be useful to know how much information is missing at a minimum, and it may be prudent to contact all investigators who are known to work in the wider field. Standardization of outcomes and prognostic factors across studies may further reduce bias. Bias diagnostics should be performed, but they are not definitive.

Finally, a prognostic marker may be of scientific interest but may be clinically useless if the conveyed prognostic information has also been captured by other prognostic factors that are more easily assessed. For example, the prognostic association between some molecular markers and mortality may be entirely mediated through parameters such as lymph node involvement or tumor size. Our empirical evaluation suggests that properly and consistently adjusted estimates are the exception in the prognostic factor literature and in meta-analyses thereof. Incorporation of molecular and other predictors into clinical practice should require large-scale validation in both unadjusted and adjusted analyses.

We conclude that major reporting biases may be operating in the literature of prognostic markers for cancer outcomes. Unless they are recognized and dealt with appropriately, these biases may create a spurious knowledge base ( 122 ) of cancer predictors that may be of no use and may be potentially harmful.

References