Original Article
Searching for unpublished trials in Cochrane reviews may not be worth the effort

https://doi.org/10.1016/j.jclinepi.2008.09.010Get rights and content

Abstract

Objective

To assess the value of searching for unpublished data by exploring the extent to which Cochrane reviews include unpublished data and by evaluating the quality of unpublished trials.

Study Design and Setting

We screened all 2,462 completed Cochrane reviews published since 2000 in the Cochrane Database of Systematic Reviews Issue 3, 2006. In a random sample (n = 61) of 292 reviews, including unpublished trials, we studied all 116 references.

Results

Unpublished trials make up 8.8% of all included trials in our sample. Thirty-eight percent of the “unpublished” trials have in fact been published. Allocation concealment was “unclear” or not adequate in 54.3% and 61.3% reported blinding. In 47.2% reported withdrawal rates were >20%. Trials that were eventually published had larger mean population sizes (P-value, 0.02). Of the reported sponsors, 87.3% were drug companies. Methodological quality and publication bias are mentioned in half of the reviews and explored in a third. Quality ratings did not have consequences for pooling, because 82.8% was included in the forest plots.

Conclusions

A minority of Cochrane reviews include “unpublished trials” and many of these are eventually published. Truly unpublished studies have poor or unclear methodological quality. Therefore, it may be better to invest in regular updating of reviews, rather than in extensive searching for unpublished data.

Introduction

The Cochrane Database of Systematic Reviews (CDSR) is one of the most important sources of reviews and meta-analyses in the field of clinical medicine. The results of Cochrane reviews are consulted by clinicians, guideline developers, and health policy makers and contribute to the growing body of “medical evidence” [1]. Reviewers use a standardized methodology as described in the reviewer's handbook [2] that makes the review process and the synthesis of evidence transparent and increases the methodological rigor [3]. Minimizing bias related to the studies that make up the review and the way these studies are collected is an important issue in Cochrane reviews. The Handbook states that, “A comprehensive search for relevant RCTs, which seeks to minimize bias, is one of the essential steps in doing a systematic review….” [2]. Because studies with significant results (in favor of the new treatment) have a higher chance of being published [4], [5], this includes attempts to locate unpublished studies [2]. Excluding them can lead to exaggerated estimates of the effectiveness of an intervention in the meta-analysis [6], [7].

Already in 1993 Cook et al discussed the controversies related to including unpublished data in meta-analyses [8]. They point to the lack of information and/or sufficient methodological quality of these unpublished studies and recommend that both published and unpublished data should be subject to the same rigorous methodological evaluation. Presenting the results with and without inclusion of unpublished data in a sensitivity analysis could improve the quality of meta-analyses [9]. An HTA report shares these concerns and mentions that by including (unpublished) trials of low quality bias could even be introduced, rather than prevented [10].

We wanted to know whether the use of unpublished material is a valuable contribution to Cochrane reviews. Therefore, we explored the quantity and methodological quality of unpublished data currently used in Cochrane reviews.

Section snippets

Methods

We retrieved all completed reviews listed in the Cochrane Library issue 3 of 2006 from the Wiley InterScience electronic interface. In all reviews published since the first issue of 2000, we checked the reference lists for included unpublished studies (marked as “unpublished data only”). We focused on the references to “unpublished data only,” rather than additional data that are obtained from the authors of published articles and are marked as “published and unpublished data.” By means of

Reviews including unpublished references

We identified 2,689 completed reviews in the Cochrane Library 2006 issue 3, of which 2,462 have been published since 2000. References to unpublished data are included in 292 reviews (11.9% of the reviews). (Fig. 1) Eight reviews are entirely based on unpublished material and are excluded from this analysis because we were interested in the role of unpublished trials in reviews based on published and unpublished data. The remaining 284 reviews use data from both published and unpublished trials.

Discussion

Our study shows that searching for unpublished trials in Cochrane reviews does not give a high yield and that the methodological quality of unpublished trials raises concern.

We found that only 11.9% of all recent Cochrane reviews included references to unpublished data. In an earlier literature review of 150 meta-analyses indexed in MEDLINE (1988–1999), 30.7% included unpublished data in their analysis [8]. Egger et al. report that only 7% of all trials included in their sample of 159

Conclusion

Our review shows that less than 10% of Cochrane reviews include unpublished data and references to unpublished studies make up only a small proportion of all included studies. The fact that a third of these “unpublished” references could be located as journal publications suggests that not including them in the review before formal publication would merely delay the evidence synthesis. The poor methodological transparency and quality of the trials that remain unpublished is an important concern

Acknowledgments

No funding was obtained for this literature review.

References (49)

  • A.R. Jadad et al.

    Methodology and reports of systematic reviews and meta-analyses. A comparison of Cochrane reviews with articles published in paper-based journals

    JAMA

    (1998)
  • K. Dickersin

    How important is publication bias? A synthesis of available data

    AIDS Educ Prev

    (1997)
  • D.K. Benjamin et al.

    Peer-reviewed publication of clinical trials completed for pediatric exclusivity

    JAMA

    (2006)
  • N.S. Abraham et al.

    Systematic review: the methodological quality of trials affects estimates of treatment efficacy in functional (non-ulcer) dyspepsia

    Aliment Pharmacol Ther

    (2004)
  • D.J. Cook et al.

    Should unpublished data be included in meta-analysis? Current convictions and controversies

    JAMA

    (1993)
  • M. Egger et al.

    How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study

    Health Technol Assess

    (2003)
  • E.T. Crumley et al.

    Which resources should be used to identify RCT/CCTs for systematic reviews: a systematic review

    BMC Med Res Methodol

    (2005)
  • Cochrane Collaboration Depression, Anxiety & Neurosis Review Group. http://web1.iop.kcl.ac.uk/IoP/ccdan/qrs.htm....
  • A.R. Jadad et al.

    Guides for reading and interpreting systematic reviews. II. How did the authors find the studies and assess their quality?

    Arch Pediatr Adolesc Med

    (1998)
  • C. Lefebvre et al.

    Identifying randomised trials

  • J. Lau et al.

    The case of the misleading funnel plot

    BMJ

    (2006)
  • M. Toma et al.

    Transition from meeting abstract to full-length journal article for randomised controlled trials

    JAMA

    (2006)
  • I. Roberts

    An amnesty for unpublished trials

    BMJ

    (1998)
  • C. De Angelis et al.

    Clinical trial registration: a statement from the International Committee of Medical Editors

    JAMA

    (2004)

    • Cited by (69)

    • A meta-analysis to determine factors associated with the severity of enteritis in Atlantic salmon (Salmo salar) fed soybean meal-based diets

      2022, Aquaculture
      Citation Excerpt :

      There is convincing evidence that results that are statistically non-significant and unfavourable to the experimental intervention are less likely to be published than statistically significant results, and hence are less easily identified by the systemic review (Higgins et al., 2019). Van Driel et al. (2009) concluded that the difficulty in retrieving unpublished work could lead to selection bias. The group of authors that argued against the inclusion of unpublished data in a meta-analysis based their position on the following reasons.

    • R0 resection rates with underwater EMR better than conventional EMR: an updated analysis

      2022, Gastrointestinal Endoscopy

    • A meta-analysis of suicide prevention programs for school-aged youth

      2021, Children and Youth Services Review
      Citation Excerpt :

      In order to include potential grey literature, we also searched the first 600 dissertation results on ProQuest, completed reference checks, and contacted authors to locate potential unpublished studies. The initial search was limited to peer-reviewed articles as van Driel et al. (2009) found that grey literature is often of a lower methodical quality and that high-quality unpublished trials are eventually published. The authors recommend that meta-analyses focus on published trials but suggest a need for regularly updating meta-analyses.

    • Contacting of authors modified crucial outcomes of systematic reviews but was poorly reported, not systematic, and produced conflicting results

      2019, Journal of Clinical Epidemiology

    • Systematic review and meta-analysis of the efficacy of reproductive management practices used to induce resumption of ovarian cyclical activity in anestrous does

      2019, Preventive Veterinary Medicine

    • Systematic reviews and meta-analysis in nutrition research

      2019, British Journal of Nutrition
    View all citing articles on Scopus
    View full text
    Copyright © 2009 Elsevier Inc. All rights reserved.