Elsevier

The Lancet

Volume 356, Issue 9237, 7 October 2000, Pages 1228-1231
The Lancet

Articles
Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses?

https://doi.org/10.1016/S0140-6736(00)02786-0Get rights and content

Summary

Background

The inclusion of only a subset of all available evidence in a meta-analysis may introduce biases and threaten its validity; this is particularly likely if the subset of included studies differ from those not included, which may be the case for published and grey literature (unpublished studies, with limited distribution). We set out to examine whether exclusion of grey literature, compared with its inclusion in meta-analysis, provides different estimates of the effectiveness of interventions assessed in randomised trials.

Methods

From a random sample of 135 meta-analyses, we identified and retrieved 33 publications that included both grey and published primary studies. The 33 publications contributed 41 separate meta-analyses from several disease areas. General characteristics of the meta-analyses and associated studies and outcome data at the trial level were collected. We explored the effects of the inclusion of grey literature on the quantitative results using logistic-regression analyses.

Findings

33% of the meta-analyses were found to include some form of grey literature. The grey literature, when included, accounts for between 4·5% and 75% of the studies in a meta-analysis. On average, published work, compared with grey literature, yielded significantly larger estimates of the intervention effect by 15% (ratio of odds ratios=1·15 [95% CI 1·04—1·28]). Excluding abstracts from the analysis further compounded the exaggeration (1·33 [1·10–1·60]).

Interpretation

The exclusion of grey literature from metaanalyses can lead to exaggerated estimates of intervention effectiveness. In general, meta-analysts should attempt to identify, retrieve, and include all reports, grey and published, that meet predefined inclusion criteria.

Introduction

A meta-analysis is multifactorial. Decisions need to be made about how to handle various factors, such as language of publication, quality, and publication status, at the individual study level. Within the domain of publication status, a major factor to consider is the inclusion of grey literature (ie, studies that are unpublished, have limited distribution, and/or are not included in bibliographical retrieval system).1

The inclusion of grey literature in a meta-analysis may help to overcome some of the problems of publication bias, and provide a more complete and objective answer to the question under consideration. However, it has been reported that only 31% of published meta-analyses include grey literature.2 This omission may be because the nature of grey literature makes its exclusion more convenient; it is difficult to retrieve, it is frequently incomplete, and its quality may be difficult to assess. We aim to provide empirical evidence about the impact of the exclusion of grey literature from meta-analyses on the estimate of intervention effectiveness.

Section snippets

Selection of meta-analyses

A sample of 135 meta-analyses were drawn randomly from an existing database of 455 meta-analyses of randomised clinical trials.3, 4 The database was established in 1996 through MEDLINE searches from 1966 to 1995 with a detailed search strategy assembled with the aid of an information specialist.

Eligibility criteria

To be eligible, a study has to be deemed a meta-analysis (included pooled analyses of the results of several independent primary studies), the associated studies had to be identifiable, and at least one

Results

From the original 135 meta-analyses retrieved, we could not establish whether grey literature was included in 12 cases (11 poorly referenced, one not retrieved). An additional eight meta-analyses were excluded because they did not fit the definition of a meta-analysis. Of the remaining 115, 38 (33%) were found to contain at least one item of grey literature. Five of these meta-analyses failed to meet the inclusion criteria. For all subsequent investigations, we used the sample of 33

Discussion

Our results suggest that the exclusion of grey literature from meta-analysis may result in an overestimate of an intervention effect by an average of 12%. In respect of quinine for nocturnal leg cramp, Hing and colleagues published a meta-analysis of four published studies.11 These investigators then repeated their meta-analysis with three previously unidentified US Food and Drug Administration (FDA) documents. They report that the published trials, compared with the FDA documents, consistently

References (16)

There are more references available in the full text version of this article.

Cited by (618)

  • Catheter-based renal artery denervation: facts and expectations

    2023, European Journal of Internal Medicine
View all citing articles on Scopus
View full text