Overviews of reviews often have limited rigor: a systematic review

Objective: To examine published overviews of systematic reviews in terms of descriptive and methodological characteristics.

Study design and setting: MEDLINE, Database of Abstracts of Reviews of Effects, the Cochrane Database of Systematic Reviews, and several Health Technology Assessment databases were searched for overviews of reviews up to February 2012. We extracted data from the methods and results sections of the included overviews. These data were analyzed descriptively as frequencies or medians and interquartile ranges.

Results: We included 126 overviews of reviews. According to our sample, publication rates for overviews have risen in the last decade. The quality of the included reviews was systematically appraised in 64% of the overviews. The most commonly used assessment tools were the Overview Quality Assessment Questionnaire (26%), Assessment of Multiple Systematic Reviews (11%), and Preferred Reporting Items for Systematic Reviews and Meta-Analyses (5%). Only three of 18 overviews restricted to Cochrane reviews in our sample performed a quality assessment. Strategies to deal with discordant reviews were reported in 5% of the overviews. Searches for additional primary studies were conducted in 5% of the overviews.

Conclusion: Overviews of reviews often lack methodological rigor. Methodological standards and reporting guidelines for overviews are needed to improve the quality of this new publication type.