Random error in cardiovascular meta-analyses: how common are false positive and false negative results?

Background: Cochrane reviews are viewed as the gold standard in meta-analyses given their efforts to identify and limit systematic error which could cause spurious conclusions. The potential for random error to cause spurious conclusions in meta-analyses is less well appreciated.

Methods: We examined all reviews approved and published by the Cochrane Heart Group in the 2012 Cochrane Library that included at least one meta-analysis with 5 or more randomized trials. We used trial sequential analysis to classify statistically significant meta-analyses as true positives if their pooled sample size and/or their cumulative Z-curve crossed the O'Brien-Fleming monitoring boundaries for detecting a RRR of at least 25%. We classified meta-analyses that did not achieve statistical significance as true negatives if their pooled sample size was sufficient to reject a RRR of 25%.

Results: Twenty three (41%) of the 56 meta-analyses reported statistically significant results, and 19 (83%) were true positives. Of the 33 non-statistically significant meta-analyses, 12 (36%) were true negatives. Overall, 25 (45%) of the 56 published Cochrane reviews were too small to detect/rule out an effect size of at least 25% - 12 were acknowledged as such by their authors. Of the 22 meta-analyses which were reported to be conclusive by their authors, 12 (55%) contained insufficient data to detect/rule out a 25% relative treatment effect.

Conclusion: False positive and false negative meta-analyses are common but infrequently recognized, even among methodologically robust reviews published by the Cochrane Heart Group. Meta-analysts and readers should incorporate trial sequential analysis when interpreting results.