Systematic reviews incorporating evidence from nonrandomized study designs: reasons for caution when estimating health effects

Abstract

Systematic reviews that include nonrandomized studies (NRS) face a number of logistical challenges. However, the greatest threat to the validity of such reviews arises from the differing susceptibility of randomized controlled trials (RCTs) and NRS to selection bias. Groups compared in NRS are unlikely to be balanced because of the reasons leading study participants to adopt different health behaviours or to be treated differentially. Researchers can try to minimize the susceptibility of NRS to selection bias both at the design stage, for example, by matching participants on key prognostic factors, and during data analysis, for example, by regression modelling. However, because of logistical difficulties in matching, imperfect knowledge about the relationships between prognostic factors and between prognostic factors and outcome, and measurement limitations, it is inevitable that estimates of effect size derived from NRS will be confounded to some extent. Researchers, reviewers and users of evidence alike need to be aware of the consequences of residual confounding. In poor quality RCTs, selection bias tends to favour the new treatment being evaluated. Selection bias need not necessarily lead to systematic bias in favour of one treatment but, even if it acts in an unpredictable way, it will still give rise to additional, nonstatistical uncertainty bias around the estimate of effect size. Systematic reviews of NRS studies run the risk of compounding these biases. Nutritional choices and uptake of health education about nutrition are very likely to be associated with potential confounding factors. Therefore, pooled estimates of the effects of nutritional exposures and their confidence intervals are likely to be misleading; reviewers need to take into account both systematic and uncertainty bias.