Poor Quality of Reporting Confounding Bias in Observational Intervention Studies: A Systematic Review
Introduction
In recent years, several guidelines have been developed to improve the quality of the reporting of empirical studies. Among those, the Consolidated Standards of Reporting Trials (CONSORT) statement was established to ensure that researchers report features of randomized controlled trials that must be considered when appraising the quality of such trials 1, 2. Regarding observational (i.e., nonrandomized) intervention studies, a similar guideline has recently been published, the so-called STROBE (STrenghtening the Reporting of OBservational studies in Epidemiology) statement, which aims to improve the reporting of these studies (3).
The key difference between randomized and observational intervention studies is the allocation of treatment which is, by definition, a chance phenomenon in the former. Although randomized studies are considered the gold standard to assess effects of medical interventions, observational studies can be reasonable alternatives, e.g., when financial, logistic, or ethical reasons preclude a randomized trial. Observational studies follow daily medical practice in that patients with a clear indication typically have a poorer prognosis and are more likely to receive the intervention and will bias the effect estimate when not prevented or taken into account 4, 5.
Also, different settings can be compared. For example, two hospitals may perform different surgical procedures for the same indication. Then, allocation of treatment may depend not only on severity of disease but also on area of residence. Different methods in the design and analysis phase of observational intervention studies have been proposed to achieve this 6, 7. However, unobserved confounders can not be adjusted for 8, 9. For example, the discrepancy between results from randomized and observational studies on the putative effect of postmenopausal hormone therapy on the risk of cardiovascular disease in women may, in part, be due to unobserved confounding 10, 11. Therefore, to be able to value findings from observational studies, adequate reporting of confounding bias is essential, which is emphasized in the STROBE statement. Earlier studies indicate that the quality of reporting of confounding in medical journals may be low 12, 13; therefore, we set out to systematically review the literature on the current quality of reporting of confounding in observational studies on medical interventions published in general medical and epidemiological articles.
Section snippets
Data Sources and Searches
We searched the MEDLINE database from January 2004 through April 2007 to identify observational intervention studies (i.e., cohort, longitudinal, prospective, follow-up, cross-sectional, retrospective, and case-control) in five general medical journals and five epidemiological journals to study current reporting on confounding. The five general medical journals included New England Journal of Medicine, The Lancet, Journal of the American Medical Association, Annals of Internal Medicine, and
Results
The Medline search resulted in 2993 publications, and 174 (5.8%) articles were included in the analysis (see Fig. 1). Cohort studies were more frequent (120, 69.0%) than case-control studies (54, 31.0%). Of the 174 articles, 47 were published in 2004, 57 in 2005, 47 in 2006, and 23 were published in 2007 between January and April.
Among the 93 articles in general medical journals, 70 articles (75.3%) were on cohort studies. Among the 81 articles from epidemiological journals, 51 (63.0%) were on
Discussion
Our study shows that the quality of reporting on confounding in observational intervention studies is rather poor as indicated by a mediocre score of 4 of 8 on our developed item list. Importantly, our results confirm that, during the last few years, the quality of reporting of confounding did not significantly improve, either for the high-impact general medical journals or for the epidemiological journals.
These data are in accordance with a previous literature review from our group that showed
References (25)
- et al.
The CONSORT statement: Revised recommendations for improving the quality of reports of parallel-group randomised trials
Lancet
(2001) - et al.
Methods to assess intended effects of drug treatment in observational studies are reviewed
J Clin Epidemiol
(2004) - et al.
A simulation study of the number of events per variable in logistic regression analysis
J Clin Epidemiol
(1996) - et al.
Reporting in randomized clinical trials improved after adoption of the CONSORT statement
J Clin Epidemiol
(2007) - et al.
Improving the quality of reporting of randomized controlled trials: the CONSORT statement
JAMA
(1996) - et al.
Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement: guidelines for reporting observational studies
BMJ
(2007) - et al.
Effectiveness of influenza vaccination in the community-dwelling elderly
N Engl J Med
(2007) - et al.
Confounding by indication in non-experimental evaluation of vaccine effectiveness: the example of prevention of influenza complications
J Epidemiol Community Health
(2002) - et al.
Readers guide to crtical appraisal of cohort studies: 3. Analytical strategies to reduce confounding
BMJ
(2005) - et al.
Assessment of control of confounding by indication in observational studies
J Am Geriatr Soc
(1999)
Sensitivity analysis and external adjustment for unmeasured confounders in epidemiologic database studies of therapeutics
Pharmacoepidemiol Drug Saf
Understanding the divergent data on postmenopausal hormone therapy
N Engl J Med
Cited by (66)
High prevalence of spin was found in pharmacovigilance studies using disproportionality analyses to detect safety signals: a meta-epidemiological study
2021, Journal of Clinical EpidemiologyCitation Excerpt :Furthermore, more than half of the abstracts failed to acknowledge any limitations of their findings. These findings are in agreement with previous reports of sub-optimal reporting of bias or confounding in observational studies of medical interventions [17,18]. This failure, in combination with an over-interpretation of results asserting a causal link, carries a significant risk of misleading readers.
Consideration of confounding in epidemiologic studies assessing alcohol consumption on the risk of breast cancer: A brief report
2020, Chemico-Biological InteractionsConsideration of confounding was suboptimal in the reporting of observational studies in psychiatry: a meta-epidemiological study
2020, Journal of Clinical EpidemiologyInterpretation of epidemiologic studies very often lacked adequate consideration of confounding
2018, Journal of Clinical EpidemiologyThe reporting of studies using routinely collected health data was often insufficient
2016, Journal of Clinical EpidemiologyMethods and processes of developing the strengthening the reporting of observational studies in epidemiology − veterinary (STROBE-Vet) statement
2016, Preventive Veterinary Medicine
This study is part of a personal grant of Dr. E. Hak to study confounding in observational intervention studies by the Netherlands Scientific Organization (VENI no. 916.56.109).