Review Article
A systematic review of tools used to assess the quality of observational studies that examine incidence or prevalence and risk factors for diseases

https://doi.org/10.1016/j.jclinepi.2010.04.014Get rights and content

Abstract

Objective

To create a comprehensive evaluation of checklists and scales used to evaluate observational studies that examine incidence or prevalence and risk factors for diseases.

Study Design

We did a literature search of several databases to abstract format, content, development, and validation of the tools.

Results

We identified 46 scales and 51 checklists. Forty-seven of these tools were created for therapeutic studies, 48 for risk factors, and 5 for incidence studies. Forty-seven percent were modifications of previously published peer-reviewed appraisals, 18% were developed based on methodological standards, and 35% did not report development. Twenty-two percent reported reliability and 10% the validation procedure. Tools did not discriminate poor reporting vs. methodological quality of studies or external vs. internal validity; 35% categorize quality by the presence of predefined major flaws in design or by total score from the scale. Level of evidence was proposed in 22% of the tools by criteria of causality or internal validity of the studies. Evaluation required different degrees of subjectivity.

Conclusions

Format, length, and content varied substantially across available checklists and scales. Development, validation, and reliability were not consistently reported. Transparent objective quality assessments should be developed in the future.

Section snippets

Background

What is new?

We conducted a comprehensive evaluation of the published checklists and scales to assess quality of observational studies. We analyzed content, development, validation, reliability, and applicability of the tools for the studies of incidence, prevalence, or risk factors of chronic diseases.

Systematic reviews of medical literature became essential tools for making informed decisions in preventive medicine and public health [1], [2]. Bias in primary studies causes bias in systematic

Literature search

We searched MEDLINE via PubMed [8] and via Ovid MEDLINE, the Cochrane Library [9] and Working Groups, WorldCat [10] and Scirus [11] to find scales or checklists used to appraise the quality of observational studies that were published in English from 1966 through June 2008 (exact search string in Supplementary Table 1).

Eligibility criteria

Three investigators independently decided on the eligibility of the studies according to recommendations from the Cochrane Manual for Systematic Reviews [12]. We reviewed

Results

We identified 397 publications that were relevant to the collaborative project to develop criteria to assess quality and reporting standards for epidemiologic nontherapeutic studies (Supplementary Fig. 1). In this article, we include 96 scales and checklists currently used to assess the quality of observational studies (Supplementary Table 2).

The goals of the tools we appraised varied. The appraisals were created for observational studies of risk factors (48 tools), prevalence or incidence

Discussion

Despite increased interest in and need for quality assessment, there is only limited information about the development, reliability, and validation of the available assessment tools. Variability of the format and applicability of the tools may suggest dissatisfaction with available appraisals. We could not find significant differences between scales and checklists and applicability to studies of incidence/prevalence or risk factors. Therefore, evaluation of the content of scales or checklists

Conclusions

Published checklists and scales used to evaluate observational studies differ by content, format, validity, and applicability to studies of incidence, prevalence, or risk factors of chronic diseases. The available tools do not discriminate poor reporting from quality of studies and do not give separate conclusions about external and internal validity. No consensus exists around individual criteria of validity and ranking of overall quality. Future collaborative efforts are needed to develop

Acknowledgments

We would like to thank our reviewers David Atkins, MD, John Hoey, MD, and Christine Laine, MD, for reviewing and commenting on the draft and Carmen Kelly, PharmD, our Task Order Officer from AHRQ for her guidance throughout the project. We also want to thank the librarian, Judith Stanke, for her contributions to the literature search; research assistants Emily Zabor candidate for MS in biostatistics, and Akweley Ablorh, candidate for MS in biostatistics, for the data abstraction, quality

References (98)

  • M.J. Lichtenstein et al.

    Guidelines for reading case-control studies

    J Chronic Dis

    (1987)
  • S. Zaza et al.

    Data collection instrument and procedure for systematic reviews in the Guide to Community Preventive Services. Task Force on Community Preventive Services

    Am J Prev Med

    (2000)
  • J.M. Esdaile et al.

    Observational studies of cause-effect relationships: an analysis of methodologic problems as illustrated by the conflicting data for the role of oral contraceptives in the etiology of rheumatoid arthritis

    J Chronic Dis

    (1986)
  • J.A. Borghouts et al.

    The clinical course and prognostic factors of non-specific neck pain: a systematic review

    Pain

    (1998)
  • R.P. Harris et al.

    Current methods of the US Preventive Services Task Force: a review of the process

    Am J Prev Med

    (2001)
  • G.G.M. Scholten-Peeters et al.

    Prognostic factors of whiplash-associated disorders: a systematic review of prospective cohort studies

    Pain

    (2003)
  • S.J. Rangel et al.

    Development of a quality assessment scale for retrospective clinical studies in pediatric surgery

    J Pediatr Surg

    (2003)
  • M. Jabbour et al.

    Life support courses: are they effective?

    Ann Emerg Med

    (1996)
  • C.M. Kreulen et al.

    Meta-analysis of anterior veneer restorations in clinical studies

    J Dent

    (1998)
  • A.C. Tricco et al.

    Few systematic reviews exist documenting the extent of bias: a systematic review

    J Clin Epidemiol

    (2008)
  • D.C. Hadorn et al.

    Rating the quality of evidence for clinical practice guidelines

    J Clin Epidemiol

    (1996)
  • A. Conde-Agudelo et al.

    Maternal infection and risk of preeclampsia: systematic review and metaanalysis

    Am J Obstet Gynecol

    (2008)
  • M. Pavia et al.

    Association between fruit and vegetable consumption and oral cancer: a meta-analysis of observational studies

    Am J Clin Nutr

    (2006)
  • R.H. Groenwold et al.

    Poor quality of reporting confounding bias in observational intervention studies: a systematic review

    Ann Epidemiol

    (2008)
  • M. Ezzati et al.

    Estimates of global and regional potential health gains from reducing multiple major risk factors

    Lancet

    (2003)
  • R. BeLue et al.

    African Americans and participation in clinical trials: differences in beliefs and attitudes by gender

    Contemp Clin Trials

    (2006)
  • D.M. Fox

    Evidence of evidence-based health policy: the politics of systematic reviews in coverage decisions

    Health Aff (Millwood)

    (2005)
  • J. Lavis et al.

    Towards systematic reviews that inform health care management and policy-making

    J Health Serv Res Policy

    (2005)
  • L.A. Bero et al.

    How consumers and policymakers can use systematic reviews for decision making

    Ann Intern Med

    (1997)
  • K.S. Chan et al.

    Systematic reviews for evidence-based management: how to find them and what to do with them

    Am J Manag Care

    (2004)
  • Agency for Healthcare Research and Quality

    Systems to rate the strength of scientific evidence

    (2002)
  • S. Sanderson et al.

    Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography

    Int J Epidemiol

    (2007)
  • National Center for Biotechnology Information, U.S. National Library of Medicine. Bethesda, MD. Available at:...
  • The Cochrane database of systematic reviews. 2000-2010. New York:...
  • OCLC

    WorldCat

    (2001)
  • Elsevier Science (Firm)

    Scirus for scientific information only

    (2001)
  • J. Higgins et al.

    Cochrane handbook for systematic reviews of interventions

    (2008)
  • D.F. Stroup et al.

    Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis of Observational Studies in Epidemiology (MOOSE) group

    JAMA

    (2000)
  • J. Higgins et al.

    The Cochrane handbook for systematic reviews of interventions

  • A. Aschengrau et al.

    Essentials of Epidemiology in Public Health

    (2003)
  • BMJ Publishing Group

    Clinical evidence

  • S.B. Hulley

    Designing clinical research: an epidemiologic approach

    (2001)
  • Centers for Disease Control and Prevention (U.S.) et al.

    Principles of epidemiology in public health practice: an introduction to applied epidemiology and biostatistics

    (2006)
  • Stata Corporation

    Stata base reference manual

    (2007)
  • SAS Institute

    SAS com

    (2001)
  • P.L. Loney et al.

    Critical appraisal of the health research literature: prevalence or incidence of a health problem

    Chronic Dis Can

    (1998)
  • M.G. Woodbury et al.

    Prevalence of pressure ulcers in Canadian healthcare settings

    Ostomy Wound Manage

    (2004)
  • S.H. Downs et al.

    The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions

    J Epidemiol Community Health

    (1998)
  • K. Slim et al.

    Methodological index for non-randomized studies (MINORS): development and validation of a new instrument

    ANZ J Surg

    (2003)
  • Cited by (164)

    View all citing articles on Scopus
    View full text