You are here

Article Text

Article menu

Download PDFPDF

XML
Evidence based practice
Original research
Examination of CIs in health and medical journals from 1976 to 2019: an observational study
  1. Adrian Gerard Barnett1,
  2. Jonathan D Wren2,3
  1. 1 Institute of Health and Biomedical Innovation, Queensland University of Technology, Kelvin Grove, Queensland, Australia
  2. 2 Arthritis and Clinical Immunology Research Program, Division of Genomics and Data Sciences, Oklahoma Medical Research Foundation, Oklahoma City, Oklahoma, USA
  3. 3 Department of Biochemistry and Molecular Biology, University of Oklahoma Health Sciences Center, Oklahoma City, Oklahoma, USA
  1. Correspondence to Dr Adrian Gerard Barnett; a.barnett{at}qut.edu.au

Abstract

Objectives Previous research has shown clear biases in the distribution of published p values, with an excess below the 0.05 threshold due to a combination of p-hacking and publication bias. We aimed to examine the bias for statistical significance using published confidence intervals.

Design Observational study.

Setting Papers published in Medline since 1976.

Participants Over 968 000 confidence intervals extracted from abstracts and over 350 000 intervals extracted from the full-text.

Outcome measures Cumulative distributions of lower and upper confidence interval limits for ratio estimates.

Results We found an excess of statistically significant results with a glut of lower intervals just above one and upper intervals just below 1. These excesses have not improved in recent years. The excesses did not appear in a set of over 100 000 confidence intervals that were not subject to p-hacking or publication bias.

Conclusions The huge excesses of published confidence intervals that are just below the statistically significant threshold are not statistically plausible. Large improvements in research practice are needed to provide more results that better reflect the truth.

  • p-values
  • statistical significance
  • confidence intervals
  • p-hacking
  • publication bias

This is an open access article distributed in accordance with the Creative Commons Attribution 4.0 Unported (CC BY 4.0) license, which permits others to copy, redistribute, remix, transform and build upon this work for any purpose, provided the original work is properly cited, a link to the licence is given, and indication of whether changes were made. See: https://creativecommons.org/licenses/by/4.0/.

https://doi.org/10.1136/bmjopen-2019-032506

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    View Full Text

    Footnotes

    • Twitter @aidybarnett

    • Contributors AGB and JW conceived the idea. JW wrote the programs to extract the data. AGB wrote the analyses code and the first draft of the paper with comments from JW. AGB is responsible for the overall content and is the guarantor.

    • Funding AGB was supported by Queensland University of Technology and the National Health and Medical Research Council grant number APP1117784.

    • Competing interests None declared.

    • Patient consent for publication Not required.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Data availability statement Data are available in a public, open access repository.