Review Article
Retrospective cohort study highlighted outcome reporting bias in UK publicly funded trials

https://doi.org/10.1016/j.jclinepi.2011.03.013Get rights and content

Abstract

Objective

To assess outcome reporting bias and dissemination bias in trials funded by the National Health System (NHS) Health Technology Assessment (HTA) program.

Study Design and Setting

A retrospective cohort study of HTA monographs and corresponding journal publications including all clinical effectiveness randomized controlled trials published as HTA monographs between 1999 and 2005 by the NHS HTA program.

Results

There was a higher median P-value (P = 0.33, interquartile range [IQR]: 0.02–0.54) among trials without a journal publication compared with those with a journal publication (P = 0.14, IQR: 0.007–0.43), although the difference was not statistically significant (Mann-Whitney U test, z = −0.70; P = 0.48). A higher proportion of statistically significant findings were reported in journal articles when compared with the outcomes reported in the HTA monographs. Trials published in general medical journals tended to have smaller P-values (median: 0.05, IQR: 0.001–0.22) than those published in more specialist journals (median: 0.33 IQR: 0.008–0.58), although this result was not significant (Mann-Whitney U test, z = −1.63; P = 0.10).

Conclusions

Among journal-published trials, there were a greater proportion of statistically significant findings included in the journal reports compared with those in the HTA monographs.

Section snippets

Background

What is new?

Key finding

  1. Some trials funded by the Health Technology Assessment (HTA) research program have a tendency toward selective publication of their findings.

What this adds to what was known?
  1. Individuals who include HTA-funded trials in their review should retrieve the HTA monograph and not rely on the journal article.

What is the implication, what should change now?
  1. Authors and journal editors should take care not to publish selectively based on positive finding.

Since the 1950s, it has been recognized that research with statistically significant or positive findings is more likely

Source of trials

The UK National Health System (NHS) HTA program was set up in 1993 as part of the National Institute for Health Research. The HTA program funds research in the NHS, which will benefit the UK patient population in terms of their clinical outcomes and quality of life and is a major commissioner of trials in the United Kingdom. A key requirement for obtaining funding from the HTA is that grant recipients are required to provide a detailed report of their trial, which is then peer reviewed and

Results

We identified 310 HTA monographs that were published before December 2005, of which 44 (14%) were RCTs assessing the clinical effectiveness of an intervention (Fig. 1). From the 44 eligible HTA monographs, 37 (84%) had published the main study findings in a separate journal article.

Discussion

This study has examined the potential for outcome reporting bias and dissemination bias in trials funded by the HTA program. The results of this study show there is a tendency for trials to underreport nonsignificant outcomes in journal publications with the consequent outcome that statistically significant findings tend to be more widely disseminated. This dissemination bias means that clinicians are more likely to read trials with statistically significant findings and may alter their

References (57)

  • C.J. Torgerson

    Publication bias: the Achilles' heel of systematic reviews?

    Br J Educ Stud

    (2006)
  • S. Hopewell et al.

    Publication bias in clinical trials due to statistical significance or direction of trial results

    Cochrane Database Syst Rev

    (2009)
  • S. Hahn et al.

    Investigation of within-study selective reporting in clinical research: follow-up of applications submitted to a local research ethics committee

    J Eval Clin Pract

    (2002)
  • A.W. Chan et al.

    Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles

    JAMA

    (2004)
  • A.W. Chan et al.

    Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research

    Can Med Assoc J

    (2004)
  • A.W. Chan et al.

    Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors

    BMJ

    (2005)
  • P.R. Williamson et al.

    Outcome selection bias in meta-analysis

    Stat Methods Med Res

    (2005)
  • McCormack K, Scott N, Grant A. Outcome reporting bias and individual patient data meta-analysis: a case study in...
  • ISI Web of Knowledge Journal Citation Reports. 2007 JCR Science Edition. Available at http://wok.mimas.ac.uk/. Accessed...
  • A. Boland

    A randomised controlled trial to evaluate the clinical and cost-effectiveness of Hickman line insertions in adult cancer patients by nurses

    Health Technol Assess

    (2003)
  • T. Cochrane

    Randomised controlled trial of the cost-effectiveness of water-based therapy for lower limb osteoarthritis

    Health Technol Assess

    (2005)
  • H.O.D. Critchley

    Evaluation of abnormal uterine bleeding: comparison of three outpatient procedures within cohorts defined by age and menopausal status

    Health Technol Assess

    (2004)
  • H. Epps

    Is hydrotherapy cost-effective? A randomised controlled trial of combined hydrotherapy programmes compared with physiotherapy land techniques in children with juvenile idiopathic arthritis

    Health Technol Assess

    (2005)
  • B.C. Reeves

    A multi-centre randomised controlled trial of minimally invasive direct coronary bypass grafting versus percutaneous transluminal coronary angioplasty with stenting for proximal stenosis of the left anterior descending coronary artery

    Health Technol Assess

    (2004)
  • J. Shenfine

    A pragmatic randomised controlled trial of the cost-effectiveness of palliative therapies for patients with inoperable oesophageal cancer

    Health Technol Assess

    (2005)
  • J. Turner

    A randomised controlled trial of prehospital intravenous fluid replacement therapy in serious trauma

    Health Technol Assess

    (2000)
  • L.N. Abaid et al.

    Reducing publication bias through trial registration

    Obstet Gynecol

    (2007)
  • J. Higgins et al.

    Cochrane handbook for systematic reviews of interventions

    (2008)
  • ClinicalTrials.gov

    A service of the U.S National Institute of Health

  • The Food and Drug Administration Amendments Act of 2007 (FDAAA) or US Public Law 110-85

  • C. MacArthur

    Redesigning postnatal care: a randomised controlled trial of protocol-based midwifery-led care focused on individual women's physical and psychological health needs

    Health Technol Assess

    (2003)
  • M. Wiggins

    The Social Support and Family Health Study: a randomised controlled trial and economic evaluation of two alternative forms of postnatal support for mothers living in disadvantaged inner-city areas

    Health Technol Assess

    (2004)
  • C.J. Morrell

    Costs and benefits of community postnatal support workers: a randomised controlled trial

    Health Technol Assess

    (2000)
  • D. Kendrick

    The role of radiography in primary care patients with low back pain of at least 6 weeks duration: a randomised (unblinded) controlled trial

    Health Technol Assess

    (2001)
  • C.J. McCarthy

    Supplementation of a home-based exercise programme with a class-based programme for people with osteoarthritis of the knees: a randomised controlled trial and health economic analysis

    Health Technol Assess

    (2004)
  • J.F. Keating

    Displaced intracapsular hip fractures in fit, older people: a randomised comparison of reduction and fixation, bipolar hemiarthroplasty and total hip arthroplasty

    Health Technol Assess

    (2005)
  • S. Simpson

    A randomised controlled trial to evaluate the effectiveness and cost-effectiveness of counselling patients with chronic depression

    Health Technol Assess

    (2000)
  • C. Price

    Cost-effectiveness and safety of epidural steroids in the management of sciatica

    Health Technol Assess

    (2005)
  • Competing interest statement: All the authors declare that the answer to the questions on the competing interest form (http://resources.bmj.com/bmj/authors/checklists-forms/competing-interests) is all “no” and therefore have nothing to declare.

    Contributors: D.J.T. suggested the idea for the review. G.A.M. did the searches and extracted data from the included studies. J.C.D. and C.E.H. checked all data extracted. G.A.M., with advice from J.C.D., C.E.H., and D.J.T., undertook the data analysis. All authors drafted and revised the manuscript and approved the final version for publication. G.A.M. acts as guarantor.

    G.A.M. has the right to grant on behalf of all authors and does grant on behalf of all authors, an exclusive license (or non exclusive for government employees) on a worldwide basis to the BMJ Publishing Group (BMJPG) Ltd and its licensees to permit this article (if accepted) to be published in BMJ editions and any other BMJPG products and exploit all subsidiary rights, as set out in our license (http://resources.bmj.com/bmj/authors/checklists-forms/licence-for-publication).

    View full text