Original articles
Location bias in controlled clinical trials of complementary/alternative therapies

https://doi.org/10.1016/S0895-4356(99)00220-6Get rights and content

Abstract

To systematically investigate location bias of controlled clinical trials in complementary/alternative medicine (CAM). Methods: Literature searches were performed to identify systematic reviews and meta-analyses, which were used to retrieve controlled clinical trials. Trials were categorised by whether they appeared in CAM-journals or mainstream medical (MM)-journals, and by their direction of outcome, methodological quality, and sample size. Results: 351 trials were analysed. A predominence of positive trials was seen in non-impact factor CAM- and MM-journals, 58 / 78 (74%) and 76 / 102 (75%) respectively, and also in low impact factor CAM- and MM-journals. In high impact factor MM-journals there were equal numbers of positive and negative trials, a distribution significantly (P < 0.05) different from all other journal categories. Quality scores were significantly lower for positive than negative trials in non-impact factor CAM-journals (P < 0.02). A similar trend was seen in low-impact factor CAM journals, but not to a level of significance (P = 0.06). There were no significant differences between quality scores of positive and negative trials published in MM-journals, except for high impact factor journals, in which positive trials had significantly lower scores than negative trials (P = 0.048). There was no difference between positive and negative trials in any category in terms of sample size. Conclusion: More positive than negative trials of complementary therapies are published, except in high-impact factor MM-journals. In non-impact factor CAM-journals positive studies were of poorer methodological quality than the corresponding negative studies. This was not the case in MM-journals which published on a wider range of therapies, except in those with high impact factors. Thus location of trials in terms of journal type and impact factor should be taken into account when the literature on complementary therapies is being examined.

Introduction

The frequent use of complementary/alternative medicine (CAM) by the general population 1, 2, 3, 4 is paralleled by an increasing amount of medical scientific information published in specialised journals. The majority of controlled clinical trials (CCTs) report positive effects for treatments such as homeopathy for inhalant allergy [5], or acupuncture for nausea [6]. While these results from systematic reviews may reflect the specific efficacy of the respective therapies, other factors may also be involved. One of these factors is the tendency of authors to submit, and editors to publish, work on the strength and direction of the results. Publication bias is a recognised phenomenon in mainstream medicine (MM) 7, 8, 9, 10 and there is some suggestion that it also occurs in CAM [11]. A related but less well investigated phenomenon among published studies is that the direction of the result may affect the location of a study in terms of language and number of publications and presentations, frequency of citation, and indexing databases [12]. Location bias may affect the probability of identifying relevant articles for references or inclusion in systematic reviews and meta-analyses and may distort their outcome.

The purpose of this study is to systematically investigate location bias in the literature on CAM by assessing the direction of outcome of CCTs, their methodological quality, and sample size in relation to the type and impact factor of the journals of publication.

Section snippets

Methods

Systematic literature searches were performed to identify systematic reviews and meta-analyses of CCTs of CAM. Computerized literature databases were Medline, Embase, the Cochrane Library (all from 1990 through October 1997) and CISCOM (Research Council for Complementary Medicine, London, UK). The search terms used were chiropractic, osteopathy, spinal manipulation, acupuncture, homeopathy, and herbal medicine. In addition a manual search was performed using our own files. No language

Results

Nineteen systematic reviews/meta-analyses 13, 14, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34 met the aforementioned inclusion/exclusion criteria. These contained 351 CCTs which could be analysed. Of these, 234 (67%) were categorised as POS and 117 (33%) as NEG trials. Analysis of the distribution of CCTs by complementary therapy showed that more POS articles than NEG were published for homeopathy (72/105 (69%) POS), spinal manipulation (44/67 (66%) POS) and herbal

Discussion

Analysis of 351 CCTs identified in this study has shown that 67% of CCTs reported a statistically significant positive outcome. This predominance of POS trials occurs for each analysed therapy, except for acupuncture and is found in mainstream as well as CAM-journals. This confirms findings of previous investigations 11, 35. Bias at the pre-submission stage [36] is most likely to result from self-censorship of negative findings, while bias at the review stage may result from editorial and

References (37)

  • J.A. Astin

    Why patients use alternative medicine. Results of a national study

    JAMA

    (1998)
  • D.M. Eisenberg et al.

    Trends in alternative medicine use in the United States 1990–1997Results of a follow-up national survey

    JAMA

    (1998)
  • P. Fisher et al.

    Complementary medicine in Europe

    Br Med J

    (1994)
  • A.J. Vickers

    Can acupuncture have specific effects on health? A systematic review of acupuncture antiemesis trials

    J R Soc Med

    (1996)
  • K. Dickersin

    The existence of publication bias and risk factors for its occurrence

    JAMA

    (1990)
  • D. Blumenthal et al.

    Withholding research results in academic life science

    JAMA

    (1997)
  • K.F. Schultz et al.

    Empirical evidence of bias

    JAMA

    (1995)
  • E. Ernst et al.

    Alternative therapy bias

    Nature

    (1997)
  • Cited by (148)

    • Opportunities and challenges of plant extracts in food industry

      2021, Plant Extracts: Applications in the Food Industry
    • Traditional Chinese medicine (TCM) massage for the treatment of congenital muscular torticollis (CMT) in infants and children: A systematic review and meta-analysis

      2020, Complementary Therapies in Clinical Practice
      Citation Excerpt :

      We only included RCTs and quasi-RCTs, which may have caused evidence on other types of studies to be overlooked. All included studies were published in Chinese, which resulted in location bias [43]. Furthermore, none of the included studies reported adverse events.

    • Publication Bias

      2019, Encyclopedia of Pharmacy Practice and Clinical Pharmacy: Volumes 1-3
    • Majority of systematic reviews published in high-impact journals neglected to register the protocols: a meta-epidemiological study

      2017, Journal of Clinical Epidemiology
      Citation Excerpt :

      Conversely, in clinical trials, the impact factor was not found to be associated with reporting of statistical significance [20]. Notably, one study found that trials published in journals with a low or no impact factor were more likely to report statistically significant results than those published in high-impact medical journals [21]. Protocol registration might have been more effective in preventing outcome reporting bias for clinical trials than SRs because of differences between clinical trials and SRs in terms of the direction of collecting outcomes and the stringency of protocol registration.

    View all citing articles on Scopus
    View full text