You are here

  • Home
  • Archive
  • Volume 3, Issue 7
  • Prospective registration, bias risk and outcome-reporting bias in randomised clinical trials of traditional Chinese medicine: an empirical methodological study

Article Text

Article menu
XML
Complementary medicine
Research
Prospective registration, bias risk and outcome-reporting bias in randomised clinical trials of traditional Chinese medicine: an empirical methodological study
  1. Jian-Ping Liu1,
  2. Mei Han1,
  3. Xin-Xue Li1,
  4. Yu-Jie Mu1,
  5. George Lewith2,
  6. Yu-Yi Wang1,
  7. Claudia M Witt3,
  8. Guo-Yan Yang1,
  9. Eric Manheimer4,
  10. Torkel Snellingen1,
  11. Brian Berman4,
  12. Christian Gluud5
  1. 1Centre for Evidence-Based Chinese Medicine, Beijing University of Chinese Medicine, Beijing, China
  2. 2Centre for Complementary Medicine Research, University of Southampton, Southampton, UK
  3. 3Institute for Social Medicine, Epidemiology and Health Economics, Charité Universitätsmedizin Berlin, Berlin, Germany
  4. 4Center for Integrative Medicine, University of Maryland School of Medicine, Baltimore, Maryland, USA
  5. 5Copenhagen Trial Unit, Centre for Clinical Intervention Research, Rigshospitalet, Copenhagen University Hospital, Copenhagen, Denmark
  1. Correspondence to Professor Jian-Ping Liu; Liujp{at}bucm.edu.cn

Abstract

Background Clinical trials on Traditional Chinese Medicine (TCM) should be registered in a publicly accessible international trial register and report on all outcomes. We systematically assessed and evaluated TCM trials in registries with their subsequent publications.

Objective To describe the characteristics of TCM trials, estimate bias risk and outcome-reporting bias in clinical trials.

Data sources and study selection Fifteen trial registries were searched from their inception to July 2012 to identify randomised trials on TCM including Chinese herbs, acupuncture and/or moxibustion, cupping, tuina, qigong, etc.

Data extraction We extracted data including TCM specialty and treated disease/conditions from the registries and searched for subsequent publications in PubMed and Chinese databases. We compared information in the registries of completed trials with any publications focusing on study design, sample size, randomisation, bias risk including reporting bias from the register protocol.

Results 1096 registered randomised trials were identified evaluating TCM, of which 505 were completed studies (46.1%). The most frequent conditions were pain (13.3%), musculoskeletal (11.7%), nervous (8.7%), digestive (7.1%), circulatory (6.5%), respiratory (6.3%), mental and behavioural disorders (6.2%) and cancer (6.0%). The trial register data identified parallel, phase II/III randomised trials with sample size estimations and blinding, but limited information about randomisation (sequence generation and allocation concealment). Comparing trial registration data of 115 completed trials (22.8%) with their subsequent 136 publications, inconsistencies were identified in one or more of the following: sample size (11%), outcome assessor blinding (37.5%), primary outcomes (29%) and safety (28%) reporting.

Conclusions Increasing numbers of clinical trials investigating a variety of TCM interventions have been registered in international trial registries. The study design of registered TCM trials has improved in estimating sample size, use of blinding and placebos. However, selective outcome reporting is widespread and similar to conventional medicine and therefore study conclusions should be interpreted with caution.

Keywords

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 3.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/3.0/

https://doi.org/10.1136/bmjopen-2013-002968

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Keywords

    Article summary

    Article focus

    • We wished to evaluate the methodological quality of clinical trials in Traditional Chinese Medicine (TCM).

    • We investigated whether the systematic identification of prior trial registration was associated with an improvement in the methodological quality of the subsequent published studies.

    Key messages

    • A substantial number of clinical trials in TCM, covering a broad range of therapies, are now being registered in international trial registries.

    • Registration is associated with more rigorous study methodology and study design (eg, randomisation protocols, secure blinding and sample size estimates).

    • Outcome-reporting bias exists when comparing the registry information and the subsequent publications, and some trials were registered after their publication.

    Strengths and limitations of this study

    • Systematic searches of all available international trial registries for any clinical trials of TCM.

    • All interventions involving any TCM were included as was the diagnosis.

    • The registered information for clinical trials is not uniform across the registries and important methodological information may be missing.

    • Subsequent publications were obtained for those studies recorded as ‘completed’ in the registry. This may not represent the true situation for trials if the registry data are not updated by the researchers.

    Many empirical studies have shown that the methodological quality of randomised clinical trials of Traditional Chinese Medicine (TCM) is poor with respect to risk of systematic errors (bias; generation of allocation sequence, allocation concealment, blinding, descriptions of drop-out or losses to follow-up, selective outcome reporting) and risk of random errors (play of chance).1–6 Moreover, publications on TCM trials are uniformly positive, raising concerns that trials initiated to investigate TCM are only published if they have positive results.7 Poor-quality trials and risk of publication bias will reduce the strength of evidence when developing clinical practice guidelines or preparing systematic reviews. One of the ways to improve trial quality is to prospectively register clinical trial protocols in international trial registries such as clinicaltrials.gov.8 ,9 The WHO established an international clinical trial registry platform (ICTRP; http://www.who.int/ictrp/en/) in 2005, which now links 14 clinical trial registries. Furthermore, several peer-reviewed journals such as The Lancet and Trials publish trial protocols to promote transparency and improve trial quality.

    In order to describe the characteristics of TCM trials, and estimate reporting bias in clinical trials, we systematically searched 15 major international trial registries to identify information about TCM trials, and compared the registered records with subsequent publications regarding outcomes and other data.

    Methods

    Inclusion criteria

    We included randomised clinical trials for any TCM intervention singly or their combination: Chinese herbs, acupuncture, acupressure, moxibustion, cupping, dietary advice, tuina (therapeutic massage), taichi, qigong and guasha (scaping massage). We excluded non-randomised studies such as quasirandomised studies, cohort studies, phase 1 trials, retrospective clinical studies, case series or case studies. There were no limitations on study type (superiority, non-inferiority or equivalence) or study phase.

    Data source

    We systematically searched 15 major international trial registries (14 linked to WHO ICTRP) from their inception to July 2012 (see online supplement 01).

    For trials listed as ‘completed’ in the registered records, we then searched for published protocols, as well as the full texts of subsequently published articles in PubMed and three Chinese electronic bibliographic databases including Chinese Biomedical Database (http://sinomed.imicams.ac.cn/index.jsp), China National Knowledge Infrastructure (http://www.cnki.net) and Chinese VIP Information (http://vip.hbdlib.cn/index.asp).

    Data extraction

    Two researchers (from MH, X-XL, Y-JM, Y-YW or G-YY) extracted data independently from each trial registry using a standardised, piloted data extraction form. The form was developed by our research group and based on general characteristics of clinical trials, methodology and the 20 minimum items required for WHO trial registration.10 The main information collected included the number of trials in each registry, year registered, trial design, methods, sample size, setting, participants and diseases/conditions, differentiation of syndrome (bian zheng lun zhi), interventions, controls, primary and secondary outcomes, inclusions and exclusion criteria, current status (eg, completed and ongoing), ethical approval, sponsors, institutions, country of origin, contact details and funding.11 The extracted data were cross checked by the authors, and any discrepancy resolved by discussion with a third author (J-PL). Conditions were classified according to the WHO international classification of diseases (http://apps.who.int/classifications/icd10/browse/2010/en; last accessed 31 July 2012). We searched for publications in bibliographic databases for those trials listed as ‘completed’ in the registries, and compared the published trials with the registered records.

    Data analyses

    We used The Cochrane Collaboration risk of bias tool to evaluate the registered records.4–6 This tool assesses the following domains: generation of allocation sequence; allocation concealment; blinding; incomplete outcome reporting; and selective outcome reporting (defined as change of primary outcome or new outcome added).6 We also evaluated the estimated sample size, explicit inclusion and exclusion criteria and the risk of funding bias.

    Two authors (MH and X-XL) compared the trial design and methodology from the registered records with the resulting publications to analyse the consistency and selective outcome reporting. Selective outcome reporting was defined as “when the full paper publications reported different primary outcomes or changed it from the original pre-defined primary outcome in the registered data.” Inconsistency was defined as “when the items were not the same as described in the registry in the subsequent paper publications.” For the sample size estimation, a discrepancy of over 20% between the registered and published information was judged as inconsistent. We judged trials as positive results based on (1) authors’ conclusion showing that the intervention was superior to the control; or (2) the comparison of between groups which showed statistically significant differences (p<0.05) for primary outcome measures. We compared the dates of registration and full paper publication to assess the proportion of retrospective registrations.12

    Results

    We identified 1096 registered records of randomised clinical trials on TCM. Five hundred and five of the 1096 (46.1%) registered records indicated that the trial had been completed. Three hundred and thirty-seven of 1096 (30.7%) were from mainland China (excluding Hong Kong and Macau) (figure 1). The first trial was registered in 1999, in clinicaltrials.gov, and the number of registered trials has increased over the past decade (figure 2).

    Figure 1
    Figure 1

    Flow diagram of included trials in this study.

    Figure 2
    Figure 2

    Number of registered randomised trials on Traditional Chinese Medicine by registry.

    Table 1 shows the frequency of the type of TCM intervention included in each registry. The combined therapies studied included 105 trials on integrating Chinese and Western medicine, and 38 trials combining two or more TCM therapies. Using the international disease classification (ICD-10), we identified the 10 most frequent conditions: pain (13.3%) and musculoskeletal (11.7%), nervous (8.7%), digestive (7.1%), circulatory system (6.5%), respiratory conditions (6.3%), mental and behavioural disorders (6.2%), as well as cancer (6%), endocrine, nutritional and metabolic diseases (5.7%) and pregnancy and childbirth (4.6%; table 2).

    View this table:
    Table 1

    Numbers of registered randomised trials on Traditional Chinese Medicine by registry and country

    View this table:
    Table 2

    Conditions based on ICD-10 classification treated by Traditional Chinese Medicine

    There were methodological variations across TCM interventions (table 3). Our analysis of 1096 registered records showed that the majority were phase II/III (332/474, 70%) with 1024 (93.4%) using parallel groups design. One thousand and seventy-six (98.2%) included a sample size estimation, 714 (65.2%) reported that participants, personnel and/or outcome assessors were blinded, and 770 (70.3%) were two-armed (table 3). The reporting of control group varied across interventions. For example, none of the trials of tuina or qigong reported the use of a placebo/sham control, while a sham control was used in almost half of the acupuncture trials (44.8%). Other commonly used controls were western medicine (214, 19.5%), no intervention (181, 16.5%), non-pharmaceutical interventions (133, 12.1%), acupuncture (70, 6.4%) and Chinese herbal medicine (52, 4.7%). In addition, our data extraction of TCM syndrome differentiation showed that 65 of 290 (22.4%) trials on Chinese herbal medicine utilised syndrome differentiation either in the recruitment of participants (54 trials) or in the prescription of the herbal formula (11 trials). All these trials except one from Australia were registered by institutions in mainland China.

    View this table:
    Table 3

    Methodological information of 1096 RCTs on Traditional Chinese Medicine from International Trial Registries

    The information about randomisation procedures, including the generation of allocation sequence or concealment, was under-registered and under-reported across all intervention types. The estimated sample size of the trials ranged from less than 10 (mostly pilot/feasibility trials) to over 1000 participants, with the majority (88.3%) of sample sizes between 20 and 500/trial (see online supplement 02).

    Five hundred and five (46.1%) of the registered trials indicated their status as ‘completed’. Nineteen of these 505 trials (3.8%) had protocol publications. Our searches in PubMed and the Chinese databases identified full paper publications for 115 from the 505 completed trials (22.8%). The 115 trials produced 136 publications reporting study outcomes, among which 123 publications were in English and 13 in Chinese. Among the 115 trials with publications, 53.9% (62/115) trials specified primary outcomes (table 4). When comparing the 115 registered trials with their 136 publications, inconsistency was identified in sample size estimation (11%), outcome assessor blinding (37.5%), secondary outcome (34.4%) and safety reporting (28.6%; table 4). Selective outcome reporting was found in 29.2% (19/65) of the subsequent publications when comparing the stated primary outcomes in the registries. Incomplete outcome data were addressed in 34 trial publications (25%). Of these, 80.9% (110/136) of the trial publications reported positive results.

    View this table:
    Table 4

    Comparison of methodological components between registered records and subsequent publications of RCTs

    Among the 115 trials reporting outcomes, six trials showed other deviations from their original protocols. For example, one study was registered as prospective cohort study, but published as randomised trial; one trial was registered as parallel group, but published as a cross-over trial; one trial used different intervention from the registered record; and three trials showed inconsistent intervention arms from the registered information.

    To understand the adequacy of the trial registration, we compared the date of publications with the date of registration and we found that 11 (9.6%) trials were registered later than the publication date, suggesting retrospective registration. Furthermore, in 41 trials (35.7%), the completion date for the trials was earlier than the approval date of the registration, also suggesting inappropriate registration.

    Discussion

    The aim of prospective registration of clinical trials is to create transparency about the research methodology, and allow access to the trial protocol before the completion of the trial. Researchers and systematic reviewers can then compare the registered information or published protocols and their subsequent publications to assess publication bias or selective outcome reporting.13 ,14

    We observed a significant increase in the number of registered TCM trials in international clinical trial registries over the past decade. Our evaluation of trial quality covers a wide variety of TCM interventions from acupuncture and Chinese herbal medicine to some very ancient interventions such as cupping and gua sha therapy. These studies mainly address chronic and long-term conditions such as pain, musculoskeletal and neurological problems. TCM trial design in terms of sample size estimation, use of placebo-control and blinding and the definition of primary and secondary outcome measures is improving according to the information held in the registers.15 However, we found that the generation of allocation sequence and concealment were under-reported in the registered trials. This might be because these two items were not mandatory for the majority of registries. Interestingly, among 291 trials on Chinese herbal medicine, less than a quarter of the trials (n=65) utilised syndrome differentiation, which is considered vital to the TCM diagnosis as the basis for classifying subtypes of participants and for the prescription of herbal formula. Almost all of these trials with syndrome differentiation were registered by institutions in China, which suggest that Chinese researchers pay more attention to the selection of optimal participants and tailored treatment based on different TCM syndromes.

    We found that 29% of registered TCM trials showed selective outcome-reporting bias (discrepancies between the outcomes registered and the outcomes published). The inconsistency between the trial protocols and subsequent publications also relates to sample size, blinding, primary and secondary outcomes as well as safety. This implies that deviations from protocols might be due to the study findings which then resulted in selective outcome reporting. A previous study demonstrated that 31% of trials in Western medicine showed discrepancies between the registered protocols and the outcomes published in high-impact general medical journals,13 which suggest that TCM researchers are not alone in reporting outcomes selectively. Our data also suggest that there might be inappropriate registration in about half of the trials. Accordingly, there is still ample space to improve the quality of trial registration.16 Clinicians and policymakers need unbiased results from clinical trials to make informed clinical decisions.

    Our study has some limitations. First, due to the lack of standardisation of the items required for registration in different registries some important information, such as randomisation methods, may be under-reported. Second, we only searched for publications for those trials indicating a ‘completed’ status. We do not know how often this information is updated in different registries. Therefore, there may be trials in the registries that are not listed as ‘completed’, but that have been completed. This may cause bias, particularly if the investigators who update the status of their trials on the registry are also more likely to adhere to the methods described in the registry when writing up their results in the subsequent publications. Third, as we undertook searches in PubMed and three Chinese databases only, we may have missed some studies that have been reported in other databases. In addition, there is a lag time between completing a study and writing for publication, submission and peer review, all of which can be considerable. Fourth, we only looked at registered trials. A very large number of TCM trials are conducted without being registered, and here we can say nothing about their risks of bias and risks of random errors. In all likelihood, these may be even worse than those we have observed.

    We conclude that the study design and the quality of reporting of TCM trials have improved through prospective international trial registration compared with previous methodological studies,1–6 although there are some inconsistencies between the registered trial protocols and subsequent publications and insufficient reporting on syndrome differentiation. Publication bias as a consequence of selective outcome reporting is still widespread and similar to conventional medicine, therefore study conclusions should be interpreted with caution. In herbal medicine trials, it would be inappropriate if a trial design does not utilise syndrome differentiation, and participants may not be properly treated.

    Acknowledgments

    We thank the following people for their contributions in the study identification, data extraction including: Nini Chen, Qianyun Cai, Zhijun Liu, Xun Li, Miao Ding, Qing Li, Pan Deng. We also thank Dr Ke Cheng from the University of Maryland School of Medicine and Ms Xun Li from the University of Western Sydney for retrieving full text articles. Jian-Ping Liu had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

    References

    1. ↵
      1. Tang JL,
      2. Zhan SY,
      3. Ernst E
      . Review of randomised controlled trials of traditional Chinese medicine. BMJ 1999;319:160–1.
      OpenUrlFREE Full Text
    2. ↵
      1. Wang G,
      2. Mao B,
      3. Xiong ZY,
      4. et al
      . CONSORT Group for Traditional Chinese Medicine. The quality of reporting of randomised controlled trials of traditional Chinese medicine: a survey of 13 randomly selected journals from mainland China. Clin Ther 2007;29:1456–67.
      OpenUrlCrossRefPubMedWeb of Science
    3. ↵
      1. He J,
      2. Du L,
      3. Liu G,
      4. et al
      . Quality assessment of reporting of randomization, allocation concealment, and blinding in traditional Chinese medicine RCTs: a review of 3159 RCTs identified from 260 systematic reviews. Trials 2011;12:122.
      OpenUrlPubMed
    4. ↵
      1. Wood L,
      2. Egger M,
      3. Gluud LL,
      4. et al
      . Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ 2008;336:601–5.
      OpenUrlAbstract/FREE Full Text
    5. ↵
      1. Savović J,
      2. Jones HE,
      3. Altman DG,
      4. et al
      . Influence of reported study design characteristics on intervention effect estimates from randomised, controlled trials. Ann Intern Med 2012;157:429–38.
      OpenUrlCrossRefPubMedWeb of Science
    6. ↵
      1. Higgins JPT,
      2. Altman DG,
      3. Sterne JAC
      , ed. Chapter 8: assessing risk of bias in included studies. In: Higgins JPT, Green S, eds. Cochrane handbook for systematic reviews of interventions version 5.1.0 (updated March 2011). The Cochrane Collaboration, 2011. http://www.cochrane-handbook.org
    7. ↵
      1. Vickers A,
      2. Goyal N,
      3. Harland R,
      4. et al
      . Do certain countries produce only positive results? A systematic review of controlled trials. Control Clin Trials 1998;19:159–66.
      OpenUrlCrossRefPubMedWeb of Science
    8. ↵
      1. Dickersin K,
      2. Rennie D
      . Registering clinical trials. JAMA 2003;290:516–23.
      OpenUrlCrossRefPubMedWeb of Science
    9. ↵
      1. Abaid LN,
      2. Grimes DA,
      3. Schulz KF
      . Reducing publication bias through trial registration. Obstet Gynecol 2007;109:1434–7.
      OpenUrlCrossRefPubMedWeb of Science
    10. ↵
      WHO. International Clinical Trials Registry Platform. Trial registration data set. 2006. http://www.who.int/ictrp/network/trds/en/index.html (accessed 30 Oct 2012).
    11. ↵
      1. Reveiz L,
      2. Chan A-W,
      3. Krleža-Jerić K,
      4. et al
      . Reporting of methodologic information on trial registries for quality assessment: a study of trial records retrieved from the WHO search portal. PLoS ONE 2010;5:e12484.
      OpenUrlCrossRefPubMed
    12. ↵
      1. Mathieu S,
      2. Boutron I,
      3. Moher D,
      4. et al
      . Comparison of registered and published primary outcomes in randomised controlled trials. JAMA 2009;302:977–84.
      OpenUrlCrossRefPubMedWeb of Science
    13. ↵
      1. Chan AW,
      2. Hróbjartsson A,
      3. Haahr MT,
      4. et al
      . Empirical evidence for selective reporting of outcomes in randomised trials: comparison of protocols to published articles. JAMA 2004;291:2457–65.
      OpenUrlCrossRefPubMedWeb of Science
    14. ↵
      1. Chan AW,
      2. Altman DG
      . Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors. BMJ 2005;330:753.
      OpenUrlAbstract/FREE Full Text
    15. ↵
      1. Liu JP,
      2. McIntosh H,
      3. Lin H
      . Chinese medicinal herbs for chronic hepatitis B. Cochrane Database Syst Rev 2001;(1):CD001940.
    16. ↵
      1. Viergever RF,
      2. Ghersi D
      . The quality of registration of clinical trials. PLoS ONE 2011;6:e14701.
      OpenUrlCrossRefPubMed

    Supplementary materials

    • Temporary Index File

    • Supplementary Data

      This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

      Files in this Data Supplement:

    Footnotes

    • J-PL and MH contributed equally.

    • Contributors J-PL conceived and designed the study, verified data extraction and analyses and drafted the manuscript. MH developed search strategies, identified trials, extracted data and analysed data and revised the manuscript. X-XL, Y-JM, Y-YW and G-YY identified trials, extracted data and revised the manuscript. GL provided methodological perspectives and made substantial revisions to the manuscript. CMW made substantial revisions to the manuscript and provided method perspectives. EM made substantial revisions to the manuscript and provided comments. TS helped with the design of tables and data analyses and made revisions to the manuscript. BB revised the manuscript and provided comments. CG conceived and designed the study and revised the manuscript and provided comments. All authors have read and approved the final manuscript.

    • Funding NCCAM of NIH, USA, Ministry of Science and Technology, China (grant number R24 AT001293, 2011-CXTD-09, 2011ZX09302-006-01-03(5))

    • Competing interests J-PL, MH, X-XL, Y-JM, Y-YW and G-YY were supported by the grant number 2011-CXTD-09 from Beijing University of Chinese Medicine. BB, J-PL and EM were partially funded by the grant number R24 AT001293 from the National Center for Complementary and Alternative Medicine (NCCAM) of the US National Institutes of Health (http://www.nccam.nih.gov). J-PL and MH were partially funded by the grant number 2011ZX09302-006-01-03(5) by the Ministry of Science and Technology of China.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Data sharing statement Original data of this study can be accessed through J-PL via email.