Figures
Abstract
Background
Systematic reviews (SRs) of TCM have become increasingly popular in China and have been published in large numbers. This review provides the first examination of epidemiological characteristics of these SRs as well as compliance with the PRISMA and AMSTAR guidelines.
Objectives
To examine epidemiological and reporting characteristics as well as methodological quality of SRs of TCM published in Chinese journals.
Methods
Four Chinese databases were searched (CBM, CSJD, CJFD and Wanfang Database) for SRs of TCM, from inception through Dec 2009. Data were extracted into Excel spreadsheets. The PRISMA and AMSTAR checklists were used to assess reporting characteristics and methodological quality, respectively.
Results
A total of 369 SRs were identified, most (97.6%) of which used the terms systematic review or meta-analysis in the title. None of the reviews had been updated. Half (49.8%) were written by clinicians and nearly half (47.7%) were reported in specialty journals. The impact factors of 45.8% of the journals published in were zero. The most commonly treated conditions were diseases of the circulatory and digestive disease. Funding sources were not reported for any reviews. Most (68.8%) reported information about quality assessment, while less than half (43.6%) reported assessing for publication bias. Statistical mistakes appeared in one-third (29.3%) of reviews and most (91.9%) did not report on conflict of interest.
Conclusions
While many SRs of TCM interventions have been published in Chinese journals, the quality of these reviews is troubling. As a potential key source of information for clinicians and researchers, not only were many of these reviews incomplete, some contained mistakes or were misleading. Focusing on improving the quality of SRs of TCM, rather than continuing to publish them in great quantity, is urgently needed in order to increase the value of these studies.
Citation: Ma B, Guo J, Qi G, Li H, Peng J, Zhang Y, et al. (2011) Epidemiology, Quality and Reporting Characteristics of Systematic Reviews of Traditional Chinese Medicine Interventions Published in Chinese Journals. PLoS ONE 6(5): e20185. https://doi.org/10.1371/journal.pone.0020185
Editor: Lisa Hartling, Alberta Research Centre for Health Evidence, University of Alberta, Canada
Received: September 12, 2010; Accepted: April 27, 2011; Published: May 25, 2011
Copyright: © 2011 Ma et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: The authors have no support or funding to report.
Competing interests: The authors have declared that no competing interests exist.
Introduction
The first systematic review addressing the effect of traditional Chinese medicine (TCM) that was published in a Chinese journal may be sourced back to Chen et al. in 1999 [1]. Since then, systematic reviews of TCM have become been increasingly published in China, and now with a large number. Given the implications of systematic reviews to policy making and clinical practice, achieving highest possible quality in design, conduct, analysis, and reporting is of paramount importance [2].
In the last decade, a few studies have examined the reporting of systematic reviews, and found that the quality of reporting was generally poor [3], [4], [5]. Three studies, using the Quality of Reporting of Meta-analyses (QUOROM) checklist, examined the quality of reporting of reviews of traditional Chinese medicine, and concluded with similar findings [6], [7], [8] Nevertheless, they were inherent with limitations, including the failure to report details about epidemiological characteristics of included reviews, and the failure to address methodological quality of those reviews.
In 2009, the newer standard of reporting systematic review, the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) was released to replace the QUOROM for guiding the review reporting [9]. Earlier than that, an instrument, assisting the assessment of methodological quality of systematic reviews, Assessment of Multiple Systematic Reviews (AMSTAR) [10] was published in 2007. This instrument, developed on the biasis of the Oxman-Guyatt Overview Quality Assessment Questionnaire (OQAQ) [11] and Sack's Quality Assessment Checklist [12] is considered a validated tool in assessing the methodological quality of systematic reviews, and receives recognition from international agencies, including The Canadian Agency for Drugs and Technologies in Health (CADTH) [13].
Our study therefore aims to address the limitations of published studies, by assessing epidemiological and reporting characteristics, and methodological quality of systematic reviews of traditional Chinese medicine. We will particularly employ the new instruments for assessing the quality of reporting and methodological quality of those reviews.
Methods
Inclusion Criteria
We included all systematic reviews about TCM published in Chinese Journals. TCM interventions may have included herbal medicine, acupuncture/acupressure, moxibustion, Tuina massage, food therapy, and physical exercise such as tai chi and shadow boxing. TCM interventions may have been administered alone or in combination with conventional western medicine. We included publications described as systematic reviews or those that provided an overview of evidence from multiple studies and where authors described their methods in explicit detail.
Search Strategy(Text S1)
Four Chinese databases (Chinese Biomedicine Literature Database (CBM), Chinese Scientific Journal Full-text Database (CSJD), Chinese Journal Full-text Database (CJFD), and Wanfang Database) were searched from inception through Dec 2009. The search terms included “Systematic Review”, “Meta-analysis”, “Traditional Chinese Medicine” and “Chinese herbs” etc.
Screening
Two researchers independently screened the titles and abstracts of identified studies. One reviewer subsequently screened the full text articles of potentially included studies (Jiwu GUO) while a second reviewer independently screened a 20% random sample (Bin MA). Disagreement was resolved by discussion.
Data Collection and Analysis
Variables extracted included publication and reporting characteristics as well as items from the PRISMA and AMSTAR checklists. Reviews were classified according to their TCM focus as Herbal (e.g. bulk herbs, decoctions, pills) or Non-herbal (e.g. acupuncture, Tuina). Conditions studied were classified using the International Classification of Diseases (ICD-10). Data was collected using a standardized form and summarized using descriptive statistics (frequency, median, interquartile range (IQR)). Analyses were performed using Excel (version Microsoft Excel 2003; http://office.microsoft.com/zh-cn/) and SPSS (version 13.0; http://www.spss.com).
Results
Search
The searches identified 10,001 records. Screening excluded 9,621 reviews due to duplication, focus on non-TCM interventions, or for not being a SR. After examination of the full texts of 380 article, a further 11 reviews were excluded because they were quality assessments of systematic reviews and meta-analyses. A total of 369 publications were included (Figure 1, Text S2).
SRs, Systematic Review.
Epidemiological Characteristics (Table 1)
The 369 reviews were all written in Chinese and were published in 145 different Chinese Journals. Frequency of citation of each review ranged from 0 to 79; nearly half (46.1%) had not been cited and only 2.4% had been cited more than 15 times. Almost half (49.9%) of the reviews were written by clinicians. The reviews were classified as either Herbal interventions (84.3%) or Non-Herbal interventions. Non-herbal interventions included acupuncture (14.6%) and Tuina (1.1%). The most common conditions studied were diseases of the circulatory system (6.5%) and digestive system (5.7%).
Descriptive Characteristics (Table 2)
The reviews included a median of three authors (IQR: 2.0–5.0). Almost half the reviews (47.7%) were published in specialty journals although this differed significantly between reviews of herbal and non-herbal interventions (79.0% versus 21.0%, respectively (p<0.05). Only one-third (32.5%) were published in journals cited by Chinese Science Citation Database (CSCD), which again varied significantly between herbal and non-herbal interventions (p<0.05). The reviews included a median of 10.0 studies each, involving 951.5 participants. Non-herbal SRs included significantly fewer studies (p<0.05) and participants compared to herbal SRs. Meta-analysis was conducted in almost all the reviews (93.0%) and did not differ significantly between groups. None of the reviews had been updated from a previous review.
PRISMA Checklist Assessment (Table 3)
Compliance with PRISMA checklist items ranged from 0–97.6%. Almost all reviews (97.6%) described themselves using the terms “systematic review” or “meta analysis”. Most reviews were compliant with the following checklist items: included a clear rationale, described information sources, described method used for assessing risk of bias of individual studies, stated the principle of summary measures, described the results of individual studies, discussed limitations at study and outcome level, provided a general interpretation of results, and presented an available conclusion. More than half of reviews were compliant with the following checklist items: reported eligibility criteria, presented search strategy, stated the process for selecting studies, specified any assessment of risk of bias that may affect the cumulative evidence, described characteristics of included studies, or described risk of bias within studies. Less than half of reviews were compliant with the following checklist items: described data collection process, described synthesis of results, described additional analysis, described how the studies were selected, presented synthesis of results (although this varied across reviews category), presented results of any assessment of risk of bias across studies, or described sources of funding and other support. Few studies provided a description of objectives or purpose, lists of data items, or gave results of additional analysis. None of the studies provided a structured summary, protocol or registration information, or provided a summary of results in the discussion.
AMSTAR Checklist Assessment (Table 4)
Compliance with AMSTAR checklist items ranged from 0–70.2%. More than half of reviews were compliant with the following checklist items: reported that a comprehensive literature search was performed, provided the characteristics of included studies, assessed and documented the scientific quality of the included studies, or appropriately addressed the quality of included studies in formulating conclusions. Less than half of reviews were compliant with the following checklist items: reported that there were duplicated study selection and data extraction, used appropriate methods to combine the findings of studies (although this varied across reviews category), or assessed the likelihood of publication bias. Few studies provided an ‘a priori’ design, reported the status of publication used as an inclusion criterion, provided a list of studies, or stated if there was a conflict of interest or not. .
Discussion
Our study identified 369 systematic reviews of TCM interventions published since 1999, most of which evaluated herbal interventions. This study updates previous reviews of this topic [6], [7], [8] by including an additional 258 SRs. This review is also the first to examine compliance of Chinese SR authors with the PRISMA reporting guideline and AMSTAR tool for assessing methodological quality.
Our review examined some of the same variables described in Moher et al.'s investigation of SRs written in English and indexed in Medline [5]. In that study, the authors reported that the 300 SRs they included were published in 132 journals, including the Cochrane Library. In contrast to our review, only 8% of their included SRs were of complementary and alternative medicine (CAM) interventions. While most (91%) of their SRs were published in specialty journals, 45% of all 132 journals did not have an impact factor. Most (93%) of our SRs included a meta-analysis compared to 54% of the English studies and while 44% of our studies assessed for publication bias, only 23% of the English studies reporting doing so. Funding sources were reported in 39% of our studies compared to 59% of the English studies.
The range of diseases addressed in our reviews are similar with those in the three reviews [6], [7], [8] focusing on the Chinese literature. The three reviews, however, failed to report details about the epidemiological and study characteristics of systematic reviews, including number of times cited, journal type, number of studies included, number of participants, and whether it is an update of a published review. Moreover, one study [6] did not report the prevalence of reviews meeting each individual item of QUOROM. In addition we found that, despite the increasing use of the terms “systematic review” and ”meta-analysis” in the title and subsequent manuscript sections, the quality of reporting remains poor.
While many deficits in reporting were evident in the Chinese SRs, we identified areas of particular concern. These included evidence that suggested that the SRs were not highly referenced by other researchers working in the same field. This may be due, in part, to the overall poor quality of this body of work, which may also be a reason that less than one third were indexed in the Chinese Science Citation Database (CSCD), which is similar to the Science Citation Index, in that indexed journals are considered of higher quality than non-indexed journals.
There is also evidence that even though SRs have become an increasingly popular source of up-to-date knowledge, they are under utilized by Chinese clinicians. Because although the Chinese Cochrane Center was established in 1997 by the Ministry of Health of the people's republic of China, recent studies reported that most clinicians and nurses had not heard of or did not understand the meaning of evidence-based medicine (EBM) [14], [15]. In addition, not all medical schools have introduced EBM curricula in China.
Although it is well known that results from reviews are most useful when they are up-to-date [16], none of the reviews included in our SR reported being an update of a previous review. This may be due to lack of policies in China to encourage updates and reluctance of Chinese journals to publish updated reviews that are not substantially different from previous publications. The review by Moher et al. reported that 18% of the English SRs were updates, however, only those SRs published in the Cochrane Library had been updated [5], suggesting that this problem is not restricted to Chinese journals.
We have demonstrated that compliance with PRISMA reporting guidelines is low for many Chinese SRs. Items of particular concern included lack of appropriate abstracts or structured summaries and details of protocols or summaries of evidence. In order to improve the quality of reporting, we strongly recommend the use of reporting guidelines by authors. We also recommend that editors of medical journals recognize and promote use of reporting guidelines in their publications. Lastly, medical schools should introduce reporting guidelines into medical education as early as possible.
Reviews of numerous medical specialties, as well as comprehensive reviews, have concluded that the quality of SR and meta-analysis (MA) reporting is generally poor [6], [7], [17]. Examination of changes in reporting quality of MAs published in Medline, Embase and the Cochrane Database of Systematic Reviews, suggested that quality had improved after the release of the QUORUM reporting guidelines [18], [19]. We anticipate that increased use of PRISMA by Chinese authors will be helpful in increasing the reporting quality of SRs published in Chinese journals.
Our study has also shown that the methodological quality of Chinese SRs of TCM is poor. Areas of particular concern include lack of report of an ‘a priori’ design, description of status of publication used as an inclusion criterion, as well as statement of conflict of interest, all factors which may be associated with biased results. In addition, statistical mistakes appeared in one-third (29.3%) of reviews. For instance, many reviews did not explore reasons for statistical heterogeneity but simply pooled results using a random effects model to account for heterogeneity. Where heterogeneity is substantial, an overall pooling of results may be inappropriate and may lead to the incorrect interpretation of results.
One explanation for limited compliance with methodological guidelines may be that AMSTAR has not been published or promoted in Chinese journals outside of a brief abstract published in the Chinese Journal of Evidence-based Medicine in September 2010 [20]. Broader promotion of methodological quality guidelines is a necessary step in enhancing dissemination and implementation of AMSTAR.
However, our study has limitations. First, the majority of articles failed to follow the PRISMA guideline. This is likely due to the fact that the PRISMA guideline had not yet been released at the time these studies were published. Second, our study included systematic reviews published only in Chinese journals, whereas Chinese investigators increasingly publish articles in international journals. Third, we have found differential results between systematic reviews addressing herbal versus non-herbal medicines. This is possibly because reviews addressing non-herbal medicines would have studies with fewer events, resulting in absence of statistical difference of comparisons. Last, in our search, we included the terms “systematic review” and “meta-analysis”. Some potentially eligible systematic review (i.e., systematic reviews needs to be plural) may, however, not use these terms in their publications.
Our purpose was to provide readers with a broad overview of the reporting and methodological characteristics of SRs of TCM published in Chinese journals. Although many such SRs have been published, the quality of these reviews is troubling. As a potential key source of information for clinicians and researchers, not only were many of these reviews incomplete, some contained mistakes or were misleading. Focusing on improving the quality of SRs of TCM, rather than continuing to publish them in great quantity, is urgently needed in order to increase the value of these studies.
Supporting Information
Text S1.
Four Chinese databases search strategy and hyperlink address.
https://doi.org/10.1371/journal.pone.0020185.s001
(DOC)
Text S2.
Three hundred and sixty-nine systematic review of traditional chinese medicine interventions published in Chinese journals
https://doi.org/10.1371/journal.pone.0020185.s002
(DOC)
Acknowledgments
We thank Dr. Denise Adams (CARE Program, Department of Pediatrics, Faculty of Medicine, University of Alberta, Edmonton, Alberta, Canada.) and Prof. Xin Sun (Center for Health Research, Northwest, Portland, Oregon, United States) for providing assistance with editing the final manuscript. We thank the library of Lanzhou University for database access and acquiring full texts.
Author Contributions
Conceived and designed the experiments: BM KY J-WG G-QQ H-ML. Analyzed the data: J-WG BM J-YP Y-LZ Y-QD. Wrote the paper: BM J-WG. Extracted data from trials: J-WG BM. Final approval of manuscript: BM J-WG G-QQ H-ML J-YP Y-LZ Y-QD K-HY.
References
- 1. Chen ZH, Zhang JH (1999) Troditional Chinese herbs for precancerous lesion of esophagus: a meta analysis. TCM Res 12: 20–21.
- 2. Petticrew M (2000) Systematic reviews from astronomy to zoology: myths and misconceptions. BMJ 322: 98–101.
- 3. Jadad AR, McQuay HJ (1996) Meta-analyses to evaluate analgesic interventions: a systematic qualitative review of their methodology. J Clin Epidemiol 49: 235–2343.
- 4. Shea B, Moher D, Graham I, Pham B, Tugwell P (2002) A comparison of the quality of Cochrane reviews and systematic reviews published in paper-based journals. Eval Health Prof 25: 116–129.
- 5. Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG (2007) Epidemiology and reporting characteristics of systematic reviews. PLoS Med 4: e78.
- 6. Li TQ, Liu XM, Zhang MM, Ma JX, Du L, et al. (2007) Assessment of Systematic Reviews and Meta-analyses on Traditional Chinese Medicine Published in Chinese Journals. Chin J Evid-based Med 7: 180–188.
- 7. Liu JP, Xia Y (2007) Quality appraisal of systematic reviews or meta-analysis on traditional Chinese medicine published in Chinese journals. Zhongguo Zhong Xi Yi Jie He Za Zhi 4: 306–311.
- 8. ZHZANG Juhua, SHANG Hongcai, GAO Xiumei, et al. (2007) Methodology and reporting quality of systematic review/meta-analysis of traditional Chinese medicine. J Altern Complement Med 13: 797–805.
- 9. Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group (2010) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Int J Surg 8: 336–341.
- 10. Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, et al. (2007) Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol 7: 10.
- 11. Oxman AD (1994) Checklists for review articles. BMJ 309: 648–651.
- 12. Sacks H, Berrier J, Reitman D, Berk A, Chalmers T (1987) Meta-analyses of randomized controlled trials. N Engl J Med 316: 450–455.
- 13. Bessa-Nogueira RV, Vasconcelos BC, Niederman R (2008) The methodological quality of systematic reviews comparing temporomandibular joint disorder surgical and non-surgical treatment. BMC Oral Health 8: 27.
- 14. Wang XP, Fan GH (2010) Investigation cognition about Evidence-based Medicine. Medical information 8: 2537–2539.
- 15. Liu K, Guo L, Zhang L (2010) Investigation into Nursing Staff's Cognition Status of Evidence- based Medicine. Journal of Liao ning Medical University 3: 259–262.
- 16. Moher D, Tsertsvadze A (2006) Systematic reviews: When is an update an update? Lancet 367: 881–883.
- 17. Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group (2009) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ 339: b2535.
- 18. Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, et al. (1999) Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses. Lancet 354: 1896–1900.
- 19. Delaney A, Bagshaw SM, Ferland A, Manns B, Laupland KB, et al. (2005) A systematic evaluation of the quality of meta-analyses in the critical care literature. Crit Care 9: R575–R582.
- 20. Xiong J, Du YH (2010) Assessing tool of methodology quality for systematic reviews and meta analysis: introduction of AMSTAR. Chin J Evid-based Med 9: 44–44.