Article Text
PDF
Abstract
Introduction We are witnessing an explosive increase in redundant and overlapping publications of systematic reviews and meta-analyses (SRs/MAs) on the same topic, which often present conflicting results and interpretations, in the current medical literature. They represent wasted efforts on the part of investigators and peer reviewers and may confuse and possibly mislead clinicians and policymakers. Here, we present a protocol for a meta-epidemiological investigation to describe how often there are overlapping SRs/MAs on the same topic, to assess the quality of these multiple publications, and to investigate the causes of discrepant results between multiple SRs/MAs in the field of major surgery.
Methods and analysis We will use MEDLINE/PubMed to identify all SRs/MAs of randomised controlled trials (RCTs) published in 2015 regarding major surgical interventions. After identifying the ‘benchmark’ SRs/MAs published in 2015, a process of screening in MEDLINE will be carried out to identify the previous SRs/MAs of RCTs on the same topic that were published within 5 years of the ‘benchmark’ SRs/MAs. We will tabulate the number of previous SRs/MAs on the same topic of RCTs, and then describe their variations in numbers of RCTs included, sample sizes, effect size estimates and other characteristics. We will also assess the differences in quality of each SR/MA using A Measurement Tool to Assess Systematic Reviews (AMSTAR) score. Finally, we will investigate the potential reasons to explain the discrepant results between multiple SRs/MAs.
Ethics and dissemination No formal ethical approval and informed consent are required because this study will not collect primary individual data. The intended audiences of the findings include clinicians, healthcare researchers and policymakers. We will publish our findings as a scientific report in a peer-reviewed journal.
Trial registration number In PROSPERO CRD42017059077, March 2017.
- meta-analysis
- systematic review
- same topic
- overlapping
- surgical Intervention
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Strengths and limitations of this study
This meta-epidemiological study is the first to focus on the cause of discrepant results between multiple overlapping systematic reviews and meta-analyses (SRs/MAs) on many and unspecified topics in surgery.
Our study will provide a more clear assessment of the results by minimising the impact of industry sponsorship that would be typical of pharmacological interventions, because our study will mainly examine those of surgical interventions.
Judging whether the topic is identical or different according to the similarity of the research question may be difficult.
The time lag between manuscript submission dates and its official publication dates should be considered as an important factor that might affect the proportion of all available randomised controlled trials that are included in a particular SR/MA.
We will perform the article search using a single database (MEDLINE via PubMed) to identify SRs/MAs.
Introduction
Description of the condition
In recent years, the number of published systematic reviews (SRs) and meta-analyses (MAs) have been increasing explosively in many medical fields.1–4 When a small number of randomised control trials (RCTs) of new research questions are published, new SRs/MAs soon follow these RCTs. Indeed, it is becoming more and more difficult to find a new research question that nobody has examined as a SR article in medical journals. Furthermore, many topics addressed by SRs/MAs often overlap with each other entirely or partially, and instances have been reported where more than 10 SRs/MAs were published on a single topic in a limited span of time.2 3 These potentially redundant publications may represent duplicated efforts for researchers, peer reviewers and editors of the medical journals.
The picture is even more confusing and distressing for readers of the medical literature. Some previous studies have shown that the numbers of pooled RCTs that included patients’ sample sizes are quite different across SRs/MAs on the same topic.2 3 5–10 It is then a natural consequence that there are discrepancies between results of SRs/MAs; different effect sizes, with different statistical precision, and even different directions of effect have been reported.2 3 5–10 Such conflicting results can confuse and sometimes mislead clinicians, who are required to make a decision and choose among all competing treatments. In addition, previous meta-epidemiological studies have reported that some SRs/MAs actually omitted and did not cite previous review articles even though the research question was identical.4 8
Why it is important to do this review
How do we explain these discrepant results between multiple SRs/MAs on the same topic? Some possible explanations for these problems have been suggested, such as the differences of each review’s eligibility criteria, the publication of updated versions, inclusion or exclusion of unpublished data, different databases searched, search dates, language restrictions, errors in data extraction and statistical methods for data synthesis.2–10 However, it is unclear which factor is the most influential and whether there are other unknown causes. Some recent studies have reported that rapid growth of SRs/MAs regarding antidepressant medications was linked to industry sponsorship.1 11 In the field of major surgery, as far as the present authors are aware, there has been no meta-epidemiological study that has examined the problem of redundant and overlapping MAs.
Objectives
The primary objectives of this study are (1) to describe how often SRs/MAs of RCTs that were published almost at the same time on the same topic overlap with one another and (2) to investigate the cause of discrepant results between multiple SRs/MAs. In addition, we will sequentially assess the quality of these multiple SRs/MAs. The hypothesis of the proposed study is that there are many redundant overlapping publications with conflicting results and various qualities, which represent wasted efforts for investigators and peer reviewers and may well mislead the clinicians and policymakers in understanding of the results.
Methods and analysis
Study design
A meta-epidemiological study.
Eligibility criteria
We will select the studies according to the following inclusion and exclusion criteria.
Inclusion criteria
SRs/MAs of only RCTs.
Study populations of RCTs are patients that underwent chest and abdominal surgeries. We define chest surgery as any surgical procedure involving intrathoracic organs (eg, surgery for coronary artery bypass graft and surgery for lung, mediastinal and pleural diseases). We define abdominal surgery as deliberate breach of peritoneum or retroperitoneum, including gastrointestinal surgery, abdominal aortic surgery, urological surgery and obstetrics and gynaecological surgery.
Interventions (or comparisons) of RCTs are any surgical interventions that were abdominal or chest procedures performed in the operating room. We will include both comparisons of surgical versus surgical interventions and comparisons of surgical versus non-pharmacological interventions.
We will include any outcomes reported by the SRs/MAs, including continuous and dichotomous outcomes.
Exclusion criteria
SRs/MAs that include both RCTs and observational studies.
SRs/MAs that include both RCTs and quasi-RCTs.
Network MAs.
Individual-patient data MAs.
Interventions (or comparisons) of RCTs are any pharmacological interventions.
Search strategy and study selection
First, we will use the MEDLINE (via PubMed) database to identify recent SRs/MAs of RCTs published in 2015 in the field of major surgical interventions. We will use a straightforward combination of Medical Subject Headings terms relevant to SRs and surgery: (systematic[sb] AND (‘surgery’ [subheading] AND 2015 [dp]) without language restriction. We will exclude studies that do not meet our eligibility criteria based on titles and abstracts, and then read the full text to decide whether papers with potentially relevant titles and abstracts meet our eligibility criteria. Those that meet our eligibility criteria at this point will be termed as the ‘benchmark’ SRs/MAs and topics. We show their eligibility criteria above. If there are more than two same/similar topics in 2015, we will set the broadest one regarding study populations as the ‘benchmark’ topic. For example, when study populations of study A are a subgroup of study B, we will set study B as a ‘benchmark’ SR/MA or topic.
Second, we will conduct screening in MEDLINE to identify similar previous SRs/MAs of RCTs on the same topics that were published from 2011 to 2015. To be more specific, after identifying ‘benchmark’ SRs/MAs of RCTs published in 2015, we will carry out this process of screening in MEDLINE to identify the previous overlapping MAs/SRs of RCTs on the same topic that were published within 5 years of the ‘benchmark’ SRs/MAs and bibliographies of included articles, according to the titles and abstracts. The reviews identified during the bench marking process will be used to identify relevant keywords and then index terms to devise a complete PubMed/MEDLINE search strategy to locate relevant SRs/MAs published between 2011 and 2015. Reference lists of review articles and original RCTs will be considered as additional sources of information. SRs/MAs will be selected according to the population, intervention and comparator only. In other words, we will not place restrictions on outcomes, as long as the SRs/MAs share common study populations, surgical interventions and comparators. We will not place language restrictions at this stage, either.
Two pairs of two trained researchers will perform the title and abstract review, full-paper screening and data extraction. Two researchers will independently evaluate if a SR/MA is eligible and if the research question is similar to that of the ‘benchmark’ review. Any disagreements will be resolved through discussions or through involvement of a third researcher.
Data extraction and management
Two researchers will independently read each full paper and extract data in duplicate using a standardised form to ensure consistency of extracted data for each study. The following information will be extracted:
Journal and study characteristics (lead author’s name and country affiliation, official publication year, journal name and journal impact factor, the presence or absence of methodologist in the list of coauthors).
Search methodology (the name and number of databases searched, date of last search, language restriction and restrictions on publication status).
Research question of SRs/MAs (PICO: participant characteristics, intervention/comparison details and outcome measures).
Characteristics of SRs/MAs (number of eligible RCTs, sample size, effect size, statistical approach (eg, fixed effects vs random effects model), citation of previous SRs/MAs and the presence or absence of industry funding).
We will resolve any disagreement in consultation with the third investigator of the review team.
Data analysis
Descriptive statistics will be used to summarise characteristics of each of the included SRs/MAs on the same research question. We will tabulate the number of previous SRs/MAs of RCTs on the same research question that were published within 5 years of publication of the ‘benchmark’ SRs/MAs. Next for each topic, we will describe the differences in number of RCTs included, sample size, effect size and variability between SRs/MAs. We will compare the coverage ratio of all RCTs that were published until the publication year of each SR/MA. The difference in treatment effect estimates, expressed either as OR or standardised mean difference (SMD), will be compared between each of the older MAs and the latest MA. If the original studies used other efficacy indexes and provided sufficient data for the calculations, we will convert them into ORs or SMDs. We will calculate relative OR or the difference in SMDs and their 95% CIs, as were done in some previous meta-epidemiological studies.12 13
We will also assess the difference in quality of each SR/MA on the same research question. To evaluate study quality, we will use ‘A Measurement Tool to Assess Systematic Reviews’ (AMSTAR)14 instrument. Regarding the AMSTAR score, reported points will be assigned as follows: yes=1, no=0, not applicable=1.15 The difference in study quality, expressed as AMSTAR scores, will be compared between each SR/MA.
Finally, we will investigate the potential reasons to explain the discrepant results, if any, between multiple SRs/MAs. We will evaluate which factors (eg, comprehensiveness of search, number of eligible RCTs, statistical approach and industry funding) influence these differences and whether there are some other unexpected causes. Thus, we might conduct additional exploratory analyses.
Statistical analyses will be two sided, with a p value of 0.05 indicating statistical significance. All analyses will be performed using Stata/SE 11 (StataCorp).
Ethics and dissemination
This meta-epidemiological study does not require ethical approval. We will publish the findings of this study in a peer-reviewed scientific journal and present them at scientific conferences. The results of this research will be disseminated electronically or in print, as well as among internet communities.
Discussion
Strengths and limitations of study
We have presented the study protocol for a meta-epidemiological investigation with respect to redundant publications of overlapping SRs/MAs on the same topic in major surgery. Our study has several strengths. First, most previous studies were restricted to some specific topics.2 3 6–11 To our knowledge, this is the first effort to investigate the cause of discrepant results between multiple SRs/MAs on many and unspecified topics all together. Readers of our research can more confidently generalise our findings. Second, unlike most previous studies that evaluated the recent SRs/MAs of pharmacological treatments,3 4 7 11 our study will examine those of surgical interventions. Thus, our study will provide a more clear assessment of the results by minimising the impact of industry sponsorship that would be typical of pharmacological treatments. Third, we will employ a well-designed systematic approach, including the use of standardised and pilot-tested screening with a well-trained research team.
There are, however, some limitations to our study. First, judging whether the topic is identical or different according to the similarity of the research question (and PICO) may be difficult. In order to overcome this problem, we will perform several pilot tests for selecting SRs/MAs and will operationalise a consensus procedure to judge the similarity of the research question in our review team. Second, we will search only MEDLINE/PubMed for SRs/MAs of surgical intervention trials. To assess effects of a certain intervention, we would have to have a comprehensive dataset of all heretofore conducted RCTs, published or unpublished, and we need to search all possibly relevant databases and beyond. However, the focus of our review is the likely redundancy and contradictions among the SRs in surgery that can be accessed to and therefore used by clinicians and policymakers. We therefore chose the source that is most often used by clinicians worldwide, namely MEDLINE/PubMed. We might therefore be missing some additional relevant SRs/MAs about a given topic. This could potentially indicate that overlapping of SRs/MAs on a single topic is more redundant than we will record. However, limiting the scope of the review to MEDLINE/PubMed will make our study more applicable to the real world use of the SRs/MAs by the clinicians. Third, publication speed may vary across journals: the time lag between manuscript submission (or acceptance) dates and its official publication dates should be considered as an important factor that might affect the coverage ratio of all RCTs that were published until the publication year/month of the SRs/MAs on the certain topic.16 17 We therefore plan to perform sensitivity analyses, if possible, by using date of the last database search to check for the robustness of the observed findings. Fourth, the findings from this review would not be directly generalisable to reviews including both RCTs and non-RCTs, which would have greater sources of heterogeneity in their study conclusions.
Despite possible methodological limitations, our findings of the proposed research may have important implications for researchers and clinical decision-makers. The high variability in results between SRs/MAs on the same topic may indicate a potential risk of the accepted wisdom about the hierarchy of evidence levels.
Acknowledgments
The authors would like to thank all faculty members and research students of Department of Health Promotion and Human Behavior in Kyoto University Graduate School of Medicine and Public Health for their contributions to the constructive discussions about planning of this study protocol in the research meeting.
References
Footnotes
Contributors MK, AK and TAF contributed to the conception and design of the research. MK, AK, MT, KY and TAF contributed to the development of the protocol. MK drafted the manuscript. AK and TAF critically reviewed the manuscript for important intellectual content and all authors read and approved the final protocol.
Competing interests TAF has received lecture fees from Eli Lilly, Janssen, Meiji, MSD, Otsuka, Pfizer and Tanabe-Mitsubishi, and consultancy fees from Takeda Science Foundation; has received royalties from Igaku-Shoin and Nihon Bunka Kagaku-sha publishers; has received research support from Mochida and Tanabe-Mitsubishi. The other authors declare no competing interests.
Provenance and peer review Not commissioned; externally peer reviewed.