Impact of searching clinical trial registries in systematic reviews of pharmaceutical treatments: methodological systematic review and reanalysis of meta-analyses
BMJ 2017; 356 doi: https://doi.org/10.1136/bmj.j448 (Published 17 February 2017) Cite this as: BMJ 2017;356:j448
- Marie Baudard, PhD student1 2,
- Amélie Yavchitz, physician and researcher1 2 3,
- Philippe Ravaud, professor and director1 2 3 4 5,
- Elodie Perrodeau, statistician1 2 3 4,
- Isabelle Boutron, professor1 2 3 4
- 1Université Paris Descartes, Sorbonne Paris Cité, Paris, France
- 2Centre de Recherche Epidémiologie et Statistique, INSERM U1153, Paris, France
- 3Cochrane France, Paris, France
- 4Centre d’Épidémiologie Clinique, Hôpital Hôtel-Dieu, Assistance Publique-Hôpitaux de Paris, 75004 Paris, France
- 5Department of Epidemiology, Mailman School of Public Health, Columbia University, New York, NY, USA
- Correspondence to: A Yavchitz amelie.yavchitz{at}aphp.fr
- Accepted 18 January 2017
Abstract
Objective To evaluate the impact of searching clinical trial registries in systematic reviews.
Design Methodological systematic review and reanalyses of meta-analyses.
Data sources Medline was searched to identify systematic reviews of randomised controlled trials (RCTs) assessing pharmaceutical treatments published between June 2014 and January 2015. For all systematic reviews that did not report a trial registry search but reported the information to perform it, the World Health Organization International Trials Registry Platform (WHO ICTRP search portal) was searched for completed or terminated RCTs not originally included in the systematic review.
Data extraction For each systematic review, two researchers independently extracted the outcomes analysed, the number of patients included, and the treatment effect estimated. For each RCT identified, two researchers independently determined whether the results were available (ie, posted, published, or available on the sponsor website) and extracted the data. When additional data were retrieved, we reanalysed meta-analyses and calculated the weight of the additional RCTs and the change in summary statistics by comparison with the original meta-analysis.
Results Among 223 selected systematic reviews, 116 (52%) did not report a search of trial registries; 21 of these did not report the information to perform the search (key words, search date). A search was performed for 95 systematic reviews; for 54 (57%), no additional RCTs were found and for 41 (43%) 122 additional RCTs were identified. The search allowed for increasing the number of patients by more than 10% in 19 systematic reviews, 20% in 10, 30% in seven, and 50% in four. Moreover, 63 RCTs had results available; the results for 45 could be included in a meta-analysis. 14 systematic reviews including 45 RCTs were reanalysed. The weight of the additional RCTs in the recalculated meta-analyses ranged from 0% to 58% and was greater than 10% in five of 14 systematic reviews, 20% in three, and 50% in one. The change in summary statistics ranged from 0% to 29% and was greater than 10% for five of 14 systematic reviews and greater than 20% for two. However, none of the changes to summary effect estimates led to a qualitative change in the interpretation of the results once the new trials were added.
Conclusions Trial registries are an important source for identifying additional RCTs. The additional number of RCTs and patients included if a search were performed varied across systematic reviews.
Introduction
Systematic reviews are considered to provide the highest level of evidence.12 They are widely used by the developers of clinical practice guidelines, granting health agencies, and journal editors.3456 A major challenge of systematic reviews is to identify all relevant randomised controlled trials (RCTs), whatever their publication status.78910111213 Indeed, results for half of RCTs are never published and the publication status is affected by the nature and direction of results, which may bias the results of the systematic review.14 In some cases, the importance of unpublished trials can be considerable—for example, the addition of unpublished data in the updated Cochrane review assessing the efficacy of neuraminidase inhibitors for influenza modified the conclusion.1516 Initiatives aimed at reducing publication bias include the trial registration policy initiated by the International Committee of Medical Journal Editors (ICMJE) in 2005.17 In 2007, the US Food and Drug Administration Amendments Act required the posting of clinical trial results at ClinicalTrials.gov no later than one year after the date of final collection of data for the prespecified primary outcome, for all phase II to IV trials of drugs, biological treatments, and devices.1819 The research community has embraced this policy, and there was a noticeable increase in trial registration around the time of implementation of the ICMJE policy.20 In April 2016, about 90 000 completed experimental studies were registered at ClinicalTrials.gov (the largest registry), and 16 500 have results posted.
When performing systematic reviews, the search of trial registries is now considered an essential tool.3212223 Previous studies showed that searches of clinical trial registries are not systematically reported by authors of systematic reviews,242526 but to our knowledge, none had systematically performed a search of trial registries to quantify the impact of searches (ie, to quantify the number of missing trials identified by a search and the change in summary statistics when these missing trials are considered). In this study we describe whether and how clinical trial registries were searched in published systematic reviews of pharmaceutical treatments and evaluate the impact of searching registries on the identification of additional RCTs (ie, eligible completed or terminated RCTs not included in the systematic review).
Methods
Identification of systematic reviews
Search strategy
On 16 March 2015 we systematically searched Medline through PubMed for all systematic reviews of RCTs assessing pharmaceutical treatments that were published in English between 1 June 2014 and 31 January 2015. We did this by searching for “Meta-Analysis[ptyp] AND (“2014/06/01”[PDAT]: “2015/01/31”[PDAT]) AND English[lang] appearing in the title, abstract, or keywords.
Inclusion and exclusion criteria
One researcher screened the titles and abstracts of citations retrieved to identify all reports of systematic reviews of RCTs with at least one meta-analysis including at least two RCTs and assessing a pharmaceutical treatment (ie, drug, health related biological product, or nutritional supplement). We excluded updates of previously published systematic reviews and systematic reviews of diagnostic test accuracy, prognosis, economics evaluations, genetics, non-RCT studies, network meta-analyses, and indirect comparison meta-analysis as well as individual patient data meta-analyses. The full text of potentially relevant citations was obtained. As a quality control procedure, another researcher independently screened 20% of the citations and confirmed the eligibility of all systematic reviews included. Discrepancies were discussed to reach consensus.
Data extraction
From the published reports and supplementary appendices, two researchers independently recorded, when available, the general characteristics of the systematic review (type of journal: general medical, specialty, or Cochrane review), the funding source (not-for-profit, for-profit, not reported, or unclear), and the number of RCTs and participants included in the systematic review. They also recorded the reporting of the clinical trial registry search (ie, whether such a search was reported, the name and type of registries searched, and whether the results of the search (the number and identification of RCTs identified from the search) were reported. Any disagreement was resolved by discussion and consensus.
Impact of searching clinical trial registries
For each systematic review that did not report a search in clinical trial registries, we systematically performed a search reproducing the conditions of the original search reported in the systematic review, particularly taking into account the date of the search and the inclusion criteria of the systematic review.
Search strategy
Our search strategy followed the same search and selection process described by the authors of the published systematic reviews.
Firstly, from the selected full text articles and all available supplementary materials we systematically recorded the search terms related to the condition and interventions used by authors and the date of the last electronic search. From this analysis we excluded systematic reviews that did not provide search terms or the date of the search.
Secondly, we searched the World Health Organization International Clinical Trials Registry Platform (WHO ICTRP) search portal, which contains the trial registration datasets provided by several registries. We chose this portal because it includes 16 national and international primary registries, including ClinicalTrials.gov. In the advanced search window of the portal (http://apps.who.int/trialsearch/) we entered the search terms recorded in the “condition” and “intervention” fields with Boolean operators. We chose “all” in the “recruitment status” field and “Search for clinical trials in children” when appropriate. Appendix 1 provides details of the search strategies and keywords for each systematic review.
Identification of completed or terminated RCTs
For each search we downloaded all the citations retrieved and identified all studies with a recruitment status recorded as “completed” or “terminated”.
For each systematic review, two researchers independently screened the records retrieved and selected all completed or terminated RCTs not already included in the systematic review that fulfilled the systematic review’s eligibility criteria for participants, interventions, and comparator. We systematically verified in the history or archives of the registry that the recruitment status was recorded as completed or terminated before the date of the search (see appendix 2). Any disagreements were resolved by consensus. A third researcher screened all selected records to confirm their inclusion.
Availability of RCT results
Two researchers independently determined whether the trial results were available (posted, published, or available on the sponsor’s website) for each selected RCT. We searched for results posted on clinical trial registries and publications referenced on the trial registry, performed an electronic search of PubMed and Google, and searched the sponsor’s website. All trials with results available were screened, and we selected only trials for which the results became available before the last electronic search of the systematic review.
Inclusion of the RCT results in meta-analyses
We recorded the number of meta-analyses reported in the systematic review, the number of meta-analyses that could include the additional RCTs, and the number of meta-analyses for which all the RCTs identified had results available and could be included in the meta-analysis.
Finally, we determined the impact of including the RCTs on treatment effect estimates. For this purpose, we used an algorithm to select one meta-analysis in which at least one RCT with results available could be included.
For each systematic review we recorded all the outcomes of the meta-analyses reported in the systematic review report. For each eligible RCT with results available, we determined whether the RCT could be included in the meta-analyses previously recorded—ie, the RCT reports included:
Continuous outcomes—sample size, mean, and one measure of dispersion (standard deviation, standard error, or confidence interval) by group. Standard errors and confidence intervals were converted into standard deviations to perform the meta-analyses. When results were given for separate subgroups, we pooled the results, the pooled sample size being the sum of the subgroup sample sizes and the pooled mean being the weighted mean of the subgroups and the pooled standard deviation combined.3
Binary outcomes—sample size and number of events by group.
Time-to-event outcomes—hazard ratio and 95% confidence interval or median survival times and confidence intervals by group.27
Finally, when RCTs could be included in several meta-analyses, we selected only one meta-analysis according to the following order of outcomes analysed: the primary efficacy outcome of the systematic review, the primary harms outcome, and the patient important outcome, such as mortality, quality of life, or morbidity. If several of these outcomes could be used to include new RCTs, we selected the first meta-analysis reported. If none of these outcomes could be used to include a new RCT, we selected the first meta-analysis reported.
For each meta-analysis selected, we extracted the outcome data from the RCTs identified (eg, number of events and number of patients in each group, means, standard deviations). When the outcome data were available from several sources, we extracted a single source according to a prespecified order; data reported in the registry, data reported in a published report, and data reported on the sponsor’s website.
Data analysis
R v3.1.0 (www.r-project.org), the R Project for Statistical Computing, was used for statistical analysis. Qualitative variables are represented by percentages, and quantitative variables by medians (interquartile ranges). In a post hoc analysis, we used a χ2 test to compare the proportion of reviews reporting a trial registry search according to the type of systematic review (Cochrane v non-Cochrane) and funding source (not-for-profit or not funded v for-profit, funding not reported or unclear).
For the meta-analysis selected for recalculation (one for each selected systematic review), we calculated the summary statistics (risk ratios, odds ratios, hazard ratios, mean differences, or standardised mean differences) and the I2 statistic (measure of heterogeneity) with and without trials retrieved by a trial registry search. We used similar methods to those of Hart et al.28 We reported the magnitude of the change in the result of the meta-analysis as a percentage change in the summary statistic after including data from the RCTs retrieved after the registry search. For risk ratios and odds ratios, we calculated the percentage change of the log transformation as (log(E)−log(I))×100/log(E), where E is the effect estimate excluding newly retrieved data and I is the effect estimate including newly retrieved data. We calculated the log transformation for relative risks and odds ratios so that the point of “no effect” was equal to 0 instead of 1, thus allowing for a calculation of percentage change. For weighted mean differences, we calculated the percentage change by using the formula (E−I)×100/E.
We reanalysed the published meta-analyses by using the same statistical method (Peto, Mantel-Haenszel, inverse variance), analysis model (fixed v random effects), and measure of effect (risk ratio, odds ratio, weighted mean difference) used by the original authors. For all meta-analyses, we assessed heterogeneity by calculating the I² statistic and τ² (DerSimonian-Laird estimate).
Results
Identification and characteristics of reports
Among the 2249 citations retrieved, we included 223 reports of systematic reviews with meta-analyses (fig 1⇓). Table 1⇓ lists the characteristics of the included systematic reviews. One third (35%) were Cochrane reviews; a median 10 (interquartile range 6-18) RCTs were included in the systematic reviews,6789101112131415161718 with a median 1594 (614-5027) patients.
Characteristics of included systematic reviews and registry searches. Values are numbers (percentages) unless stated otherwise
Reporting of registry search in systematic reviews
Among the 223 systematic review reports included, 107 (48%) mentioned searching at least one clinical trial registry: 48 of these (45%) reported searching only individual registries, 11 (10%) only portals, and 44 (41%) a combination of individual registries and portals. Four did not report the type of registry or portal searched. The portal and individual register most frequently searched were the WHO ICTRP search portal (n=53, 50%) and ClinicalTrials.gov (n=89, 83%), and for 40 studies (37%), both were searched. In only 47 of the 107 (21%) reports were the results of the clinical trial registry search clearly described (ie, with a description of the number and identification of RCTs found from the search) (fig 1⇑, table 1⇑): 16 of these 47 reviews (34%) did not retrieve any eligible RCTs, 11 (23%) retrieved only ongoing studies, 13 (28%) retrieved at least one completed or terminated RCT with no results available, and 7 (15%) retrieved at least one completed or terminated RCT with results identified. Of these last seven, three included RCTs in at least one meta-analysis.
A search of a trial registry was more frequent in Cochrane than in non-Cochrane reviews (65/77 (84%) v 42/146 (29%), P<0.001) and in not-for-profit funding or no funding than for-profit funding, funding not reported, or funding unclear (79/139 (57%) v 28/84 (33%), P<0.001).
Impact of searching registries
Identification of completed or terminated RCTs
Among the 116 systematic reviews not reporting a search in trial registries, we were not able to perform the registry search for 21 (18%) because the search date or the keywords were not reported. Therefore the search was performed for 95 systematic reviews. Among the 15 282 records screened (median 23 (interquartile range 6-150) for each systematic review), we identified 122 eligible RCTs terminated or completed (involving 52 743 patients) not originally included in the systematic review. Among the 122 RCTs, 104 (85%) were classified as completed and 18 (15%) as terminated. Among the 18 RCTs classified as terminated, three had results available and were included in meta-analyses: two were stopped early because of adverse events and one was stopped early because of futility. The remaining 15 RCTs had no results available and no information on the reason for stopping early.
Availability of RCT results
Overall, the trial registry searches identified at least one eligible RCT for 41 of 95 (43%) systematic reviews, with a median 9% (interquartile range 4 to 18)456789101112131415161718 additional patients for each systematic review (fig 2⇓, table 2⇓, see appendix 3). Among these 41 systematic reviews with additional RCTs identified, the number of patients included increased by 10% to 20% in nine, 20% to 30% in three, 30% to 40% in two, 40% to 50% in one, and more than 50% in four.
Fig 2 Identification of trials by searching clinical trial registries
Effect on meta-analyses of adding randomised controlled trials (RCTs) retrieved from clinical trial registries
We identified results for 63 of 122 RCTs (52%) involving 42 202 patients. Of these 63 RCTs, 45 (71%) involving 21 358 patients could be included in the quantitative analysis (ie, reported sufficient data to be included in at least one meta-analysis of the systematic review). The 18 remaining RCTs with results could not contribute to the quantitative analysis because of differences in definition or metrics used between the outcome reported in the RCT and the outcome of the systematic review or outcome reporting bias.
The results of the 63 RCTs identified were posted (n=41, 65%), published as identified by a reference reported on the registry (n=21, 33%) or from a complementary search (n=10, 16%), or were available on the company’s website (n=31, 49%). The results were available in one (n=29, 46%), two (n=27, 43%), or three sources (n=7, 11%).
For 14 systematic reviews, the trial registry searches allowed for identifying RCTs with results (n=45) that could contribute to the quantitative analysis. Among the 73 meta-analyses reported in these 14 systematic reviews; the search in trial registries retrieved additional results that could be included in 59 meta-analyses. Overall, 31 of 59 (53%) meta-analyses were considered complete (ie, all the RCTs identified had available results and could be included in the meta-analysis).
Inclusion of RCT results in meta-analyses
Finally, we recalculated the effect estimates for the selected meta-analyses from the 14 systematic reviews including RCTs that could contribute to the quantitative analysis. The 14 meta-analyses selected included eight efficacy outcomes and six harms outcomes. In the meta-analysis without additional RCTs, results for 12 of 14 outcomes statistically significantly favoured the experimental treatment, and results for two did not differ from the comparator.
The weight of the eligible RCTs included ranged from 0.2% to 58% and was greater than 10% for five of 14 systematic reviews, 20% for three, and 50% for one. The change in summary statistics ranged from 0% to 29% and was greater than 10% for five of 14 systematic reviews and greater than 20% for two. For example, in the meta-analysis with a 29% change in summary effect, the mean difference changed from −0.35 (95% confidence interval −0.51 to −0.19) to −0.45 (−0.55 to −0.36), for a larger effect after inclusion of the new RCTs. However, including the RCTs identified by a trial registry search did not change the statistical significance or direction of the results. Table 2⇑ provides detailed descriptions of the 14 meta-analyses.
Discussion
Despite recommendations,23 about half of the published systematic reviews performed a search of clinical trial registries and only one fifth reported the results of the search. When we performed the registry search, we identified additional studies for 43% of the systematic reviews. We reanalysed 14 meta-analyses to include data from randomised controlled trials (RCTs) retrieved by the trial registry search. The addition of data from registries mainly adds to the precision of summary estimates, but none of the changes led to a qualitative change in the interpretation of the results once the new trials were added.
Comparison with other studies
Our results are consistent with other studies, showing that the search for unpublished trial data is still often lacking in systematic reviews.2425262930 In a random sample of 300 recent systematic reviews indexed in Medline in February 2014, only 19% reported searching trial registries.31 A previous study by Hart in 2012 aimed to reanalyse meta-analyses by adding unpublished outcome data from trials obtained from the US Food and Drug Administration to published meta-analyses.28 The study documented that the addition of unpublished data obtained from the FDA could change the magnitude of the effect size or, in a few cases, the statistical significance of meta-analyses. A systematic review in 2016, aimed at quantifying the impact of the underreporting of adverse events in systematic reviews, showed that the inclusion of unpublished data might reduce the imprecision of pooled effect estimates in meta-analysis of adverse events.32 However, to our knowledge the impact of searching trial registries in terms of identifying trials and their inclusion in the analysis when results are available has never been evaluated.
Limitations of this study
Our study has some limitations. Firstly, we searched only the World Health Organization International Trials Registry Platform (WHO ICTRP) search portal using the keywords reported by authors for their electronic search. Consequently, we cannot claim that we identified all RCTs. However, this portal brings together 16 national and international primary registries, including ClinicalTrials.gov. Furthermore, in a previous study, the overlap between ICTRP and ClinicalTrials.gov was good, because all records identified in ClinicalTrials.gov were also identified in WHO ICTRP.33 Secondly, we did not account for eligibility criteria related to trial quality. The quality assessment of data recorded from trial registries is difficult and some trials could secondarily be excluded because of insufficient quality. Thirdly, we did not attempt to contact investigators of the unpublished trials to obtain results. In fact, we aimed to reproduce the condition the authors encountered and it would not be appropriate to ask authors for results after such a delay. Furthermore, we did not search for additional data presented in conference abstracts or search FDA and European Medicines Agency websites. Therefore, the number of systematic reviews with trials identified by a search of clinical trial registries and the amount of data from RCTs retrieved from such registries might be underestimated. Fourthly, to make the workload manageable, we included only one meta-analysis for each systematic review. Finally, we focused on only systematic reviews of pharmaceutical treatments and cannot extrapolate to non-pharmaceutical treatments because the regulation for trial registration and posting of results is less stringent with these treatments.
Implications for clinicians and policy makers
Clinical trial registries have been developed to reduce waste in research, and publication bias. Their use has been enforced by editors and policy makers. They have been considered to contribute to greater transparency and increasing the value of research.
Searching clinical trial registries is recommended when performing systematic reviews. In our study, the addition of new RCTs in meta-analyses affected treatment effect estimates but did not change the statistical significance of the results or the direction of the treatment effect, although it increased precision.
Nevertheless, searching clinical trial registries remains an essential recommendation for the conduct of systematic reviews and should be enforced. In fact, the objective of systematic reviews is to collate all empirical evidence.3 However, overall, results for only about half of clinical trials are published, and searching only electronic bibliographic databases gives access to just the “tip of the iceberg”.242526 Finally, searching trial registries in general represented a low burden. The number of records to screen by systematic review was low: median 23 (interquartile range 6-150). The results for 41 of 63 trials were posted at ClinicalTrials.gov and therefore were immediately available. Furthermore, a previous study showed that the reporting of results was more complete at ClinicalTrials.gov than in published reports.34 However, one important limitation of this search is the lack of availability of the results for completed trials and the low level of details on the methodological quality recorded in the registries. Some initiatives to facilitate access to clinical trial results, such as the Food and Drug Administration Amendments Act 2007, which requires the posting of clinical trial results18 or pharmaceutical company policies,35 have been implemented. Some researchers have developed an intervention to improve posting, such as emailing a reminder about section 801 of the Food and Drug Administration Amendments Act requirement to responsible parties36; other interventions are necessary. Recently, the new rules of trial registration at ClinicalTrials.gov require submission of a full protocol and statistical analysis plan at the same time as submission of results.37 Registries could be an even more important source of results in the future.
Conclusion
Searching clinical trial registries is essential for identifying additional trials that could increase the value of systematic reviews. However, the lack of available RCT results limits the value of the search. Searches of trial registries should be promoted and enforced, as should the posting of trial results.
What is already known on this topic
Despite recommendation, searches of clinical trial registries are not systematically reported by authors of systematic reviews
What this study adds
Searching clinical trial registries does identify additional trials for a systematic review of the literature, thus increasing the potential value of the review
Footnotes
We thank Laura Smales (BioMedEditing, Toronto, Canada) for English language proofreading and Carolina Riveros for data extraction.
Contributors: MB, AY, IB, and PR conceived, designed the study, and wrote the protocol MB, IB, and AY wrote the first draft. AY, IB, EP, and PR contributed to the writing of the manuscript. MB, AY, EP, and IB analysed the data. AY and IB had access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. MB and AY contributed equally to the manuscript. AY is the guarantor.
Funding: This study received no funding.
Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.
Ethical approval: Not required.
Data sharing: All data from this study—including literature searches, additional explanatory material, and data extraction forms—are available on request.
Transparency: The lead author (AY) affirms that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned have been explained.
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.