Article Text
PDF
Abstract
Objective To investigate to what extent evidence from previous similar trials or systematic reviews was considered before conducting new trials.
Design Cohort study of contemporary protocols for trials with ethical approval.
Methods All protocols for randomised trials approved by the five ethical committees in Denmark between January 2012 and March 2013 were screened for eligibility. Included protocols were read in full to determine whether a systematic search had been conducted and references were checked to evaluate whether trial rationale and design could be challenged for not adequately considering previous evidence. To investigate whether protocols cited relevant trials, we used simple search strategies that could easily be conducted by researchers without experience with literature searches.
Results Sixty-seven protocols were included. Only two (3%) of the protocols explicitly stated to have conducted a literature search and only one (1%) provided information that allowed the search to be replicated. Eleven (16%) of the protocols described trials where we found the information insufficient to judge if the trial was ethically justified, either due to a comparator that was not supported by the presented evidence (six protocols), because they did not present a rationale for conducting the trial (two protocols), or for both reasons (three protocols). For eight (12%) of the protocols, our search identified trials that could have been relevant to cite as justification.
Conclusions While most protocols seem to adequately consider existing evidence, a substantial minority of trials might lack a sufficient evidence base. Very few trials seemed to have been based on a literature search which makes it impossible to know whether all relevant previous trials had been considered. Rules for ethical approval should include requirements for systematic literature searches to ensure that research participants are not exposed to sub-optimal treatments or unnecessary harms as well as to reduce research waste.
- medical ethics
- clinical trials
- statistics & research methods
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
For the first time, we have examined to what extent contemporary protocols describe systematic literature searches and use the results to inform trial design.
We performed our own searches to identify any relevant trials that could have been cited.
Our sample only contain trials approved in Denmark and the sample size was not large enough to perform inferential statistics.
The legal regulation has changed since the protocols in our sample were approved, meaning that the ethical committees now have access to the investigator’s brochure (a document containing clinical information on the studied intervention) which could be important for our conclusions.
Introduction
Medical research involving humans must meet high ethical standards. The Declaration of Helsinki specifies that a research project should only be carried out ‘if the importance of the objective outweighs the risks and burdens to the research subjects’.1 As a prerequisite, the Declaration underlines that trial participants are properly informed about the harms and benefits of the studied interventions. Similar requirements are found in the European Clinical Trials Directive2 and in the Good Clinical Practice guidelines published by the European Medicines Agency.3 It follows that before a new trial is undertaken, knowledge gained from previous similar trials needs to be considered for trial planning and must also be communicated to the participants.
In 2013, the Standard Protocol Items: Recommendations for Interventional Trials statement was published. It outlines 33 items (with sub-items) that should be adequately reported in clinical trial protocols. Item six is ‘Background and Rationale’ which describes the importance of justifying a new trial in the context of the available evidence. It is ‘strongly recommended that an up-to-date systematic review of relevant studies be summarised and cited in the protocol’.4
In Denmark, a systematic literature review is not required in protocols for randomised trials although the Danish Medicines Agency state in their guidelines for applications for clinical trials that relevant results from previous clinical and non-clinical studies must be reported in trial protocols.5–7 For decades, researchers have argued that in order for a study to be scientifically and ethic-ally justifiable its design should take previous research into consideration based on a systematic review.8 9 We obtained a cohort of trial protocols approved by one of the five regional research ethics committees in Denmark and used this cohort to study whether the ethical approval system ensures that trials justify their scientific rationale and use of comparators based on previous trials and take their results, whether positive or negative, into account.
Methods
Access to trial protocols is possible through the Danish Freedom of Information Act. Between 1 October 2013 and 28 February 2014, we screened the titles of all research projects approved by either one of the five regional ethics committees in Denmark between January 2012 and March 2013. The research projects could be found on the website of the Danish National Committee on Health Research Ethics which functions as a common web-page for all five regional committees. Eligible protocols were then requested from the ethics committees.
Inclusion and exclusion criteria
A protocol was eligible if it described a randomised, parallel group trial and had prespecified non-surrogate primary outcomes. We excluded trials with only surrogate outcomes as determining the relevance of such outcomes to patients require detailed content area knowledge from diverse fields. We also excluded trials that could not be identified via trial registries as our initial assessment of eligibility was based on information from these registries.
As we identified substantially more eligible trials than we could realistically extract data from, we limited our predefined period of inclusion to 1 October 2012 to 31 March 2013.
The website of the ethics committees only contained information on the date of approval, the project title, the Danish region where the trial would take place and the name of the coordinating investigator. We sought additional information (described below) about the research projects through clinicaltrials.gov, the EU Clinical Trials Register (EudraCT) and the WHO International Clinical Trial Registry Platform using information from the website of the ethics committee (for example, intervention or a trial identifier found in the project title). If we were unable to identify the eligible studies in trial registries, we attempted to identify a trial ID through Google searches using the information from the committee’s website. Trial characteristics from these registries were extracted, and eligibility was assessed by one observer. We extracted information on the following characteristics: study type, design, population, interventions, inclusion and exclusion criteria of the trial, primary outcomes and desired sample size. When there was uncertainty about eligibility, a second observer was consulted.
For trials that we considered potentially eligible based on information from trial registries, we contacted the relevant regional ethics committee and requested copies of the protocols, informed consent forms, financial and publication agreements between the study sponsors and the investigators, and any other relevant information about the trials, for example, the investigators' brochure. We emphasised that the results would be published in a manner that would not allow identification of individual trials.
Based on the protocols, we made a final assessment of eligibility and assigned each trial a unique, anonymised identifier.
Data extraction
As the protocols were long and contained much information irrelevant to our project, one observer entered applicable text into a Word document. The introduction or background section as well as any sections of the protocol addressing ethical issues or clinical information on the used interventions were extracted this way. All subjective judgements based on the extracted texts were performed by two observers independently and all ambiguities or disagreements were discussed, if necessary involving a third observer.
Additionally, all trials where the choice of treatment and comparator could be questioned were discussed with a senior researcher. Our assessments were entered into a standardised data extraction sheet.
Any information in the protocols about the source of funding and the type of comparator used were also extracted from the protocol and entered into the data extraction sheet.
Funding issues
A trial was considered fully industry sponsored when a commercial company was listed as the primary or only sponsor; partially industry sponsored when the primary sponsor was a non-commercial entity but a commercial company provided either funding, devices, medications, manpower or similar to the project; and non-industry funded when the sponsor was a non-commercial entity and no commercial funding, devices, medications, manpower or similar was received.
Type of comparator
The type of comparator used in the control arm was classified as either an active comparator, placebo or ‘nothing’. In trials that used a comparator classified as ‘nothing’ participants received either no treatment or were put on a waiting list. Trials with more than two arms could use more than one type of comparator; in this case we classified a trial as using a combination of the above, for example, a trial could be classified as having a placebo arm as well as an active comparator arm. Comparators described in the protocol as ‘usual/standard care’ could be either ‘active treatment’ or ‘no treatment’ and was classified according to the description in the protocol.
Justification for choice of comparator
We noted whether the choice of comparator in the individual protocol was justified as recommended in the Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) reporting guidelines.4 10 We distinguished between an explicit and an implicit justification.
We defined that a protocol explicitly justified the choice of comparator when a specific reason for the choice was given, for example, a section such as ‘Rationale for choice of comparator’ or statements such as ‘Regarding justification of placebo: "Placebo is the appropriate comparator, since the approved therapies available in some countries are not routinely used for treatment of lower-risk disease."’
The justification for choice of comparator was considered implicit when, for example, the control group was stated to simply receive the usual standard of care or when there was documented uncertainty about which of two active interventions was superior.
Literature search
We noted whether it was stated or implied in protocols and related documents whether a systematic literature search had been performed and if search strings, restrictions, filters, dates for searches and names of databases were described.
Citations of relevant trials or systematic reviews
We read the included protocols in full and checked their references to identify trials of similar interventions, for similar indications and in similar populations. We also checked all references in any systematic reviews that were cited in the protocol to see if these identified relevant trials. When we checked the existing trials and systematic reviews, we looked for both evidence on benefits and harms.
Our searches for relevant trials
For each protocol, we conducted systematic searches to identify additional randomised trials or systematic reviews that could have been relevant to cite in the protocols. We restricted the study design to randomised trials when searching and we used simple and broad search strategies that could have been performed by researchers without experience with systematic literature searches. We searched PubMed and EMBASE and the search strings followed a general template:
[Indication] AND [intervention]
For example, the following search string was used for a study of the use of surgical mesh in inguinal hernia operations: inguinal hernia AND mesh AND fixation.
All searches were restricted to publications entered into the databases at least one month before the first submission of the protocol to the research ethics committee. We screened titles and abstracts from our searches and potentially eligible trials were read in full text to assess their relevance.
Analysis
We compared the references in the protocols with the results of our own systematic search.
We deemed a comparator questionable if:
One or more previous randomised trials conducted with the same intervention, for the same condition and using the same outcome had found that the intervention was superior to the proposed comparator and the choice was not further justified in the protocol.
We deemed the rationale for conducting a study questionable if a protocol did not provide any evidence of clinical equipoise to justify a randomised controlled trial. We based this judgement on the principles outlined in item 6A of the SPIRIT statement4 and in the SPIRIT explanation and elaboration paper10 where it is explained that a protocol should ‘summarise the importance of the research question, justify the need of the trial in the context of available evidence, and present any available data regarding the potential effects of the interventions (benefits and harms)’.4 10 Thus we examined if protocols:
Identified a lack of studies of direct relevance.
Established that results of previous studies were inconclusive.
We also considered whether the choice of outcomes and methodology (for example, timing of treatment) was appropriate for the scientific question posed, based on the available evidence.
For both the justification of comparators and the scientific rationale for the research question, we did not judge whether a trial was unethical but examined whether the information presented would enable ethics committees to evaluate if the trials were justified.
For studies where we found insufficient information, we summarised the reasons in a tabular format.
Patient and public involvement
Patients were not involved in this study.
Results
Screening and retrieval of protocols
The regional ethics committees approved a total of 1401 protocols between 1 January 2012 and 31 March 2013. We excluded 1189 protocols either because the trials could not be identified in trial registries or via Google searches (n=794) or because they did not fulfil our eligibility criteria (n=395). This is summarised in figure 1.
Flow chart of included studies.
The remaining 212 protocols all seemed to describe trials with a randomised parallel group design and patient relevant outcomes, based on the available information. As our desired sample size was 60 protocols we excluded all trials approved prior to October 2012 which limited our sample to 78 protocols. We applied to the committees for full access to the protocols and any related documents.
Even though we stressed in our application that results of this project would be published in a way that would not allow identification of individual trials access was initially denied or the protocols were redacted for 25 trials (37%). Details of the redactions in protocols for trials that were industry sponsored are published elsewhere.11 We gained access to the full unredacted protocols after an appeal to the Danish National Committee on Health Research Ethics, a process which took several years and involved lawyers. This process is also described elsewhere.11
After reading the documents we had received, we excluded 11 trials because it turned out they either did not use a parallel group design (n=4), only used surrogate primary outcomes (n=6) or was a duplicate (n=1). This led to our final sample of 67 eligible protocols.
Study characteristics
Of the 67 included protocols, 33 (49%) were for fully industry sponsored trials, 10 (15%) were partially industry sponsored, and 24 (36%) were non-industry sponsored. Thirty of the fully industry sponsored trials (91%) were multinational, in contrast to five partially industry sponsored trials (50%) and three non-industry sponsored trials (13%).
The prespecified sample sizes in the protocols ranged from 30 to 18 000 patients (mean 1124, median 400). Industry sponsored trials generally had considerably larger sample sizes than trials with no or partial industry funding (see table 1).
Prespecified sample size by type of funding
Nineteen protocols (28%) described trials in oncology and 10 protocols (15%) surgical interventions. The specialities of the remaining 38 protocols can be seen in table 2.
Medical specialities of included protocols
Comparators
Placebo was the only comparator in 18 (27%) of the trials. Thirty-two trials (48%) used active comparators only and 10 trials (15%) used no treatment as the only comparator. Six trials (9%) used both a placebo-arm and an arm with an active comparator. One trial (1%) used both an active comparator arm and an arm with no treatment.
Twenty protocols (30%) described the comparator as ‘usual care’ which in 18 cases was an active comparator and in two was no treatment.
The protocol authors justified their choice of comparator in 42 protocols (63%). The justification was explicit in 21 protocols and implicit in 21, for example, by mentioning that participants in the control group would receive ‘standard care’.
We identified 11 protocols (16%) where the choice of treatment or comparator could be questioned given the evidence available at the time and the information provided in the protocols. The reasons for our judgements are described in box 1.
Examples of studies where the justification could be challenged, based on the information available in the protocols.
A study examined the effect of a procedure on the fertility rate. The control group did not get a procedure that the protocol authors described as the gold standard.
A phase 3 study was initiated before phase 2 studies were completed although the studied drug belonged to a class known to be quite toxic.
A study examining treatment for serious cardiovascular disease used a bare metal stent as comparator, even though a systematic review found that a drug-coated stent was more effective regarding the primary outcome of the study.
A study examining the effect of a special form of exercise used a control group that received no intervention even though the effects of exercise on the primary outcome (quality of life) were well established.
A study compared ‘liberal’ and ‘restrictive’ red blood cell transfusion practices despite citing a systematic review which found that, ‘According to the results of the largest RCT, maintaining a higher haemoglobin level… seems to confer little clinical benefit.’ Furthermore, it has been shown that blood transfusions carry important harms.
A study compared a training programme that started in the hospital and continued at home with usual practice (ie, very little rehabilitation) in elderly patients. The benefits of training were well established.
A study convincingly established that a special diet is effective in reducing postoperative infections but the most effective timing of intervention had yet to be established; however, preoperative administration of the diet was compared with usual diet rather than a different timing of the diet.
A study examining relapse free survival for a type of cancer compared an investigational drug with observation only although other treatments had shown effects on the primary outcome when compared with observation only.
Three studies examined the efficacy of analgesic drugs or techniques and used placebo as comparator, although other treatments had been proven effective. All the studies allowed for the use of rescue medication in some form, but it was clear that patients in the placebo groups received sub-optimal treatment.
Literature searches described in protocols and comparison with our search results.
Only 2 (3%) of the 67 protocols explicitly stated to have conducted a literature search and only one of these provided full information that allowed the search to be replicated. The other protocol disclosed only when the search was performed and the databases searched; there were no search strings or information on any restrictions used. Four additional protocols used phrases indicating that a literature search may have been done, for example, ‘Review of literature… suggest’, ‘According to searches on the PubMed database’, ‘Unfortunately only a few studies examine the effect of MP on acute pain… (PubMed search Feb. 2012)’ and ‘A review of the literature thus indicates…’.
Thus, if very broad criteria are applied, the protocol authors did a literature search in six cases (9%).
Twelve protocols (18%) cited either a systematic review or a randomised trial with clear, direct relevance for the intervention, population and/or indication studied. Two protocols (3%) cited one or more systematic reviews, eight protocols (12%) cited one or more randomised trials and two protocols (3%) cited both systematic reviews and randomised trials. For 11 of these 12 protocols (92%), we did not find additional relevant trials through our own systematic search. For the remaining one protocol (8%), we identified one systematic review and two randomised trials (819 included trial participants) that could have been relevant to cite.
The remaining 55 protocols (82%) cited no relevant systematic reviews or randomised trials. However, for 48 of these 55 protocols (87%), we did not identify any studies that could have been cited. For the remaining seven protocols (13%), we identified 10 randomised trials and one systematic review (with a total of 2080 included trial participants) that could have been cited. These results are summarised in table 3.
Citations of trials and reviews of direct relevance
Discussion
Principal findings
We found that only one (1%) of the 67 included protocols described a reproducible systematic search for previous randomised trials of the same intervention. Even when applying very broad criteria, only six (9%) protocols indicated that a search may have been carried out, whether systematic or not. We found 12 (18%) protocols that cited relevant systematic reviews or randomised trials. The remaining 55 (82%) protocols cited no such evidence but our own searches did not identify any relevant systematic reviews or randomised trials for 48 of them.
Sixteen percent of included trial protocols either did not present a rationale for conducting the study or used comparators that could be questioned based on the evidence available at the time and did not provide information to explain these choices. While these trials may be ethically acceptable, we were unable to confirm this based on the information available in the protocols. Thus, a considerable number of research participants could potentially have been exposed to sub-optimal treatment or unnecessary inconvenience, discomfort or risk of harm.
A systematic search for previous trials is not an explicit requirement for ethical approval in Denmark but it is difficult to see how researchers can know whether relevant previous trials exist without performing a systematic search, especially considering the high number of new publications today. In 2010, a study found that 75 reports of trials and 11 systematic reviews were published every day and these numbers had been rising.12
Although conducting trials on topics where previous trials exists can be warranted (eg, to replicate findings or when a previous trial could not answer the research question due to either size or quality), there is evidence that superfluous trials represent a considerable waste of resources, both in terms of financial and intellectual resources.13 A formal requirement for systematic searches in trial protocols may reduce this waste.
Comparisons with other studies
There are historical cases of superfluous trials, for example, of intravenous streptokinase as thrombolytic treatment. In 1992, a cumulative meta-analysis showed that in 1973, after just eight trials of 33 total performed since the late 1950s, a consistent and statistically significant reduction in total mortality was shown. The remaining 25 trials (with a total of 34 542 participants enrolled) had little effect on the OR and only narrowed the CI.14 Still, new trials of the intervention were performed until the late 1980s. As meta-analyses were not routinely used at this point in time, we should not judge this by modern standards. However, it highlights the importance of examining existing evidence before conducting a new trial.
Several previous studies have examined published protocols to assess if they live up to ethical requirements. In 2016, a study found that 41% of 101 trial protocols cited any randomised trial or a systematic review, whether the trials addressed a similar question or not.15 We found that only 18% of protocols cited a randomised trial, which may be because we only included trials that we found to be of direct relevance for the protocol. Additionally, the 2016 study included only published protocols whereas we included any protocol given ethical approval.
Several studies have investigated whether publications of randomised trials reference previous trials, and it has repeatedly been shown that this is often not the case. In 2011, a study of 1523 trials found that less than a quarter of the relevant previous trials were cited, and that for the 1101 RCTs where five or more previous trials could have been cited, 23% cited no trials and another 23% cited only one.16 In 2010, Clarke et al. reproduced their previous findings from 1997, 2001 and 2005 that most reports of trials fail to cite updated systematic reviews when discussing their results.9
Strengths and limitations
To our knowledge, this is the first cohort of contemporary protocols approved by an ethics committee that have studied whether they live up to the ethical standards expressed in the Declaration of Helsinki.1 We chose to be conservative when judging whether the choice of comparator was reasonable given existing evidence for potentially effective treatments and whether an evidence base supporting the rationale for the investigated treatment was provided, because such assessments are inevitably subjective.
Our included protocols are over five years old but as systematic literature searches are still not mandatory for approval of protocols in Denmark, our results are likely valid today. Ethics committees in Denmark now have access to the investigators’ brochure which might contain some of the information we looked for but did not find.
Some of the protocols in our sample described trials involving medications or devices at an early stage of development and for such trials systematic reviews are unlikely to exist. However, a systematic search could still be relevant as similar interventions may have been tested. In 2006, six participants developed multiple organ failure after a phase 1 trial in the UK, and it has been suggested that a systematic review of preclinical and clinical data could have predicted the life-threatening adverse effects.17 18
We did not search for unpublished trials in our own systematic searches. Publication bias is a significant problem in medical research,19 and trials with positive results are more likely to be published. Thus, there could be relevant studies that we did not identify. However, expecting protocol authors to find these may be unreasonable.
We did not search for observational studies, although they can be important for detecting rare or unexpected harms. It is therefore possible that we would have found additional interventions or comparators to be problematic, had we included such studies
Finally, the reporting and reproducibility of our study is limited by the confidentiality agreements signed in order to obtain access to protocols.
Implications for practice and research
Other researchers have highlighted problems with the current system of ethical approval and shown examples of cases where unethical studies were granted ethical approval.15 20 Our review supports the need for policy changes.
We suggest that protocols and other documents should be made publicly available as soon as the protocol has received ethical approval. These documents are currently very difficult to get access to, especially for trials with commercial sponsors.10 Access to such documents is of vital public interest as it is the public that participate in trials, and the safety and rights of research participants should be weighted higher than commercial interests. Indeed, commercial interests are likely not at stake as trial protocols rarely contain commercially sensitive information.21
We also propose a requirement to conduct a systematic literature search prior to applications for ethical approval. In 2005, The Lancet made it a requirement for authors of clinical trials to include a clear summary of previous research and explain how their trial results affect the summary.22 In 2010, the executive editor and editor-in-chief of the Lancet commented on the disappointing implementation of this policy and made the policy more specific by requiring that authors either conduct their own systematic review or cite a recent systematic review and put their own trial results into this context.23 Almost no other journals have a similar policy and the responsibility to safeguard the rights of participants should be shared with ethics committees.
Lastly, we recommend that ethics committees formally endorse and apply the SPIRIT statement4 ,10 which is an evidence-based set of items that should be addressed in a protocol. We believe all protocols for randomised trials submitted to ethics committees should follow the format and report on the items presented in the SPIRIT statement.
References
Footnotes
Contributors PCG came up with the idea for the study and wrote the protocol. MCO, MM and ASPM collected data. MCO, ASPM and JV analysed the data. ASPM, JV, KJJ and PCG interpreted the data. ASPM wrote the first draft manuscript. All authors read and commented on the final manuscript.
Funding The study was funded by the Laura and John Arnold Foundation and the Nordic Cochrane Centre.
Competing interests None declared.
Patient consent for publication Not required.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement No data are available.