Article Text
PDF
Abstract
Objective The Swiss National Science Foundation (SNSF) promotes academic excellence through competitive selection of study proposals and rigorous evaluation of feasibility, but completion status and publication history of SNSF-supported randomised clinical trials (RCTs) remain unclear. The main objectives were to review all healthcare RCTs supported by the SNSF for trial discontinuation and non-publication, to investigate potential risk factors for trial discontinuation due to poor recruitment and non-publication, and to compare findings to other Swiss RCTs not supported by the SNSF.
Design We established a retrospective cohort of all SNSF-supported RCTs for which recruitment and funding had ended in 2015 or earlier. For each RCT, two investigators independently searched corresponding publications in electronic databases. In addition, we approached all principal investigators to ask for additional publications and information about trial discontinuation. Teams of two investigators independently extracted details about study design, recruitment of participants, outcomes, analysis and sample size from the original proposal and, if available, from trial registries and publications. We used multivariable regression analysis to explore potential risk factors associated with discontinuation due to poor recruitment and with non-publication, and to compare our results with data from a previous cohort of Swiss RCTs not supported by the SNSF.
Results We included 101 RCTs supported by the SNSF between 1986 and 2015. Eighty-seven (86%) principal investigators responded to our survey. Overall, 69 (68%) RCTs were completed, 26 (26%) RCTs were prematurely discontinued (all due to slow recruitment) and the completion status remained unclear for 6 (6%) RCTs. For analysing publication status, we excluded 4 RCTs for which follow-up was still ongoing and 9 for which manuscripts were still in preparation. Of the remaining 88 RCTs, 53 (60%) were published as full articles in peer-reviewed journals. Multivariable regression models suggested that discontinued trials were at higher risk for non-publication than completed trials (adjusted OR 7.61; 95% CI 2.44 to 27.09). Compared with other Swiss RCTs, the risk of discontinuation for SNSF-supported RCTs was higher than in industry-initiated RCTs (adjusted OR 3.84; 95% CI 1.68 to 8.74), but not significantly different from investigator-initiated RCTs not supported by the SNSF (adjusted OR 1.05; 95% CI 0.51 to 2.11). We found no evidence that the proportion of discontinued or unpublished RCTs decreased over the last 20 years.
Conclusions One out of four SNSF-supported RCTs were prematurely discontinued due to slow recruitment, 40% of all included RCTs and 70% of all discontinued RCTs were not published in peer-reviewed journals. There is a case to reconsider how public funding bodies such as the SNSF could improve their feasibility assessment and promote publication of RCTs irrespective of completion status.
- randomized controlled trial
- early termination of clinical trials
- non-publication
- swiss national science foundation
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
- randomized controlled trial
- early termination of clinical trials
- non-publication
- swiss national science foundation
Strengths and limitations of this study
Access to all healthcare randomised clinical trial (RCT) proposals supported by the Swiss National Science Foundation (SNSF) since its existence without any restrictions by the funder or the applicants themselves.
The search for subsequent publications and the data extraction were performed by methodologically trained reviewers independently and in duplicate.
We combined information derived from funding proposals, trial registries, publications and an investigator survey (response rate of 86%).
Robustness due to various sensitivity analyses using different outcome definitions and multiple imputation for missing data.
Low reporting quality of some of the older RCT proposals leading to missing data.
Low power for our prespecified risk-factor analyses due to the rather small number of SNSF-supported healthcare RCTs over a time period of almost three decades.
Focus on SNSF-supported RCTs means that the results are not necessarily generalisable to publicly funded RCTs outside of Switzerland.
Introduction
Randomised clinical trials (RCTs) are the gold standard to assess preventive, therapeutic, diagnostic or rehabilitative interventions and constitute a cornerstone of evidence-based medicine.1 The conduct of an RCT usually requires substantial resources and investigators often face multiple practical challenges. A study of trial protocols approved by research ethics committees between 2000 and 2003 suggested that 25% of initiated RCTs were prematurely discontinued and up to 60% remained unpublished. The most frequent reason for discontinuation was poor recruitment of trial participants,2 which has ethical and economic implications: patients consent to participate in a trial on the premise that the findings will foster new medical knowledge. However, if recruitment fails, results are inconclusive, and collected data are not published, which undermines patients’ trust in clinical research. Moreover, non-publication of (discontinued) RCTs compromises systematic reviews and meta-analyses that help answer clinically important research questions. Finally, discontinuation and non-publication represent a waste of precious (public) funding resources.3 4
The Swiss National Science Foundation (SNSF) is the main public funding body for basic and applied research on healthcare in Switzerland. The SNSF promotes academic excellence and has high expectations in beneficiaries about ethical standards and research integrity. During peer-review of proposals, SNSF committees critically evaluate the feasibility of proposed trials among other criteria. In addition, the SNSF explicitly states that grantees are obliged to make research results available to the public.5
The risk of trial discontinuation and non-publication in SNSF-supported RCTs is unknown. Our earlier research included nine SNSF-supported RCTs.2 All were successfully completed suggesting that SNSF-supported RCTs are at lower risk for discontinuation than other RCTs.2 However, a study analysing 122 RCTs funded by two major public research funding bodies in the UK found that u45% of RCTs were discontinued due to poor recruitment.6
We aimed to investigate the completion and publication status of all RCTs supported by the SNSF. We tested a small number of prespecified risk factors for trial discontinuation and non-publication. In addition, we conducted an exploratory analysis comparing the SNSF-supported RCTs with other RCTs that were conducted in Switzerland but not supported by the SNSF.
Methods
Study design and data source
We conducted a retrospective cohort study of RCTs. We systematically searched titles and abstracts provided on the project database of the SNSF (p3 research database, p3.snf.ch), which contains all research projects that the SNSF approved and supported fully or partially since its existence. We excluded RCTs supported through personal grants that were carried out exclusively outside of Switzerland (ie, SNSF mobility grants), RCTs that were never started and RCTs for which recruitment or funding was still ongoing at the time of the cut-off date (30 April 2015). The search strategy included synonyms for ‘randomisation’ and ‘trial’ and was limited to human healthcare categories using the database's ‘advanced search’ option (for details see online supplementary appendix 1).
Supplementary file 1
For each eligible RCT, two methodologically trained investigators independently searched for corresponding publications through information available in the p3 research database and electronic literature searches in Medline, Embase, Cochrane Central Register of Controlled Trials, Google Scholar and trial registries, using relevant keywords for study design, population, interventions and outcome if applicable. Furthermore, we screened personal websites of principal investigators and coinvestigators, and reference lists of related articles.
In addition, we approached all principal investigators of included RCTs using an online questionnaire sent by email. We asked them to provide a) a copy of the latest RCT protocol, b) all publication(s), c) recruitment details such as timeframe, duration and achieved sample size, d) recruitment problems with reasons and any measures taken in response, e) whether a trial was prematurely discontinued with reasons and f) additional funding sources and estimate overall cost of their trial. If a principal investigator did not respond, we sent several reminders by email, or a paper version of the questionnaire by regular mail, and eventually tried to call the investigator by phone or approach a coinvestigator.
Data extraction
Data extractors trained in trial methodology signed confidentiality agreements and extracted data from trial proposals and corresponding publications independently and in duplicate. Disagreements were resolved by discussion and consensus. Extracted items included RCT properties (eg, setting and clinical area, study design, primary outcome, type of intervention and control group, single-centre/multicentre status, planned sample size, planned respectively performed interim analysis), funding sources, statistical methods (eg, sample size calculation, intention-to-treat principle, subgroup analysis) and recruitment strategies (recruitment projections, availability of logistic/methodological support, strategies to support/monitor recruitment) and premature discontinuation and reported reasons (in publications only). We used a password-protected web-based data extraction tool (www.squiekero.org) that included a detailed manual with instructions for each variable of interest.
Outcome definitions
We considered an RCT discontinued if this was explicitly reported by the investigators in a journal publication or in their response to our survey. If we could not elucidate the reason for trial discontinuation, we considered an RCT discontinued for poor recruitment if poor recruitment was mentioned (in the publication or survey) and the actual sample size was less than the prespecified 90% of the target sample size (sensitivity analysis with cut-off at 80%).
For analysing publication status, we excluded RCTs for which follow-up was still ongoing, or manuscripts were still in preparation (according to survey responses). We considered peer-reviewed journal publications other than conference abstracts, research letters or book chapters as full publications in our primary analysis.
Statistical analysis
We described categorical variables as absolute and relative frequencies and continuous variables as medians and IQRs. We prespecified two multivariable logistic regression models to test the following prespecified risk factors:
For RCT discontinuation (discontinued vs completed, unclear excluded): centre status (single vs multicentre; hypothesis: multicentre RCTs are less frequently discontinued), target sample size (continuous; hypothesis: larger trials are less frequently discontinued2) and start year of the RCT (continuous; hypothesis: older RCTs are more likely to be discontinued). We used complete cases in the full dataset.
For non-publication (non-publication vs peer-reviewed journal publication): RCT completion status (discontinued vs completed; hypothesis: discontinued RCTs are less likely to be published), target sample size (continuous; hypothesis: smaller RCTs are less likely to be published) and start year of the RCT (continuous; hypothesis: older RCTs are less likely to be published). Here we used a reduced dataset excluding four RCTs for which follow-up was still ongoing and nine RCTs for which manuscripts were still in preparation.
We calculated unadjusted and adjusted ORs with 95% CIs. In two separate sensitivity analysis, we (a) used multiple-imputation techniques to impute missing data and (b) an alternate threshold of <80% of the target sample size to define RCT discontinuation (see the ‘Outcome definitions’ section). In the logistic regression model for non-publication, we performed two additional sensitivity analyses: first, we used an alternative definition of ‘published’ adding RCTs with results published in conference abstracts, book chapters or letters to the editor. Second, we excluded RCTs approved by the SNSF between 1986 and 1996, that is, the four oldest SNSF-supported RCTs, that all remained unpublished.
In further exploratory analyses, we compared RCT discontinuation and non-publication between SNSF-supported RCTs and two previously established samples of (a) investigator-initiated RCTs without SNSF support and (b) industry-initiated RCTs conducted in Switzerland (online supplementary figure 1). Both these samples were subsets of a large cohort of RCTs approved by research ethics committees between 2000 and 2003.2 To enhance comparability, we prespecifed a similar timeframe and only included SNSF-supported RCTs with start dates between 1995 and 2008. We fitted multivariable logistic regression models using SNSF support as the independent variable of interest. With RCT discontinuation as dependent variable, we adjusted models for centre status and sample size, and with non-publication as dependent variable we adjusted for discontinuation, centre status and sample size.
A two-sided p value of <0.05 was set as level of statistical significance. All analyses were carried out using R V.3.0.3 (www.r-project.org).
Findings
Trial flow and characteristics of included trials
The search of SNSF's p3 database yielded 455 entries, of which 354 were excluded after screening titles and abstracts (figure 1, last update in November 2015 for the cut-off date of 30 April 2015). We identified a total of 101 eligible healthcare RCTs funded by the SNSF between 1986 and 2015 (since the existence of the p3 database). The response rate in our survey of principal investigators was 86%; 87 of 101 responded to the two main questions: ‘Was the RCT stopped prematurely?’ and ‘Were the results of the RCT published?'. For the assessment of publication status, we excluded 13 RCTs for which follow-up was still ongoing or the manuscript still in preparation, leaving a total of 88 RCTs. Table 1 summarises the characteristics of the included RCTs. The majority were single-centre, parallel-group trials testing a medication or behavioural intervention in adults. The median target sample size was 120 and the median planned duration of recruitment was 14 months. Of the 68 trials approved after the ICMJE recommendation to prospectively register all RCTs (ie, in 2005 or later),3 51 (76.5%) were registered in a trial registry.
Flow chart of trial selection. RCT, randomised clinical trial; SNSF, Swiss National Science Foundation.
Characteristics of included RCTs supported by the SNSF
Discontinuation of SNSF-supported trials
Overall, 68% (69/101) RCTs were completed as planned, 26% (26/101) were prematurely discontinued and the completion status remained unclear in 6% (6/101) (table 2). A reason for discontinuation was explicitly reported for 42% (11/26) of the discontinued RCTs (one was reported in a registry, one in a publication and nine by the investigators in the survey). In all 11 cases, the RCT was discontinued due to recruitment problems. For the remaining 58% (15/26), we were unable to obtain any clarification of the completion status from the investigators. All mentioned slow recruitment when responding in the survey, and the trials achieved <90% of the initial target sample size. Therefore, we considered all 26 RCTs prematurely discontinued due to slow recruitment. Of the completed RCTs, 61 of 69 completed RCTs were carried out as planned. Five of the remaining eight completed RCTs were modified at an early stage in terms of the research question or planned sample size; all achieved the new target sample size. The remaining three completed RCTs achieved a conclusive result with a smaller sample size than anticipated.
Completion status of SNSF-supported RCTs
In the sensitivity analysis, when we used a threshold of 80% instead of 90% of target sample size achieved to define RCT discontinuation (see the ‘Outcome definitions’ section), 24% (24/101) of RCTs were discontinued (online supplementary table 1).
Fifty-seven per cent (50/87) of the principal investigators indicated in the survey that recruitment was slower than expected, irrespective of the completion status of their RCT.
Risk factors for discontinuation
In the univariable and multivariable analysis of risk factors for discontinuation, none of the prespecified variables (sample size, centre status and year of RCT) were significantly associated with trial discontinuation (table 3). Sensitivity analyses with multiple imputation for missing data or with the alternate threshold of 80% of achieved target sample size did not change our results (see online supplementary tables 2 and 3.
Factors associated with discontinuation of SNSF-supported RCTs
Publication of SNSF-supported trials
Of 88 RCTs, 60% (53/88) were published as peer-reviewed journal articles 1– 13 years after their start, 22% (19/88) were not published at all and the remaining 18% (16/88) RCTs only as conference abstracts, book chapters or brief reports/letters to the editor (table 4). From the discontinued RCTs, 30% (6/20) were published as peer-reviewed journal articles. We identified 52 of the 53 full journal articles through searching electronic databases and one, a publication in an Indian journal not indexed in common databases, through our survey.
Publication status of SNSF-supported RCTs stratified by completion status
Risk factors for non-publication
In multivariable analysis, trial discontinuation was significantly associated with non-publication (adjusted OR 7.61; 95% CI 2.44 to 27.09), but target sample size and start year were not (table 5). A univariable association between start year and non-publication (unadjusted OR 0.90; 95% CI, 0.81 to 0.98) did not persist when adjusting for discontinuation and target sample size. In addition, when we excluded the four oldest RCTs approved between 1986 and 1996 (all unpublished), this association in univariable analysis was no longer statistically significant (see online supplementary table 4). Sensitivity analyses with multiple imputation for missing data, with the alternate threshold of 80% of achieved target sample size, or by considering conference abstracts, letters to the editor and book chapters as ‘full publications’ did not change our findings (see online supplementary tables 5, 6 and 7).
Factors associated with non-publication of SNSF-supported RCTs
Comparison with RCTs conducted in Switzerland but without SNSF support
Online supplementary figure 1 and table 8 provide details about the comparison of investigator-initiated SNSF-supported RCTs with (a) investigator-initiated RCTs without SNSF support and (b) industry-initiated RCTs conducted in Switzerland. Multivariable analyses (table 6) suggested that the odds of premature trial discontinuation did not differ between SNSF-supported RCTs and investigator-initiated RCTs conducted in Switzerland but not supported by the SNSF (adjusted OR 1.05; 95% CI 0.51 to 2.11); however, the odds of premature discontinuation were significantly higher than in industry-initiated RCTs conducted in Switzerland (adjusted OR 3.84; 95% CI 1.68 to 8.74). The odds of non-publication did not differ significantly in both comparisons. Sensitivity analyses with multiple imputation for missing data did not change these findings (see online supplementary table 9).
Comparison with RCTs conducted in Switzerland but not supported by SNSF
Discussion
Summary of findings
Recruitment problems caused premature discontinuation in one out of four RCTs supported by the SNSF. Forty per cent of RCTs (70% of discontinued RCTs) were not published in a peer-reviewed journal. We found no evidence that the proportion of discontinued or unpublished RCTs decreased over the last 20 years. Trial discontinuation was an important risk factor for non-publication. SNSF-supported RCTs were more likely to be discontinued due to poor recruitment than industry-initiated RCTs and about equally likely to be discontinued or not published than investigator-initiated RCTs without SNSF support.
Strengths and limitations
We had access to all healthcare RCT proposals supported by the SNSF without any restrictions by the funder or the applicants themselves.7 The search for subsequent publications and the data extraction were performed by methodologically trained reviewers independently and in duplicate. We combined information derived from funding proposals, trial registries, publications and an investigator survey. In the latter, we achieved a response rate of 86%, which was higher than in other similar studies surveying biomedical researchers.8 9 Our results proved robust in various sensitivity analyses using different outcome definitions and multiple imputation for missing data.
Limitations of our study include the low reporting quality of some of the older RCT proposals leading to missing data. Furthermore, we had low power for our prespecified risk-factor analyses due to the rather small number of SNSF-supported healthcare RCTs over a time period of almost three decades. Our focus on SNSF-supported RCTs means that the results are not necessarily generalisable to publicly funded RCTs outside of Switzerland. However, evaluation criteria and processes of the SNSF are similar to other national funding agencies in Europe or North-America based on information from agencies’ websites. Finally, we could not include some well-established factors associated with non-publication of RCTs in our regression analysis such as negative results because we did not ask investigators of unpublished RCTs for direction of results of their trials.8 10
Comparison with other studies
In a study in the UK, McDonald et al followed 122 multicentre investigator-initiated RCTs funded by two major public funding bodies between 1994 and 2002 and showed that 20%–45% of the trials were discontinued due to insufficient recruitment (defined as 80% or less of original target).6 The proportion of MRC/HTA-supported RCTs failing to recruit 80% of their target decreased in the period between 2002 to 2008, but still accounted for 22% of included RCTs. As in our study the proportion of completed RCTs did not significantly improve over time.11 Walters et al. reviewed 151 HTA-supported RCTs published between 2004 – 2014 and found that 21% of the included RCTs recruited less than 80% of their final target sample size.12 Another retrospective cohort study from Switzerland reported a lower rate of trial discontinuation (57/508, 11%).13 Likely explanations for this difference are a larger proportion of industry-initiated RCTs and a restriction to drug intervention trials, both of which are factors potentially protecting against discontinuation.2 Our study suggests that recruitment problems seem to be equally common in RCTs with and without competitive public funding, and confirms that recruitment problems are the most important reason for trial discontinuation.2 6 14
Overall, 60% of SNSF-supported RCTs were published as peer-reviewed journal articles, which is in line with publication rates found in other empirical studies on RCTs.13 15 The publication rate of 30% among discontinued SNSF-supported RCTs is similar to a comparable Dutch study14 with 32% of discontinued RCTs being published. Similar to the results of Kasenda et al, 2 our analysis suggests that RCT discontinuation (due to slow recruitment) is a major determinant of non-publication.
Implications
Premature discontinuation of RCTs due to poor recruitment constitutes a considerable waste of scarce public resources for clinical research, in particular when data and lessons learnt remain unpublished. If the challenges resulting in RCT discontinuation are not shared with the scientific community, mistakes may be repeated. Results from discontinued and therefore underpowered RCTs might be inconclusive. However, their results are still valid, can provide important pilot data for future trials and may contribute to systematic reviews and meta-analyses. Trial discontinuation and failure to publish the results puts the public's and patients’ trust in clinical research at stake and compromises the willingness of future patients to participate in clinical studies.
In line with earlier findings,6 our results question whether the assessment of an RCT's feasibility by public funding agencies meets its goals. No doubt, such an evaluation is challenging since grant applicants are inherently optimistic about the feasibility of their trial and projected numbers of eligible and recruited participants. How could the feasibility assessment of proposed RCTs be improved? First, public funding agencies, and institutional review boards and ethics committees, should require investigators to marshal empirical evidence to support the feasibility of achieving sample size targets within acceptable recruitment periods: published or at least registered pilot trials that also include obtaining informed consent by participants probably provide the best proof of feasibility of a trial protocol and yield the most realistic estimates for participant recruitment.16 In the present study, only 1 RCT proposal was based on a pilot trial and only 10 described plans for a pilot. To promote the conduct of pilot trials, public funders should include a specific funding scheme for feasibility trials in their portfolio. In our own experience, reviewer comment received after submission of grant proposals often include the well-meaning suggestion to pilot trial methods while funding for such feasibility studies is difficult to obtain. Second, trial protocols should describe the planned recruitment process in sufficient detail to enable referees to judge about feasibility based on clear and comprehensive information. Endorsement of the Standard Protocol Items: Recommendations for Interventional Trials guidance and respective templates may help to facilitate well-structured and informative trial protocols as a prerequisite for improved feasibility assessment.17 18 Third, funding agencies could stipulate stringent recruitment monitoring, regular feedback about recruitment progress with updated recruitment projections particularly in the first months of the trial and plans for specific actions to be taken if projection goals are not achieved. ‘Pull funding’ schemes could be introduced that condition the payment of grant instalments on the achievement of certain recruitment milestones. Providing ‘backup grants’ is another option, if a well-defined limited extension of the recruitment period is needed to allow investigators to complete the RCT. Fourth, current recommendations encourage the adoption of multiple recruitment strategies to enhance participant enrolment in RCTs,4 19 20 but evidence on what actually works to improve recruitment is sparse and high-quality studies on promising interventions are urgently needed.21–23 An appropriate and efficient way to test such interventions may be to incorporate them with a randomised design in a host trial as recently promoted by the Systematic Techniques for Assisting Recruitment to Trials programme.24 So-called ‘Studies Within A Trial’ could be encouraged by funding agencies and would add extra value to grant proposals for RCTs.
Public funding agencies have a genuine interest to ensure that results, data and lessons learnt from funded RCTs are made available to the scientific community and that ultimately the public benefits from it. This can be achieved through peer-reviewed publications or accessible data depositories or trial registries and supporting initiatives such as AllTrials (www.alltrials.net). Academic institutions (eg, universities) hosting clinical trials should mandate that investigators will publish all results in full, for example, within 2 years of the last data collection of the primary end point.25 Public funding agencies should help endorse such policies by conditioning transfer of funds to investigators on adherence to standards of publication and data sharing.
Conclusion
Trial discontinuation due to poor recruitment and non-publication are common among SNSF-supported RCTs. More rigorous measures are necessary to ensure that successful grant applications have the best possible prospects for trial completion and that results and encountered challenges are reliably disseminated to the scientific community and larger public.
References
Footnotes
Contributors 1a) Study conception and design: AA, SS, MB. 1b) Acquisition of data: all authors. 1c) Analysis and interpretation of data: AA, SS, MB. 2a) Drafting of manuscript: AA, SS, MB. 2b) Critical revision of manuscript: all authors. 3) Final approval of the version to be submitted: all authors.
Competing interests All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: The study was funded by the Swiss National Science Foundation (SNSF) and the SNSF provided original proposals of SNSF-supported randomized clinical trials, but the SNSF had no role in the design of the study, data extraction, data analysis, interpretation of the results and writing the manuscript. No other financial relationships with any other organisations that might have an interest in the submitted work in the previous three years and no other relationships or activities that could appear to have influenced the submitted work to declare. There are no conflicts of interest to declare.
Ethics approval Similar to study protocols submitted to research ethics committees, those submitted to the SNSF are considered confidential. Therefore, we consulted the legal department of the SNSF, which confirmed that our study was in line with the Cantonal data protection laws and explicitly stated that no ethical approval was necessary. We included only proposals for which the SNSF support period had ended and we maintained the confidentiality of the accessed files following a framework that has earlier been proposed for this type of research.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement Full dataset and technical appendix and statistical code are available from the corresponding author.