Article Text
PDF
Abstract
Objectives To evaluate the adequacy of reporting of protocols for randomised trials on diseases of the digestive system registered in http://ClinicalTrials.gov and the consistency between primary outcomes, secondary outcomes and sample size specified in http://ClinicalTrials.gov and published trials.
Methods Randomised phase III trials on adult patients with gastrointestinal diseases registered before January 2009 in http://ClinicalTrials.gov were eligible for inclusion. From http://ClinicalTrials.gov all data elements in the database required by the International Committee of Medical Journal Editors (ICMJE) member journals were extracted. The subsequent publications for registered trials were identified. For published trials, data concerning publication date, primary and secondary endpoint, sample size, and whether the journal adhered to ICMJE principles were extracted. Differences between primary and secondary outcomes, sample size and sample size calculations data in http://ClinicalTrials.gov and in the published paper were registered.
Results 105 trials were evaluated. 66 trials (63%) were published. 30% of trials were registered incorrectly after their completion date. Several data elements of the required ICMJE data list were not filled in, with missing data in 22% and 11%, respectively, of cases concerning the primary outcome measure and sample size. In 26% of the published papers, data on sample size calculations were missing and discrepancies between sample size reporting in http://ClinicalTrials.gov and published trials existed.
Conclusion The quality of registration of randomised controlled trials still needs improvement.
This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/2.0/ and http://creativecommons.org/licenses/by-nc/2.0/legalcode.
Statistics from Altmetric.com
Article summary
Article focus
Outcome reporting bias is a considerable problem.
A number of journals (International Committee of Medical Journal Editors journals) only publish clinical trials that are registered in relevant trial databases such as http://ClinicalTrials.gov before recruitment of participants. Older trials commenced after 1 July 2005 will be considered for publication only if they are adequately registered before journal submission.
Previous studies of published trials suggest that many are registered inadequately.
Key messages
A number of trials are registered inadequately in http://ClinicalTrials.gov without information about basic methodological issues.
Several trials published in journals that require registration in online databases are registered after their date of completion.
Discrepancies between the registered information in trial registrations and the trial publications still exist (such as the planned sample size calculations).
Strength and limitations of this study
The study is small, only evaluating 105 trials. The real extent of inadequate trial registration may be under- or overestimated.
Only trials concerning gastrointestinal diseases were evaluated, which makes it difficult to generalise to other medical specialties.
Introduction
Since 2005, the International Committee of Medical Journal Editors (ICMJE) has initiated a policy requiring investigators to register clinical trials on healthcare interventions in a public trial registry as a condition for consideration for publication.1 http://ClinicalTrials.gov and similar clinical trial registries available through the WHO International Clinical Trials Portal are classed as acceptable. Trials must register at or before the onset of patient enrolment. The registration should include information about the trial design. The registration policy is applicable to any trial starting enrolment after 1 July 2005. For trials that began enrolment prior to this date, the ICMJE member journals originally required registration by 13 September 2005.1 However, beginning on 13 September 2005, ICMJE journals will consider such trials only if they are adequately registered before journal submission.2 Trials are only eligible for publication in ICMJE journals if registered correctly.
The purpose of trial registries is to reduce the risk of dissemination and reporting bias and to ensure that clinicians, researchers and patients can find key information about every clinical trial whose principal aim is to shape medical decision-making. Overviews of published trials suggest that the registration of several trials is inadequate.3–11 Likewise, the quality and completeness of trial registration in specific trial registries have been found inadequate.5 11 Inadequate registration will not reduce the risk of bias, but will give the reader and journals a false impression of adequate bias control.
The objective of this study was to evaluate the adequacy of reporting of randomised trials on diseases of the digestive system registered in http://ClinicalTrials.gov and to evaluate the consistency between primary outcomes, secondary outcomes and sample size specified in trial protocols and published trials.
Methods
The present work is based on a written protocol, which is included as a supplementary file to the paper at BMJ Open.
Identification of eligible trials
To obtain a homogenous sample, and as all authors have a special interest in gastroenterology and hepatology, we focused on phase III trials on adult patients with diseases in the digestive system. To achieve a larger proportion of trials with subsequent publication, trials registered after January 2009 were not included. Accordingly any randomised trial on adult patients with gastrointestinal diseases registered before January 2009 in http://ClinicalTrials.gov was eligible for inclusion.
Eligible trials were identified through electronic searches in http://ClinicalTrials.gov using the search strategy: closed studies | interventional studies | digestive system disease | adult senior | phase III | updated on or before 1 January 2009. Subsequent publications of clinical trials were identified through electronic searches in PubMed, Medline, Embase, Science Citation Index and the Cochrane Library using investigator names and keywords.
Data extraction
The adequacy of reporting of protocols was assessed through data extracted from the http://ClinicalTrials.gov database and their subsequent publications. Initially, all trials identified through the electronic search in http://ClinicalTrials.gov were listed and two authors evaluated whether the trials fulfilled the inclusion criteria. Excluded trials were listed with the reason for exclusion. The authors independently extracted data from included trials based on pilot tested data extraction forms. Disagreements between authors were resolved through discussion before analysis.
Data extracted from http://ClinicalTrials.gov included all data elements in the database required by the ICMJE member journals (table 2). For each trial, the start date, completion date and registration date in http://ClinicalTrials.gov were recorded. Data extracted from the published articles included publication date, journal name and whether the journal adhered to ICMJE principles. Primary and secondary outcome measures were extracted; any differences between primary and secondary outcomes specified in http://ClinicalTrials.gov and those defined in the published articles were recorded, as was whether changes were reported. Finally data regarding sample size and sample size calculation were extracted, and potential differences between the planned sample size and number of randomised patients were recorded, as was whether discrepancies between sample size data in http://ClinicalTrials.gov and in the published paper were present.
Statistical analysis
Statistical analysis was performed using STATA V.10.0 for Windows. Characteristics of included trials were summarised as frequencies or medians with ranges. The relation between key trial characteristics and whether trials were published was assessed based on multiple logistic regression analysis with results presented as ORs with 95% CIs and p values. All tests were two tailed and p values <0.05 were considered significant.
Post hoc analysis
The proportion of published studies that identified positive versus negative results was examined. We defined authors' conclusions as the reported interpretation of the extent to which the overall trial result favoured the experimental over the control intervention. We graded authors' conclusions according to the phrasing in the abstract and the summarised conclusion on a previously validated six-point scale12 13 (box 1). Higher scores indicate a more positive conclusion towards the experimental intervention: scores of 1–3 favoured the control and scores of 4–6 favoured the experimental intervention.
Equipoise scale
Experimental intervention highly preferred and should now be considered the standard intervention in all patients or similar statement (6 points).
Experimental intervention preferred to standard but further trials still indicated; may be more costly or similar disclaimer (5 points).
Experimental and control intervention about equal but experimental intervention successful because of minor advantage (4 points).
Experimental and control intervention about equal, but experimental intervention disappointing as control intervention had some minor advantage (3 points).
Control intervention preferred to experimental intervention but experimental intervention might be promising under some circumstances or similar (2 points).
Control intervention highly preferred and is best alternative; should be considered the standard intervention in all patients or similar (1 point).
Results
Initially, we retrieved 150 references through our electronic search. After excluding trials that turned out to be observational, trials that were not initiated (because of lack of funding or for logistic reasons), and trials that turned out to be safety studies in healthy participants, we identified 105 trials that fulfilled our inclusion criteria. The description of the included trials is presented in table 1. The majority of trials assessed interventions for malignant diseases, inflammatory bowel disease or liver diseases. Most trials investigated drugs.
Characteristics of 105 trials registered in http://ClinicalTrials.gov
The included trials were registered in http://ClinicalTrials.gov during 1998 to 2008 (median 2005). Ninety-nine of the 105 trials provided information regarding the date the trial was initiated (table 2). Based on the registered data, the trials were initiated during 1978 to 2006 (median 2002). Seventy-six of the 99 trials were initiated before 2005. Only 73 trials reported the date the trial was completed. The date of completion ranged from 1996 to 2009 (median 2006). Fifty-eight (55%) trials were registered correctly according to ICMJE criteria (ie, before or at the time of initiation for trials conducted after 1 July 2005, and before 13 September 2005 for trials starting before 1 July 2005). Thirty-one (30%) trials were registered after the trial was completed.
Data extracted from 105 trials in http://ClinicalTrials.gov
Of the 105 included trials, 23 (22%) did not describe the primary outcome measures and 24 (32%) did not describe the secondary outcome measures (table 2). In two trials one of the initially registered outcome measures was changed to a secondary outcome measure (NCT00204750) and (NCT00606619). Secondary outcomes were changed in three trials (some outcomes were omitted and new outcomes introduced). However, for several trials changes were difficult to classify due to the wording and non-specific definitions.
All 105 trials reported the study type, and provided a brief title, design, condition, intervention, recruitment, eligibility criteria and contacts. Twelve trials (11%) did not report the planned number of patients enrolled in the trial (table 2).
We identified published reports for 66 trials (63%), in 28 different journals. The trials were published during 1980 to 2011 (median 2008). Twelve of the 31 trials registered after the completion date in http://ClinicalTrials.gov were published after 2005 in journals proclaiming to adhere to ICMJE principles (data not shown). No changes were identified in the definitions of the primary outcome measure between registrations in http://ClinicalTrials.gov and the published reports (table 3). Changes in the secondary outcomes were identified for six trials and primarily included introduction of additional outcomes (eg, compliance) or omission of outcomes (eg, quality of life). Of the 66 trials that were published, 49 (74%) described sample size calculations in the published report. Based on the published trial reports, nine trials did not reach the planned sample size due to unexpectedly low recruitment rates, or because the trial was terminated early after promising or disappointing interim analyses. For seven trials, discrepancies between the planned sample size registered in http://ClinicalTrials.gov and the number of patients randomised based on the published reports were identified without any apparent explanation. In four cases the number of patients in the preset sample size and/or the number of patients randomised were less in the published report than the sample size specified in the trial registry. In three cases the preset sample size and/or randomisation number were higher in the published reports than specified in the trial registry. The number of participants in the included trials ranged from 16 to 1135 (median 305). The difference ranged from 545 participants fewer than planned to 138 participants more than planned.
Characteristics of 66 published trials from a sample of 105 trials registered in http://ClinicalTrials.gov—concerning changes in outcome measures and sample size
The planned sample size provided in http://ClinicalTrials.gov ranged from 36 to 1500 participants (median 220) in trials that were published and from 30 to 660 (median 60) for trials that were not published. There was no clear association between the reported planned sample size reported in http://ClinicalTrials.gov and the chance that the trial was published (OR 1.01; 95% CI 1.00 to 1.01). There was no apparent association between publication and registrations in http://ClinicalTrials.gov of the primary outcome measure (OR 0.80; 95% CI 0.29 to 2.17) or the secondary outcome measure (OR 0.58; 95% CI 0.224 to 1.44).
In most published trials authors' conclusions favoured the experimental intervention (figure 1).
Equipoise scale, authors' scores for conclusions in 66 published trials.
Discussion
The results of this study suggest that the reporting of data in the public trial registry http://ClinicalTrials.gov is inadequate. First, 30% of the trials analysed were registered after the completion date. Second, several data of the required ICMJE data list were not filled in, with missing data in 22% and 11%, respectively, of cases concerning the important issues primary outcome measures and sample size. Third, only 63% of the analysed trials registered were published. Several articles (26%) lacked sample size calculations and there were discrepancies between sample size reporting in http://ClinicalTrials.gov and published trials.
One of the purposes of reporting protocols in a clinical trial registry is to improve the quality of reporting of biomedical research. The policy of registration of protocols at the start or before termination of the study is to minimise the likelihood of authors introducing bias into their papers by making major changes that otherwise might remain undisclosed and undetectable.14 We were surprised to see that about a third of trials examined in our study were registered after the completion date; this kind of registration seems inappropriate or irrelevant and is a challenge to the credibility of the registers. Registration after the study is complete gives the reader a false sense of security since the published information in the journal only consists of the trial registration number. In one study (NCT00766805) that was registered post hoc both number of patients and number of events differed from a previously published abstract. These changes in the numbers and outcomes altered the conclusions of a meta-analysis on the subject.15 Such changes clearly hamper the validity of trials, however it is not possible to trace and document if the trials are not registered correctly. Since the requirements regarding trial registration have been established for several years, investigators have had time to get acquainted with the procedures. In most submission procedures for randomised clinical trials, journals ask authors to provide their registration number but not the date. Providing the date of registration and the start and completion dates for the trial would provide a better overview than just the registration number.
Bias in trial registry has previously been demonstrated. Zarin et al and Ross et al found that the primary outcome measure field in http://ClinicalTrials.gov was completed in 66–89% of cases.5 11 These data correspond to our findings, which show that 78% of trials provided information regarding their primary outcome measure. However, this also means that several trials still lack crucial information regarding basic components. Likewise it has previously been found that changes in sample size exist between protocols and articles, and that statistically significant outcomes were favoured in being fully reported.3 16 In this study we did not examined whether outcome-reporting bias favoured significant primary outcomes, but we did find that studies in favour of the experimental intervention were published more frequently than studies in favour of the control intervention. On the positive side, we did not find any discrepancies between the data in the trial registration and in published papers concerning primary outcome measures. We did, however, find changes concerning secondary outcome measures. These discrepancies suggest that post hoc changes are being made to the trial protocols after the trial is initiated. None of the publications provided any explanations regarding the underlying reasons for the discrepancies.
In the present study, we found disagreements between the sample size calculations reported in the trial registry and the subsequent trial publications in seven cases. Since discrepancies were identified for registrations that were made for trials that were already running, the data suggest that alterations were made post hoc. As changes in sample size might lead to changes in study power, deviation from the planned sample size should be explained in the article. Without this information, we were unable to determine whether the trials were continued beyond the originally intended size after interim analyses, or stopped for some other reason, such as lack of funding, lower than expected recruitment rates or low event rates. Since we did not have information from the authors, we were unable to analyse this finding further.
Our study has its limitations. First, it is a small study only evaluating 105 trials. http://ClinicalTrials.gov has several thousands of trials registered, and consequently the real extent of inadequate trial registration may be under- or overestimated. Second, we only evaluated phase III trials of gastrointestinal diseases, which makes it difficult to generalise to other medical specialties. Third, we only found published articles from 66 of 105 trials. A greater proportion of trials may be or may become published, as we did not contact the primary investigator to double-check publication status; as some studies were completed within the year of this study, publication may be possible in the near future.
In conclusion, adequate registration is supposed to protect against publication bias and to ensure that clinicians, researchers and patients can find key information about clinical trials. Our findings emphasise that inadequate reporting can make the transparency and interpretation of randomised clinical trial results difficult and that timing and quality of registration of randomised controlled trials still needs improvement. Editors should be encouraged to enforce correct registration of trials to be published.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Files in this Data Supplement:
- Data Supplement - Manuscript file of format pdf
Footnotes
Correction notice The “To cite: …” information and running footer in this article have been updated with the correct volume number (volume 1).
To cite: Wildt S, Krag A, Gluud L. Characteristics of randomised trials on diseases in the digestive system registered in ClinicalTrials.gov: a retrospective analysis. BMJ Open 2011;1:e000309. doi:10.1136/bmjopen-2011-000309
Funding This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None.
Contributors All authors contributed to the design, data extraction and interpretation of results. All authors contributed to the writing and final approval of the paper.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement Dataset available at the Dryad repository (doi:10.5061/dryad.s38s0g00).