Article Text
PDF
Abstract
Objectives We sought to evaluate the characteristics and publication fate of improperly registered clinical trials submitted to a medical journal (The BMJ) over a 4-year period to identify common types of registration issues and their relation to publication outcomes.
Design Research articles submitted to The BMJ and identified as unregistered or retrospectively registered by editors were included if they reported outcomes of a clinical trial. Relevant data regarding the trials were then extracted from each paper. Trials were categorised as prospectively registered, registered in an unapproved registry, unregistered or other, and explanations for registration deficiencies were grouped into six categories. We searched PubMed and Google to determine whether, where and when improperly registered studies were subsequently published and whether registration issues were disclosed.
Results 123 research papers reporting apparently unregistered or retrospectively registered clinical trials were identified. 110 studies (89.4%) were retrospectively registered, nine (7.3%) were unregistered, three (2.4%) had been registered in an unapproved registry and one study originally lacking registration details was later discovered to have been prospectively registered. 82 studies (66.6%) were funded entirely or in part by government sources, and only seven studies (5.7%) received funding from industry. Of those papers submitted to The BMJ through the end of 2015, 67 of the 70 papers rejected for registration problems (95.7%) were subsequently published in another journal. The registration problem was disclosed in only 2 (2.9%).
Conclusions Improper registration remains a problem, particularly for clinical trials that are government or foundation-funded. Nonetheless, improperly registered trials are almost always published, suggesting that medical journal editors may not actively enforce registration requirements.
- journalology
- clinical trial registration
- medical publishing
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Strengths and limitations of the study
This study provides the first comprehensive look at registration issues and publication outcomes among research papers submitted to a high-impact medical journal.
This represents the experience of a single high-impact journal and may not be representative of the situation in other, particularly smaller, journals.
We relied on journal editors to report studies with registration problems; some such trials may have been missed and therefore not included in this audit.
Information about the reason for late registration or lack of registration is incomplete because not all authors provided explanations.
Background
Prospective registration of clinical trials is an important safeguard against selective reporting and non-publication of research. Since 2005, the International Committee of Medical Journal Editors (ICMJE) has insisted on prospective trial registration as a condition of publication.1 Registration must occur before enrolment of the first study participant in a trial registry that meets quality criteria developed by WHO. Hundreds of biomedical journals subsequently endorsed this policy.2 Adherence to these requirements is imperfect, however, both on the part of researchers and journal editors. A recent study found that about one-third of contemporary trials in the ClinicalTrials.gov database had been registered more than 3 months after the study began.3 In one survey, 67% of editors from journals that endorse the ICMJE registration policy nonetheless indicated that they would consider retrospectively registered studies for publication.4 Many published trials have registration deficiencies or do not report outcomes that are consistent with registry entries. This problem exists in the subspecialty literature as well as at major journals.5 6
There are many possible reasons for the slow uptake of trial registration. Among them are that in most cases there are no strong penalties for failure to properly register a trial other than the threat of non-publication in an ICMJE-adherent journal. Not all journals are equally committed to enforcing registration requirements and many editors are not persuaded of the need for registration.7 8 The influential Committee on Publication Ethics has advocated a soft line on registration, suggesting that ‘whether or not a trial is registered has little bearing on the quality or ethics of the study, and so it is up to the editor to decide whether or not a study should be published.’9 For the small proportion of trials where financial or other penalties theoretically are in place, enforcement is lax.10
Although the persistence of trial registration deficiencies has been well documented, only one previous study has described the magnitude of the problem in a family of journals.11 Of the 108 clinical trials submitted to BioMed Central journals in 2013, just 33 (31%) had been registered prospectively. The median time between enrolment of the first trial participant and registration was 356 days, with a range of 5–1677 days. This study did not describe the detailed characteristics of the papers with registration problems, or provide information about the publication fate of papers that are rejected from a journal because of such problems. This information is needed to better understand the problem of deficient trial registration and develop strategies to encourage timely registration.
Methods
As part of an internal BMJ audit of adherence to trial registration requirements, we aimed to identify all trials submitted from June 2013 to June 2017 that did not comply with ICMJE trial registration requirements. The BMJ employs four full-time staff and five part-time freelance research editors who appraise and handle submitted articles. In June 2013, EL began keeping a list of all clinical trials submitted to The BMJ that appeared to be unregistered or retrospectively registered. As one of the founding members of the ICMJE, The BMJ since 2005 has required appropriate registration of clinical trials as a condition of publication. All BMJ research editors were asked to notify EL whenever they determined that a submitted trial was unregistered, retrospectively registered (ie, registered after the study began) or registered in an unapproved registry.
All papers on this list were screened to identify those that met the following inclusion criteria: submitted to The BMJ as a research article between 1 June 2013 and 30 June 2017; reported outcomes of a clinical trial according to the ICMJE definition of a trial.12 Papers reporting on observational analyses of clinical trials were excluded. All articles meeting these criteria were entered into an Excel spreadsheet. EL searched the ScholarOne database and extracted the following information for each included paper: full title; unique BMJ paper number; year of submission; country and region of the corresponding author (Europe, North America, Africa/Middle East, Asia, Central and South America—including Mexico or Australia/New Zealand); the trial registry and number, if present; funding sources (categorised as no funding or none when authors specifically declared they had received no funding for the work, unknown funding when authors did not provide funding information, government, academic, industry, foundation or combinations) and any explanation provided by the authors for the registration problem. A second researcher, SC, verified data abstraction for a random sample of 10% of all trial entries. Disagreements were resolved by consensus.
Where a trial registration number could be located in the submitted paper, the corresponding registry entry was searched to verify the date the trial was registered, and if present, the registry-reported date of enrolment of the first participant, date of submission, and anticipated or actual start dates, if present. Additional information about these dates was also sought in the text of the article itself and this information was recorded in the Excel spreadsheet. We then categorised articles as follows:
Prospectively registered, that is, trial registration had occurred in an approved trial registry before the date of first participant enrolment as recorded in the trial registry.
Prospectively or retrospectively registered in a registry that did not meet ICMJE requirements at the time of registration (eg, European Clinical Trials Database is a WHO-approved registry only for trials registered after 20 June 2011).1
Unregistered.
Other (eg, identified by editor as improperly registered but later found to be properly registered).
For registries such as the International Standardised Randomised Controlled Trials Number (ISRCTN) registry, where entries report both the date researchers submitted their request for registration and the date that trial registration was completed, we considered the date registration actually was assigned (not when the application was first submitted) to be the date of registration. For registries that reported only the date of registration submission, we considered that to be the date of registration. To determine the date when a study began, or when participants were enrolled, we used ‘anticipated start date’ for ISRCTN, ‘study start date’ for ClinicalTrials.gov. For the Australian New Zealand Clinical Trials Registry, which reports both an anticipated and actual date of first enrolment, we considered the actual date to be the start date. In cases where the starting date of a trial was reported differently on a trial registry than in the paper, we considered the date reported in the paper itself to be the correct date. In cases where only the month and not the date of registration or first enrolment were reported, we assumed the trial had been registered prospectively.
EL and SL searched PubMed and Google to determine whether, where and when improperly registered studies were subsequently published and, if published, whether the registration problem was disclosed. We first searched Google using the title of the rejected paper; if unsuccessful in locating a publication, we then searched Google using keywords and author names. If unsuccessful, we performed the same searches in PubMed. We searched the full-text versions of subsequently published papers for information about trial registration by first visually scanning the abstract and full text and, if unsuccessful, using the ‘find’ function in Adobe Acrobat Reader or the Google Chrome browser and the search term reg*. We recorded the impact factor of the publishing journal for the year when the study was published, using the Thomson Reuters Journal Citation Reports database.13
We determined the number and proportion of included trials that were unregistered, retrospectively registered or registered in an unapproved registry. We classified the interventions studied in each trial as regulated (eg, drug, biological, medical device or other intervention regulated by the US Food and Drug Administration—FDA) or unregulated (eg, procedures, behavioural interventions, dietary supplements or other interventions that are not subject to FDA regulation). We then evaluated the proportion of papers from different regions of the world (Europe, North America, Africa/Middle East, Asia, Central and South America—including Mexico or Australia/New Zealand). Results are presented as absolute numbers with percentages. If authors had provided explanations for the registration problem, EL and SL classified these explanations into one of six categories (ambiguity in the definition of a clinical trial; error attributed to another team member; author belief that the trial was not subject to registration requirements; registry error; unaware of or misunderstood registration requirements; technical difficulties) and selected representative quotations to illustrate each category. We removed potentially identifying information from quotations, either summarising it in a non-identifiable way in square brackets or omitting words, indicated by an ellipsis.
Results
BMJ editors identified 123 unregistered or retrospectively registered clinical trials over the 49-month period of the study. Table 1 shows the characteristics of these studies. One hundred and ten (89.4%) were retrospectively registered and nine (7.3%) were unregistered. Three studies had been registered in an unapproved registry. One study (classified as ‘Other’) was originally assumed to be unregistered because the authors did not supply registration information. However, we discovered that it had subsequently been published elsewhere and that publication included registration information showing the trial had been prospectively registered.
Characteristics of 123 trials rejected for registration deficiencies
Over three-quarters of studies had only a single funder. Almost two-thirds were funded entirely or in part by government sources, while about a quarter received some or all of their funding from philanthropic foundations. Only seven studies (5.7%) received funding from industry, and in all but one case other funding sources were also listed; thus, only one trial in our sample was solely industry-funded. The number of improperly registered trials varied over the years of the study, but averaged 2.1 a month. From 2014 to 2016, the 3 years for which data are complete, the number ranged from 26 in 2016 to 36 in 2015. Almost three-quarters of retrospectively registered studies were registered in ClinicalTrials.gov or ISRCTN. Of the 123 trials, 19 (5.5%) in our series evaluated FDA-regulated interventions. Three (15.8%) of these trials were unregistered and 16 were retrospectively registered. Among the 104 trials in our series that reported on unregulated interventions, 6 (5.8%) were unregistered.
Table 2 shows the subsequent publication fate of papers with registration problems that were rejected by The BMJ through the end of 2015.Of the 70, 67 papers rejected for registration problems (97%) were subsequently published in another journal. We could not locate a subsequent publication for three papers. One paper had been published in another journal but then withdrawn. No reason was given for the withdrawal. Only two papers were published in a journal without an impact factor. One paper was eventually published in The BMJ, after the authors appealed and provided an explanation that was deemed to justify retrospective registration. The registration problem was explained in the paper. No paper was published in a journal with a higher-impact factor than The BMJ. The median impact factor of the journals where papers were subsequently published was 4.972 (IQR: 3.586). Only two of the subsequently published papers provided a description of the registration problem and explained why the journal chose to publish it despite the problem.
Publication fate of the 70 papers submitted through 2015*
We noted several inconsistencies and alterations in some of the subsequently published papers that we located (online supplementary appendix). For example, one study no longer reported a trial registration number and the research was described as an ‘implementation-effectiveness study’ rather than a clinical trial. Another paper was listed on the researcher’s website as having been published in The BMJ even though it had been rejected. In several papers, the date participants were enrolled in the study had been changed from that reported in the version of the paper submitted to The BMJ.
Supplementary file 1
The majority of unregistered trials were from Europe or Asia (table 3). One-third of unregistered studies were submitted in 2017. The most common reason given for lack of registration was that the researchers did not believe their study met the ICMJE definition of a clinical trial. In all of these cases, researchers had randomised healthcare providers or groups of participants and examined outcomes that they claimed were not medical, although BMJ editors disagreed with this assessment. In several cases, authors were unaware of requirements for trial registration, or assumed that ethical approval was the same thing as registration.
Characteristics and publication fate of the nine unregistered studies
Table 4 lists common reasons for late or non-registration of trials, grouped by category and illustrated with quotations. Authors of 40/123 (32.5%) of the papers provided explanations for registration problems. Overall, the most common explanation authors gave for registration deficiencies was the belief that trials randomising health professionals did not need to be registered.
Selected explanations for registration deficiencies
Discussion
Twelve years after trial registration requirements were imposed by ICMJE journals, compliance is imperfect. In this study of trials submitted to a single high-impact journal, it was rare that trials were unregistered but retrospective registration remains a problem. This is a concern because retrospective registration may abet selective reporting of outcomes.14 The number of trials with registration problems is small in comparison to the number of research papers received by The BMJ but ‘large in relation to zero, which should be the goal.’15
Unregistered and retrospectively registered trials come from all parts of the world, not just those with less training and tradition in research. Many such trials originated from well-established research groups in the USA, England, Canada and other countries with a stronger tradition and more training in clinical research. It is possible that this situation may improve, at least in the USA where the US FDA Amendments Act of 2007 imposed a legal mandate to register clinical trials of many FDA-regulated products. Research suggests this has increased trial registration, publication and outcome reporting fidelity, but clearly more progress is needed.16
It is notable that there are so few industry-funded trials on this list of 123 late or unregistered studies. The problem of deficient trial registration is almost entirely confined to government, foundation and academically funded studies. This is consistent with other research and with the espoused commitment of most pharmaceutical companies to trial registration.17 18 Most clinical trials submitted to The BMJ-tested interventions that are not regulated by The FDA. Previous work has shown that trials of unregulated interventions are more likely to have registration deficiencies, compared with trials that evaluate FDA-regulated interventions.19
Contrary to the worries of some authors and others, trials that have been retrospectively registered seem to be easily published elsewhere, and are not exclusively relegated to obscure, low-impact journals. This is heartening for authors of studies lacking timely registration. It suggests, however, that while many journals say they require prospective registration, they do not mean it. They are prepared to overlook registration problems, especially in borderline situations, and are encouraged to do so by authorities.4 9 Still other journals have deliberately moved away from requirements for prospective registration requirements, citing ‘the need to uncover all trials and their results.’20 These moves may put journals that do enforce registration requirements at a disadvantage in the competition to publish clinical trials and maintain the goodwill of trialists.
Authors offer a variety of reasons for failure to register trials on a timely basis, including lack of awareness of requirements, failure of a junior team member or study coordinator to register the study or problems with the registration process. In checking on the publication fate of rejected papers, we identified several instances in which authors appeared to have changed dates in trial registries or manuscripts before submission to the publishing journal. We also noted other instances of possible misbehaviour. These things suggest that registration deficiencies may be markers of poor organisation and supervision of a research project, or a disregard for accuracy and rules. This is consistent with previous research showing that small errors and discrepancies in published papers reflected more serious problems with the quality of the research.21 Also of note, in several cases authors were successful in persuading a trial registry (ISRCTN) to amend dates and change the categorisation of their study.
Our study has a number of limitations. We relied on BMJ research editors to report studies with registration problems; some such trials may have been missed and therefore not included in this audit. Information about the reason for late registration or lack of registration is incomplete because not all authors provided explanations. This represents the experience of a single journal and may not be representative of the situation in other journals. Authors submitting to higher-impact journals such as The BMJ are a self-selected group that might be more aware and observant of registration requirements. The problem might be worse at low-impact journals. High-impact journals (those with an impact factor of 10 or greater) are more likely than low-impact journals to endorse ICMJE trial registration requirements and they publish a lower proportion of unregistered trials.22
Conclusions
This paper makes clear that one notable area of success for trial registration has been in industry-funded research, where pharmaceutical companies and device manufacturers often have both a legal obligation to register and financial and reputational interests at stake when planning and publishing trials. In our sample, the majority of studies with registration problems were funded by government or private grants. Such funders appear poised to address the problem of improperly registered trials. In January 2017, the US National Institutes of Health along with 20 other funders committed themselves to providing funds only for prospectively registered trials, regardless of the intervention that is assessed.23
In their explanations for trial registration problems, many authors cited a lack of familiarity with the registration process, misunderstanding about what constitutes a clinical trial, or confusion about which member of the study team was responsible for registering the trial. Our findings suggest that the ICMJE should clarify its definition of a clinical trial, commenting specifically on two matters: (1) whether quality improvement or implementation trials are exempt from registration requirements and (2) whether changes in health provider behaviour (eg, medical errors, prescription of specific treatments) constitute medical outcomes.
Our data show that even 12 years after many medical journals have endorsed ICMJE registration requirements and indicated that they require prospective registration, adherence to this standard is not universal. The fact that trials with registration issues were almost always published in another journal, and that these journals did not disclose registration problems, indicates that journal editors, as well as researchers and funders, can play an important role in improving the registration system. By refusing to publish such trials or at least by prominently disclosing registration issues, medical journals could provide a strong motivation for researchers to register their trials going forward.
Future research
Future research should evaluate the link between registration deficiencies and the underlying quality of the scientific data included in these trials. We noted several instances of possible methodological problems or questionable researcher conduct for some of the trials we examined. Furthermore, research is needed to understand the reasons for lax enforcement of trial registration requirements by editors.
Acknowledgments
The authors thank the research editors of The BMJ for their assistance in identifying unregistered and retrospectively registered trials.
References
Footnotes
Contributors EL conceived the idea for the study, obtained, recorded, analysed and organised the data, wrote the first draft of the paper and was a major contributor in revising the manuscript. SL participated in obtaining, organising, analysing and interpreting data, created the figure, formatted tables and edited the final version of the manuscript. SC independently checked and verified a portion of the data, analysed and interpreted data and was a major contributor in revising the manuscript. All authors read and approved the final manuscript.
Funding This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests EL has an academic interest in trial registration and research integrity and has published on the topic. EL and SC are research editors for The BMJ, which is a well-recognised proponent of trial registration. The BMJ is a founding member of the ICMJE and was involved in the development of requirements for clinical trial registration as a condition of publication.
Patient consent Not required.
Ethics approval The data used in this study were part of an internal BMJ audit of author adherence to trial registration requirements. Authors are informed upon submission of a research paper that The BMJ routinely conducts audits of papers submitted to the journal and that their paper may be included.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement No additional data are available. All deidentified data supporting the conclusions of the paper are included in this published article (and its online supplementary appendix).