You are here

Article Text

Article menu

This article has a correction. Please see:

XML
Urology
Research
Do urology journals enforce trial registration? A cross-sectional study of published trials
  1. Frank Kunath1,2,
  2. Henrik R Grobe1,3,
  3. Bastian Keck2,
  4. Gerta Rücker4,
  5. Bernd Wullich2,
  6. Gerd Antes1,
  7. Joerg J Meerpohl1,5
  1. 1German Cochrane Centre, Institute of Medical Biometry & Medical Informatics, University Medical Centre Freiburg, Freiburg/Br., Germany
  2. 2Department of Urology, University Clinic Erlangen, Erlangen, Germany
  3. 3Department of General & Visceral Surgery, Freiburg University Hospital, Freiburg/Br., Germany
  4. 4Institute of Medical Biometry and Medical Informatics, University Medical Centre, Freiburg/Br., Germany
  5. 5Paediatric Haematology & Oncology, Centre for Paediatrics & Adolescent Medicine, University Medical Centre Freiburg, Freiburg/Br., Germany
  1. Correspondence to Dr Frank Kunath; kunath{at}cochrane.de

Abstract

Objectives (1) To assess endorsement of trial registration in author instructions of urology-related journals and (2) to assess whether randomised controlled trials (RCTs) in the field of urology were effectively registered.

Design Cross-sectional study of author instructions and published trials.

Setting Journals publishing in the field of urology.

Participants First, the authors analysed author instructions of 55 urology-related journals indexed in ‘Journal Citation Reports 2009’ (12/2010). The authors divided these journals in two groups: those requiring and those not mentioning trial registration as a precondition for publication. Second, the authors chose the five journals with the highest impact factor (IF) from each group.

Intervention MEDLINE search to identify RCTs published in these 10 journals in 2009 (01/2011); search of the clinical trials meta-search interface of WHO (International Clinical Trials Registry Platform) for RCTs that lacked information about registration (01–03/2011). Two authors independently assessed the information.

Outcome measures Proportion of journals providing advice about trial registration and proportion of trials registered.

Results Of 55 journals analysed, 26 (47.3%) provided some editorial advice about trial registration. Journals with higher IFs were more likely to mention trial registration explicitly (p=0.015). Of 106 RCTs published in 2009, 63 were registered (59.4%) with a tendency to an increase after 2005 (83.3%, p=0.035). 71.4% (30/42) of the RCTs that were published in journals mentioning and requiring registration, and 51.6% (33/64) of the RCTs that were published in journals that did not mention trial registration explicitly were registered. This difference was statistically significant (p=0.04).

Conclusions The existence of a statement about trial registration in author instructions resulted in a higher proportion of registered RCTs in those journals. Journals with higher IFs were more likely to mention trial registration.

This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/2.0/ and http://creativecommons.org/licenses/by-nc/2.0/legalcode.

https://doi.org/10.1136/bmjopen-2011-000430

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Article summary

    Article focus

    • Trial registration can increase scientific transparency, but its implementation in specialty fields such as urology is unclear.

    • To assess the endorsement of trial registration in the author instructions of urology-related journals.

    • To assess whether randomised controlled trials in the field were effectively registered.

    Key messages

    • A statement of trial registration in author instructions resulted in a higher proportion of registered randomised controlled trials.

    • Journals with high impact factors were more likely to mention trial registration.

    • We suggest, though, that ensuring trial registration is not the responsibility only of the editors. Medical scientists should realise that trial registration is necessary to contribute to transparency in research.

    Strength and limitations of this study

    • Two authors independently assessed information regarding editorial advice about trial registration and identified the randomised controlled trials.

    • Potential bias occurred if registered randomised controlled trials were reported without giving a registration number and we could not identify them in the meta-search interface of WHO (International Clinical Trials Registry Platform).

    • Results might not be representative of the uro-nephrological field as a whole and reported figures may overestimate compliance with trial registration.

    Introduction

    Transparency is essential for the efficient transfer of scientific knowledge gained from research into practice. Nevertheless, recent studies have shown that transparency is impaired by non-publication or inappropriate publication of trial results. A major reason is selective publication, either of whole trials or of certain outcomes, which is referred to as publication bias or outcome reporting bias, respectively.1 2 Other reasons that hamper the generation of valid results from trials are limitations in trial methodology and reporting of trial results. A recently published Cochrane Review evaluated studies that compared the content of protocols and register entries with published reports of randomised controlled trials (RCTs). The authors found that it is common practice to add, omit or change outcome parameters in the reporting of study results, which leads to discrepancies between study protocols or register entries and reported results.3 As very recently shown by a retrospective analysis of data entries in http://clinicaltrials.gov on randomised trials on diseases in the digestive system, information available in trial registers was partly inadequate, 30% of trials were registered incorrectly after their completion date and registered information and trial publications showed discrepancies.4

    The complete registration of all trials has the potential to increase transparency in scientific research. Members of the International Committee of Medical Journal Editors (ICMJE) promoted trial registration in 2004 by publishing a statement on it.5 The ICMJE defined 1 July 2005 as a key date requiring prospective trial registration (registration before recruitment started) for trials started after this date. For trials that began enrolment prior to this date, the ICMJE member journals allowed for retrospective registration by 13 September 2005.5 WHO supported this joint effort of journals by founding the International Clinical Trials Registry Platform (ICTRP), which brings together data sets provided by different primary registers worldwide. As of 1 July 2007, also the Surgery Journal Editors Group member journals required prospective registration of all clinical trials. From this date on, the Surgery Journal Editors Group member journals planned to consider trials only if they are registered before enrolment of the first study subject.6 In 2007, several editors from major medical journals stated that ‘three years ago, trial registration was the exception and now it is the rule’,7 suggesting that trial registration nowadays is in fact an integral part of clinical research.

    It is unclear, however, to what extent trial registration is implemented in specialty fields such as urology. To address this question, we aimed to assess the endorsement of trial registration in journal author instructions in relevant urology journals. Second, we sought to determine whether a statement of trial registration in author instructions resulted in a higher proportion of registered RCTs published in 2009.

    Methods

    We included in this study all journals that were indexed in the subject categories ‘Urology & Nephrology’ or ‘Andrology’ in the ‘Journal Citation Reports 2009’.8 We examined author instructions available via journal websites (12/2010) and did not impose any language restriction. Two authors (FK and HRG) independently read each document and used relevant key words in electronic full-text searches to extract information about whether the journal (1) publishes original investigations, (2) gives editorial advice about trial registration and (3) endorses the ‘Uniform Requirements for Manuscripts Submitted to Biomedical Journals’ (URM) recommendation of the ICMJE.9 Disagreements were resolved by rechecking the respective websites and, if necessary, by discussion with a third author (JJM).

    Journals reporting reviews or commissioned articles only were excluded. We noted if journals recommended or required trial registration as a precondition for publication. In addition, we assessed the context in which the URM appeared in a journal. If the journal guidelines mentioned the URM only in the context of ‘conflicts of interest’ or ‘style of references’, we classified the URM for this journal as ‘not relevant’ for advice about trial registration. If trial registration was either required or recommended or if the guidelines mentioned the URM in the context of trial registration, we classified the journal as ‘direct reference to trial registration’. If the guidelines mentioned the URM in the context of general manuscript preparation without other editorial advice on trial registration, we classified it as ‘indirect reference to trial registration’.

    In a second step, we assessed whether a statement of trial registration in author instructions of uro-nephrology journals resulted in a higher proportion of registered RCTs. On the basis of our evaluation, we selected the 10 journals (five journals that mentioned and required trial registration and five journals that did not mention trial registration directly in their author instructions) with the highest impact factor (IF) in their respective group (table 1). We have chosen these journals to obtain a reasonably sized, representative sample of trials published in the major uro-nephrological journals supporting or not supporting prospective trial registration. We then searched Ovid MEDLINE (01/2011, box 1) to identify RCTs that were published in 2009 in these journals using the ‘Cochrane Highly Sensitive Search Strategy’.10

    View this table:
    Table 1

    Groups of journals giving some or no advice on trial registration in their author instructions

    Box 1

    Search strategy for identifying randomised controlled trials in journals relevant to urology

    Database: Ovid MEDLINE(R) In-Process & Other Non-Indexed Citations, Ovid MEDLINE(R) Daily and Ovid MEDLINE(R) 1946 to Present.

    Searched: 17.01.2011.

    Responsible searcher: Kunath

    1. european urology.jn.

    2. kidney international.jn.

    3. american journal of kidney diseases.jn.

    4. clinical journal of the american society of nephrology cjasn.jn.

    5. journal of the american society of nephrology.jn.

    6. bju international.jn.

    7. journal of urology.jn.

    8. american journal of nephrology.jn.

    9. nephrology dialysis transplantation.jn.

    10. prostate.jn.

    11. 1 or 2 or 3 or 4 or 5 or 6 or 7 or 8 or 9 or 10

    12. randomized controlled trial.pt.

    13. controlled clinical trial.pt.

    14. randomized.ab.

    15. placebo.ab.

    16. clinical trials as topic.sh.

    17. randomly.ab.

    18. Trial.ti.

    19. 12 or 13 or 14 or 15 or 16 or 17 or 18

    20. exp animals/ not humans.sh.

    21. 19 not 20

    22. 11 and 21

    23. limit 22 to yr="2009"

    One author (FK) screened the titles and abstracts of all retrieved references and excluded only citations that were clearly irrelevant or duplicates. Two authors (FK and HRG) then independently from each other identified RCTs and excluded non-randomised trials and discussed discrepancies with a third party (JJM). Trials recruiting patients prior to 2004 were excluded because registration was not strongly promoted at that time. The 2004 ICMJE statement promoting trial registration had defined 1 July 2005 as a key date for prospective trial registration (registration before recruitment started). To investigate possible changes in registration practice, we subdivided trials into two groups according to the start of patient recruitment: 2004–2005 and ≥2006. The ICMJE accepted retrospective registration (registration when recruitment was already happening) of trials that began before 1 July 2005.9 For this reason, we did not investigate if trial registration was done pro- or retrospectively.

    We searched the ICTRP (01–03/2011) for every report of an RCT without information about trial registration by using relevant key words based on the Patients, Intervention, Control, Outcome, Time scheme.11 The ICTRP is periodically updated and contains data sets of registers from Australia, New Zealand, China, India, Republic of Korea, Germany, Iran, Japan, Africa, Sri Lanka and the Netherlands as well as from ‘ClinicalTrials.gov’ and the ‘International Standard Randomised Controlled Trial Number Register’.

    The influence of the IF on trial registration endorsement was evaluated using logistic regression. Statistical tests were two sided, and p<0.05 was considered significant.

    Results

    We identified 69 journals in the subject categories ‘Urology & Nephrology’ (n=63) and ‘Andrology’ (n=6). Fourteen journals were excluded for various reasons (figure 1); 55 journals were evaluated. The IFs ranged from 0.054 to 7.689.

    Figure 1
    Figure 1

    Selection of relevant journals from the ‘Journal Citation Reports 2009’.

    Twenty journals (36.4%) mentioned trial registration as a direct reference in their author instructions, two of these (10%, IF<2.0) recommended trial registration and 18 journals (90%, IF>0.01) required trial registration as a precondition for publication (table 2).

    View this table:
    Table 2

    Endorsement of trial registration in author instructions

    A total of 32 journals (58.2%) mentioned the URM of the ICMJE. However, 18 journals (32.7%) mentioned the URM only in the context of conflict of interest (n=8) or style of references (n=10). Fourteen journals (25.5%) mentioned the URM in the context of trial registration (n=2) or general manuscript preparation (n=12). All journals that mentioned the URM in the context of trial registration also required or recommended trial registration. This was additionally the case for six of the 12 journals mentioning the URM in the context of general manuscript preparation. Thus, six journals (10.9%) gave an indirect reference to trial registration by mentioning the URM exclusively in the context of general manuscript preparation (table 2).

    Twenty-six journals (47.3%) made some kind of reference to trial registration, either by directly or indirectly referring to the URM in their author instructions (table 2). Six of seven journals (85.7%) with IFs above 4.01 included such indirect editorial advice. The likelihood of a direct reference to trial registration was significantly higher with increasing IF (p=0.015).

    Our search strategy for the evaluation of the registration practice for RCTs published in 2009 retrieved 429 citations; our evaluation included 106 RCTs (figure 2). Fifty-two RCTs (49%) had started patient recruitment in 2004–2005, 36 (34%) during or after 2006 and in 18 RCTs (17%), the start of patient recruitment was not specified (table 3). These 18 trials were cross-over trials, RCTs with an interventional duration of <12 weeks (n=14, 77.8%) or RCTs with a follow-up of <2 years (n=4, 22.2%).

    Figure 2
    Figure 2

    Selection of randomised controlled trials (RCTs).

    View this table:
    Table 3

    Registration of RCTs

    In total, 63 of the 106 RCTs (59.4%) were registered, 33 with recruitment from 2004 to 2005 (33/52, 63.5%) and 30 with recruitment in 2006 or later (30/36, 83.3%) (table 3). The difference between these two proportions was statistically significant (RR 0.76, 95% CI 0.59–0.98, p=0.035). Of the 63 registered RCTs, 48 specified the registration number (76.2%). Following the search of the ICTRP, 15 additional trials (23.8%) were classified as ‘registered’, although no reference to their registration was given in their publication. We found no differences in reporting of registration between RCTs with patient recruitment in 2004–2005 and during or after 2006 (table 3, registration mentioned in article: 75.8% vs 76.7%; no registration mentioned in article but registered in the ICTRP: 24.2% vs 23.3%, respectively). Forty-three RCTs (40.6%) did not report registration and could not be identified in the ICTRP (table 3).

    Of the 42 RCTs that were published in journals that mentioned and required trial registration as a precondition for publication, 71.4% (30/42) were registered and 28.6% (12/42) were not registered (table 4). In contrast to this, of the 64 RCTs that were published in journals that did not mention trial registration directly in their author instructions, only 51.6% (33/64) were registered and 48.4% (31/64) were not registered (table 4). The proportion of reports of registered trials was significantly higher in journals that mentioned and required trial registration as a precondition for publication than those that did not mention trial registration directly in their author instructions (p=0.04).

    View this table:
    Table 4

    Registration of RCTs published in journals mentioning and requiring or not mentioning trial registration in their author instructions

    Discussion

    We assessed the endorsement of trial registration in author instructions of urology-related journals. Only 36.4% of uro-nephrology journals gave direct editorial advice about trial registration. Of interest, a higher IF was significantly associated with the likelihood of giving direct advice about trial registration. More than half of all RCTs published in 2009 and the vast majority (83%) of trials with patient recruitment during or after 2006 were registered (table 3). Three quarters of the RCTs that were published in journals mentioning and requiring trial registration as a precondition for publication were registered (table 4).

    Although the ICMJE, WHO and the World Medical Association support trial registration,9 12 13 the advice given in the author instructions of uro-nephrology journals does not sufficiently reflect this recommendation. This situation, however, is not as comparatively negative as it might seem at first. Meerpohl et al14 evaluated the editorial policies of paediatric journals and demonstrated that only one-quarter of all selected journals required or recommended trial registration in their author instructions. Another study compared Italian journals (n=76) indexed in Medline and randomly chosen journals from the UK (n=76) with regard to editorial advice on trial registration. Surprisingly, none of the Italian journals required trial registration, while 28% of the UK journals mentioned registration.15

    The data presented here show that the reporting of registration can still be improved, and we suggest that this problem is general in the scientific literature. Reveiz et al evaluated the key methodological items of RCTs published in 55 of the highest-ranking journals. Only 36% of 148 included RCTs reported that the study was registered in any trial register.16 Awareness of trial registration was raised in the scientific community in 2005. This awareness translated into a statistically significant increase in trial registrations for RCTs with patient recruitment after 2005 in urology and better registration of trials in other medical specialties. Hamm et al17 found that only one-quarter of all paediatric RCTs published in 2007 were in fact registered. Similarly, a study of 323 RCTs from medical departments published in journals with high IFs identified only 46% of those as adequately registered trials.18 However, these studies did not differentiate between trial registration for RCTs with patient recruitment before, during or after 2005.

    Potential bias in our data may have emerged when registered RCTs could not be identified in the ICTRP, and the start of patient recruitment was not mentioned in the manuscript. Because this potential limitation hinders the detection of studies in registries, it is also a handicap for transparency in research. If we assumed that patient recruitment in trials not reporting a registration number or not specifying their conduct happened in or after 2006, the proportion of registered trials would decrease. In addition, the RCTs analysed were taken from the five uro-nephrological journals with the highest IF in their respective categories. It may well be that RCTs published in these journals with higher IFs are more likely to be registered. Our results might, therefore, overestimate compliance with trial registration and be, accordingly, not representative of the uro-nephrological field as a whole.

    In an ideal world, knowledge translation would be supported by the prospective registration of trials as well as the comprehensive and transparent publication of all trial results in either a register or a journal. This transparency would guarantee public access to research results. Our study points to some limitations on the road to this ideal setup. On the one hand, not all trials are registered yet. Furthermore, some registered trials are not identified as such in their publication, so to obtain access to the additional information provided by the entry in the register, a separate search is required. Consequently, a reference to the registration number of trials should be mandatory in publications. One consideration, though, is that if journal editors rigorously require trial registration prior to the start of patient recruitment as a precondition for publication, only these prospectively registered trials could be published in the future. Although such a practice would be consistent with the ICMJE recommendations, it could lead to the non-publication of non-registered trials, increasing the proportion of non-published studies.

    Conclusions

    A statement of registration in author instructions resulted in a higher proportion of registered RCTs. However, journal editors could still do more by giving advice about trial registration in their author instructions and by ensuring that manuscripts include registration details of trials published in their journal. It is not, however, only the editors' responsibility to ensure trial registration. In fact, the scientists and authors need to act to ensure trial registration. Researchers and sponsors should consider trial registration as an ethical imperative to enhance transparency in research and not as an obstacle to research.

    References

    1. ↵
      1. Dickersin K,
      2. Chan S,
      3. Chalmers TC,
      4. et al
      . Publication bias and clinical trials. Control Clin Trials 1987;8:343–53.
      OpenUrlCrossRefPubMedWeb of Science
    2. ↵
      1. Hopewell S,
      2. Loudon K,
      3. Clarke MJ,
      4. et al
      . Publication bias in clinical trials due to statistical significance or direction of trial results. Cochrane Database Syst Rev 2009;(1):MR000006.
    3. ↵
      1. Dwan K,
      2. Altman DG,
      3. Cresswell L,
      4. et al
      . Comparison of protocols and registry entries to published reports for randomised controlled trials. Cochrane Database Syst Rev 2011;(1):MR000031.
    4. ↵
      1. Wildt S,
      2. Krag A,
      3. Gluud L
      . Characteristics of randomised trials on diseases in the digestive system registered in ClinicalTrials.gov: a retrospective analysis. BMJ Open 2011;1:e000309.
      OpenUrlAbstract/FREE Full Text
    5. ↵
      1. De Angelis C,
      2. Drazen JM,
      3. Frizelle FA,
      4. et al
      . Clinical trial registration: a statement from the International Committee of Medical Journal Editors. Ann Intern Med 2004;141:477–8.
      OpenUrlCrossRefPubMedWeb of Science
    6. ↵
      1. Anonymous
      Consensus statement on mandatory registration of clinical trials. Ann Surg 2007;245:505–6.
      OpenUrlPubMed
    7. ↵
      1. Laine C,
      2. Horton R,
      3. DeAngelis CD,
      4. et al
      . Clinical trial registration: looking back and moving ahead. Lancet 2007;369:1909–11.
      OpenUrlCrossRefPubMedWeb of Science
    8. ↵
      ISI Web of Knowledge. Journal Citation Reports. New York: Thomson Reuters, 2011.
    9. ↵
      International Committee of Medical Journal Editors. Uniform Requirements for Manuscripts Submitted to Biomedical Journals: Writing and Editing for Biomedical Publication. http://www.icmje.org. (accessed 6 Jan 2011).
    10. ↵
      1. Higgins JPT,
      2. Green S
      1. Lefebvre C,
      2. Manheimer E,
      3. Glanville J
      . Chapter 6: Searching for studies. In: Higgins JPT, Green S, eds. Cochrane Handbook for Systematic Reviews of Interventions. Chichester, UK. Wiley, 2008.
    11. ↵
      1. O'Connor D,
      2. Green S,
      3. Higgins JPT
      . Chapter 5: Defining the Review Question and Developing Criteria for Including Studies. Chichester (UK): John Wiley & Sons, 2008.
    12. ↵
      World Medical Association. WMA Declaration of Helsinki—Ethical Principles for Medical Research Involving Human Subjects. 2008. http://www.wma.net/en/30publications/10policies/b3/ (accessed 2 Mar 2011).
    13. ↵
      World Health Organization. International Clinical Trials Registry Platform (ICTRP). http://www.who.int/ictrp/trial_reg/en/index.html (accessed 2 Mar 2011).
    14. ↵
      1. Meerpohl JJ,
      2. Wolff RF,
      3. Niemeyer CM,
      4. et al
      . Editorial policies of pediatric journals: survey of instructions for authors. Arch Pediatr Adolesc Med 2010;164:268–72.
      OpenUrlCrossRefPubMedWeb of Science
    15. ↵
      1. Matarese V
      . Relationship between quality and editorial leadership of biomedical research journals: a comparative study of Italian and UK journals. PLoS One 2008;3:e2512.
      OpenUrlCrossRefPubMed
    16. ↵
      1. Reveiz L,
      2. Cortes-Jofre M,
      3. Asenjo Lobos C,
      4. et al
      . Influence of trial registration on reporting quality of randomized trials: study from highest ranked journals. J Clin Epidemiol 2010;63:1216–22.
      OpenUrlCrossRefPubMedWeb of Science
    17. ↵
      1. Hamm MP,
      2. Hartling L,
      3. Milne A,
      4. et al
      . A descriptive analysis of a representative sample of pediatric randomized controlled trials published in 2007. BMC Pediatr 2010;10:96.
      OpenUrlCrossRefPubMed
    18. ↵
      1. Mathieu S,
      2. Boutron I,
      3. Moher D,
      4. et al
      . Comparison of registered and published primary outcomes in randomized controlled trials. JAMA 2009;302:977–84.
      OpenUrlCrossRefPubMedWeb of Science

    Supplementary materials

    • Supplementary Data

      This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

      Files in this Data Supplement:

    Footnotes

    • The manuscript has been read and approved by all authors and has not been submitted elsewhere. The authors confirm that they have no conflict of interest.

    • To cite: Kunath F, Grobe HR, Keck B, et al. Do urology journals enforce trial registration? A cross-sectional study of published trials. BMJ Open 2011;1:e000430. doi:10.1136/bmjopen-2011-000430

    • Funding Supported by a Ferdinand Eisenberger grant of the Deutsche Gesellschaft für Urologie (German Society of Urology) grant ID KuF1/FE-10.

    • Competing interests None.

    • Contributors (1) Conception and design: FK, BW, GA and JJM. (2) Data acquisition: FK and HRG. (3) Data analysis and interpretation: FK, HRG, BK, GR, BW and JJM. (4) Drafting the manuscript: FK, HRG, BK and JJM. (5) Critical revision of the manuscript for scientific and factual content: HRG, BK, GR, BW, GA and JJM. (6) Statistical analysis: FK, GR and JJM. (7) Supervision: FW, GA and JJM. (8) Final approval of the version published: FK, HRG, BK, GR, FW, GA and JJM.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Data sharing statement Dataset and statistical code available from the Dryad repository, doi:10.5061/dryad.3fb861d0.

    Linked Articles