Article Text
PDF
Abstract
Objective To explore the implementation of the International Committee of Medical Journal Editors (ICMJE) data-sharing policy which came into force on 1 July 2018 by ICMJE-member journals and by ICMJE-affiliated journals declaring they follow the ICMJE recommendations.
Design A cross-sectional survey of data-sharing policies in 2018 on journal websites and in data-sharing statements in randomised controlled trials (RCTs).
Setting ICMJE website; PubMed/Medline.
Eligibility criteria ICMJE-member journals and 489 ICMJE-affiliated journals that published an RCT in 2018, had an accessible online website and were not considered as predatory journals according to Beall’s list. One hundred RCTs for member journals and 100 RCTs for affiliated journals with a data-sharing policy, submitted after 1 July 2018.
Main outcome measures The primary outcome for the policies was the existence of a data-sharing policy (explicit data-sharing policy, no data-sharing policy, policy merely referring to ICMJE recommendations) as reported on the journal website, especially in the instructions for authors. For RCTs, our primary outcome was the intention to share individual participant data set out in the data-sharing statement.
Results Eight (out of 14; 57%) member journals had an explicit data-sharing policy on their website (three were more stringent than the ICMJE requirements, one was less demanding and four were compliant), five (35%) additional journals stated that they followed the ICMJE requirements, and one (8%) had no policy online. In RCTs published in these journals, there were data-sharing statements in 98 out of 100, with expressed intention to share individual patient data reaching 77 out of 100 (77%; 95% CI 67% to 85%). One hundred and forty-five (out of 489) ICMJE-affiliated journals (30%; 26% to 34%) had an explicit data-sharing policy on their website (11 were more stringent than the ICMJE requirements, 85 were less demanding and 49 were compliant) and 276 (56%; 52% to 61%) merely referred to the ICMJE requirements. In RCTs published in affiliated journals with an explicit data-sharing policy, data-sharing statements were rare (25%), and expressed intentions to share data were found in 22% (15% to 32%).
Conclusion The implementation of ICMJE data-sharing requirements in online journal policies was suboptimal for ICMJE-member journals and poor for ICMJE-affiliated journals. The implementation of the policy was good in member journals and of concern for affiliated journals. We suggest the conduct of continuous audits of medical journal data-sharing policies in the future.
Registration The protocol was registered before the start of the research on the Open Science Framework (https://osf.io/n6whd/).
- clinical trials
- statistics & research methods
- epidemiology
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
This original research is a comprehensive assessment of data-sharing policies based on the International Committee of Medical Journal Editors (ICMJE) recommendations.
In our approach we focused on both ICMJE-member and ICMJE-affiliated journals.
Data-sharing policies are set in a fast-changing environment and results obtained today might be outdated in the future.
One limitation is that we only relied on online information.
Introduction
In June 2017, the International Committee of Medical Journal Editors (ICMJE) published a statement supporting data-sharing practices for randomised controlled trials (RCTs). For the ICMJE, ‘there is an ethical obligation to responsibly share data generated by interventional clinical trials because trial participants have put themselves at risk’ with the aim to ‘maximize the knowledge gained’ from these outstanding studies. The ICMJE policy requires a specific data-sharing statement to be included in each newly submitted paper (and prespecified in study registration) containing clinical trial data, starting 1 July 2018.1
Examples of medical journals having a data-sharing policy before this requirement were few. In 2007, the Annals of Internal Medicine was the first journal to adopt a policy encouraging data-sharing practices.2 The BMJ adopted a similar policy encouraging data-sharing in 2009,3 and went further by making it mandatory in 2013 for drugs and devices4 and for all RCTs in 2015.5 PLOS journals also adopted a strict policy enforcing RCT data-sharing in 2014.6 No other leading general medical journal has had a specific policy for data-sharing in RCTs.
The ICMJE policy could therefore have an impact on biomedical literature as a whole. At the time of the present research, the ICMJE included two organisations (the US National Library of Medicine and the World Association of Medical Editors) and 14 journals, including leading medical journals such as The New England Journal of Medicine (NEJM) and The Lancet. In addition, about 5000 affiliated journals follow the ICMJE recommendations.7 As this policy is now in place, it is important to monitor its implementation both in the ICMJE-member journals and in the ICMJE-affiliated journals. It is also important to assess intentions to share data among RCTs published in the journals implementing a data-sharing policy.
Methods
The protocol was registered before the start of the research on the Open Science Framework (OSF) (https://osf.io/n6whd/). This study was divided into two parts: a survey of journal data-sharing policies and a survey of published RCTs.
Survey of journal data-sharing policies
Journal eligibility criteria
Two samples of journals were surveyed: the 14 ICMJE-member journals at the time of the present research (Annals of Internal Medicine, British Medical Journal, Bulletin of the World Health Organization, German Medical Journal, Ethiopian Journal of Health Sciences, Iranian Journal of Medical Sciences, Journal of the American Medical Association, Journal of Korean Medical Science, New England Journal of Medicine, New Zealand Medical Journal, PLOS Medicine, The Lancet, Medical Journal of Chile and Danish Medical Journal) and a sample of ICMJE-affiliated journals listed on the ICMJE website on 1 February 2019. Journals were included if they (1) had medical content; (2) had published at least one RCT in 2018; (3) had published articles in English, German, French, Spanish or Portuguese; (4) had an accessible online website; and (5)were not considered as ‘predatory’ journals according to Beall’s list.8
Search strategy for journals
The ICMJE website was consulted to copy the list of all ICMJE-member journals and all 4892 ICMJE-affiliated journals. In cases where an affiliated journal changed its name after registration on the ICMJE website (eg, Cancer Immunity changed to Cancer Immunology Research), we checked whether the new name was also listed on the ICMJE website. If this was the case, the journal was considered as non-eligible and was marked as ‘discontinued’; otherwise it was included.
All 4892 ICMJE-affiliated journals were assessed for eligibility in random order obtained using the R statistical software.9 The results of the randomisation can be found in the supplementary material on the OSF page.10 The first 489 journals that met the selection criteria (10% of affiliated journals) were included, enabling us to estimate a proportion of 50% (the worst-case scenario for precision estimates) with a precision (boundaries of the 95% CI) of about ±4.5%.
Journal selection and data extraction
A data extraction sheet and a data extraction explanatory document were developed.10 Three investigators in charge of data extraction (MS, LC, HG) had a 1-hour training session and completed a pilot data extraction on 10 journals. For each journal, pairs of these investigators independently assessed eligibility (with reasons in case of non-eligibility) and extracted the data on all the outcomes listed in the following sections from each included journal. Disagreements were resolved by consensus or in consultation with a third investigator (FN).
Outcomes describing journal data-sharing policies
Our primary outcome was the existence of a data-sharing policy (specific data-sharing policy, no data-sharing policy or a policy merely referring to ICMJE requirements) as reported on the journal website. This outcome had to be changed from our initial protocol due to non-response to our emails from the sample of ICMJE-affiliated journals and because some email addresses could not be identified. The change took place before any analysis. For journals mentioning a specific data-sharing policy on their website, 'the explicit statement and various features of these policies were collected: the start date of the data-sharing policy, the type of policy: ICMJE compliant, more stringent than required by ICMJE or less demanding than required by ICMJE (for instance, less demanding could mean that there was no obligation for a data-sharing statement, and more stringent could mean that data were to be shared with other researchers). We also noted whether the policy was limited to clinical trials, and furthermore the indication of one or more preferred data-sharing platform (and if so, which ones) and the existence of any sanctions in case of non-compliance with data-sharing (and if so, what they were). Any existing policy demanding trial registration was also extracted (and if there was one, we noted whether it mentioned prospective registration). The following features of the journals were also extracted: indexed on PubMed, International Standard Serial Number (ISSN or print ISSN), number of issues per year, 2017 journal impact factor (JIF), publisher or publishing group, gender of the editor in chief (‘men’, ‘women’ and ‘both genders represented’, if the coeditors in chief were men and women), country of the journal head office, wealth category of the country where the editorial office is located as defined by the World Bank11 and the research domain covered by the journal.
Survey of RCTs published in journals with a data-sharing policy
RCT eligibility criteria
Eligible studies were RCTs published after 1 January 2019 in a journal with an explicit data-sharing policy reported on its website and submitted after 1 July 2018. Any RCTs, including cluster trials and cross-over trials, non-inferiority designs and superiority designs, were included. No distinction was made in terms of patients, interventions, comparators or outcomes. We had originally planned to include only phase III studies but realised that this information was not always reported in the publications. Consequently, no distinction in terms of study phase was applied.
Search strategy for RCTs published in journals with a data-sharing policy
ICMJE-member journals were contacted to gather the list of RCTs they published after 1 January 2019. This approach was not used for ICMJE-affiliated journals due to non-response from most of the 14 member journals and because it was not possible to identify an email contact for all these journals. The following search strategy was applied to retrieve all RCTs. For journals indexed on PubMed/Medline, a search algorithm to identify RCTs was developed with the help of a librarian from Rennes 1 University using the Cochrane sensitivity maximising approach12 and adding further keywords. The exact filter can be found in the supplementary material on the OSF page.10 For journals not indexed on PubMed, an investigator (MS) screened all articles published after 1 January 2019 to identify RCTs.
All identified RCTs published in ICMJE-member journals with a data-sharing policy were assessed in random order using R.10 The first 100 that met our selection criteria were included, enabling us to estimate a proportion of 50% (the worst-case scenario for precision estimates) with a precision (boundaries of the 95% CI) of ±9.8%. We followed the same approach to include a second sample of 100 RCTs published in ICMJE-affiliated journals.
RCT selection and data extraction
As for the journal selection procedure, a data extraction sheet and a data extraction explanatory document were developed.10 Three investigators (MS, LC, JFG) had a 1-hour explanation and were trained via a pilot data extraction performed on 10 RCTs. For each RCT, two of these investigators independently assessed eligibility (giving reasons in case of non-eligibility) and extracted the characteristics listed in the following sections from each included published article. Disagreements were resolved by consensus or in consultation with a third investigator (FN).
Outcomes describing data-sharing statements in published RCTs
For this part of our survey, the primary outcome was the intention to share individual patient data (IPD) expressed by the authors in the data-sharing statement (yes/no/unclear). The latter of the three response options, ‘unclear’, was recorded if the statement was written in a general tone without specifically mentioning that IPD would be available. Secondary outcomes were trial registrations: the existence of trial registration, prospective trial registration and registration of a data-sharing plan. If the trial report mentioned the existence of a data-sharing plan, we checked whether there was an intention to share data or not. For data-sharing under the secondary outcomes, we checked whether a statement was included in the article, whether the statistical code was shared, whether other data sets than IPD were available and if not whether only parts were available, and lastly whether any other documents were available. Under the section data accessibility, we checked to see whether there was a time restriction for data access, whether it was freely accessible or with restrictions, whether the data could be used for any type of purpose and if not whether there was an aim for data use suggested in the proposal, whether there was a specific access mechanism, and whether data requests were reviewed by an independent committee.
Statistical analysis
All outcomes were reported and described by counts, percentages, means (or medians) and SD (or range) with all the corresponding 95% CIs. If available, verbatim quotes from journal policies were used (qualitative analysis). For ICMJE-affiliated journals, the features of the included journals were compared as part of an exploratory analysis between journals with and without a specific data-sharing policy using univariate logistic regression and multivariate logistic regression (which included covariates identified in univariate analyses at a threshold of p<0.25). Due to complete separation observed in our data set, the ‘brglm’ package in R was used, implementing the bias reduction method developed by Firth.13 All analyses were conducted using R V.3.4.1. The supplementary material on the OSF page contains our statistical analysis plan.10
Changes to the initial protocol
The definition of our primary outcome for journal policies was changed. Indeed, we had initially planned to contact journals with no explicit policy on their website to ask them about the existence of a data-sharing policy. Due to non-response from some of the 14 member journals, and because it was not possible to identify an email contact for all ICMJE-affiliated journals, we decided to rely only on the information presented on the journal websites. Some minor changes were also made. Our selection criteria were simplified so that only one RCT in 2018 was necessary (instead of three over the last 3 years as initially planned). No distinction was made for RCTs in terms of clinical phase. Lastly, we added a secondary outcome, whether or not data requests were reviewed by an independent committee.
Patient and public involvement
We had no established contacts with specific patient groups in this project. No patients were involved in defining the research question or the outcome measures, nor were they involved in the design and implementation of the study. There are no plans to involve patients in the dissemination of results, nor will we disseminate results directly to patients.
Results
Survey of journal data-sharing policies
Journal selection and data extraction
Search for and extraction of eligible journals started on 1 February 2019, ended with a consensus on 11 July 2019, and resulted in 14 ICMJE-member journals and 4892 ICMJE-affiliated journals. Of the affiliated journals, 2367 were randomly screened and 1878 (79%) were excluded, including 745 journals (31% of all screened journals) for which the journal and/or the publisher were listed as ‘predatory’ on Beall’s list. Therefore, 489 ICMJE-affiliated journals were included in analyses as initially planned. The selection process is reported in figure 1.
Flow chart of the selection and analysis process for journals and articles. ICMJE, International Committee of Medical Journal Editors; RCT, randomised controlled trial.
ICMJE-member journals
The characteristics of the 14 ICMJE-member journals are detailed in table 1.
Characteristics of journal policies for ICMJE-member and ICMJE-affiliated journals
Twelve of 14 (86%) journals published an RCT in 2018. The New Zealand Medical Journal and the Ethiopian Journal of Health Sciences did not publish an RCT in 2018. Eight (57%) journals had a specific data-sharing policy on their website: three (38%) of these had a more stringent policy than required by the ICMJE (IPD to be available for The BMJ and PLOS Medicine, or explicit demands of data for peer review for the Annals of Internal Medicine), four were ICMJE-compliant (50%; NEJM, Danish Medical Journal, Journal of the American Medical Association and The Lancet), and one (12%; Journal of Korean Medical Science) had a less demanding policy than required by the ICMJE that did not require a data-sharing statement but merely encouraged it. Of 14 medical journals, 5 (35%; Bulletin of the World Health Organization, German Medical Journal, Ethiopian Journal of Health Sciences, Iranian Journal of Medical Sciences and Medical Journal of Chile) referred to the ICMJE guidelines and 1 (8%; The New Zealand Medical Journal) did not have any policy mentioned on its website (its editorial office said that they had no time to clarify this point with us). Only three journals had a data-sharing policy before 2017. The earliest was presented in 2007 by the Annals of Internal Medicine, followed by PLOS Medicine and The BMJ in 2014 and 2015, respectively.
Three journals (out of 14, 21%; The BMJ, PLOS Medicine and The Lancet) indicated specific data-sharing platforms in their policy: Dryad and Mendeley.
For the eight journals with specific data-sharing statements, five referred specifically to clinical trial data, and for the three others it was for all research data submitted.
Sanctions were described in two journals, PLOS Medicine and the Annals of Internal Medicine: possible rejection of the manuscript if the data were not provided.
Except for the Ethiopian Journal of Health Sciences and the Iranian Journal of Medical Sciences, all journals had their editorial office in high-income countries.
ICMJE-affiliated journals
The characteristics of the 489 ICMJE-affiliated journals are also presented in table 1. Of these journals, 145 (30%; 95% CI 26% to 34%) had a specific data-sharing policy on their website. Two hundred and seventy-six journals (56%; 52% to 61%) merely referred to the ICMJE guidelines, without any specific mention of a data-sharing policy. Sixty-eight (14%; 11% to 17%) had no data-sharing policy and did not allude to the ICMJE in their recommendations. In contrast, 178 (37%; 32% to 41%) required prospective trial registration, 142 (29%; 25% to 33%) asked for trial registration without specifications, 114 (23%; 20% to 27%) referred to the ICMJE, and 55 (11%; 9% to 14%) did not refer to any trial registration.
Among the 145 journals with a specific data-sharing policy, 11 (7%; 4% to 13%) had a more stringent policy than that required by the ICMJE, 49 (34%; 26% to 42%) journals were ICMJE-compliant, and 85 (59%; 50% to 67%) had a less demanding policy than required by the ICMJE that did not explicitly require a data-sharing statement. Nineteen (out of 145; 13%) journals with data-sharing policies referred only to clinical trial data, while for the rest the statement comprised a more general statement. Out of 145 journals, 94 (65%) had no start date found for the policy, 25 (17%) had a policy starting in early 2018 (January and February), and 26 (18%) had a policy starting on 1 July 2018.
One hundred and one (out of 145; 70%) journals indicated a preferred data-sharing platform, with Mendeley (81 journals), Figshare (79 journals) and Dryad (67 journals) being the three most often cited.
Except for the gender of the editor, all features explored in univariate analyses were associated (p<0.25) with the explicit mention of a data-sharing policy on the journal website and were therefore used in the multivariate analyses. Publisher and wealth category of country of journal offices remained associated with the explicit mention of a data-sharing policy in multivariate analysis. The respective adjusted ORs can be found in table 2.
Journal characteristics associated with an explicit data-sharing policy
Survey of RCTs published in journals with a data-sharing policy
RCT selection and data extraction
Search for and extraction of eligible RCTs started on 6 August 2019 and ended with a consensus on 26 September 2019. Among the 12 eligible ICMJE-member journals, the New Zealand Journal of Medicine and the Ethiopian Journal of Health Sciences did not present any RCT in 2018. PLOS Medicine and the Bulletin of the World Health Organization provided a list of published articles. Two hundred and ninety-seven RCTs published in member journals were found.10 We could only confirm for 20 articles that they had been submitted after 1 July 2018. For six articles without data-sharing statements in the NEJM and The Lancet, we were not sure if they were eligible with respect to the submission date. Authors were contacted and we were able to confirm for two journals that they had been submitted before 1 July 2019. These were replaced, as were the four others where doubt persisted. Among the affiliated journals 722 RCTs were identified and were randomly sorted and assessed for eligibility criteria. Figure 1 details the selection process for both ICMJE-member and ICMJE-affiliated journals.
RCTs published in ICMJE-member journals
The results are displayed in table 3 for the 100 selected articles. Among these, 30 were from NEJM, 28 from The Lancet, 17 from JAMA, 13 from PLOS Medicine, 5 from the Annals of Internal Medicine, 3 from The BMJ, 2 from the Journal of Korean Medical Sciences and 2 from the German Medical Journal.
Characteristics of all published randomised controlled trials included
Almost all the articles (98%; 92% to 99%) had a data-sharing statement. The two articles without data-sharing statements were from the Journal of Korean Medical Science and were confirmed as having been submitted after 1 July 2018. Of the statements 67% (57% to 76%) indicated an intention to share data, while the intention was unclear for an additional 10 (10%; 5% to 18%). The characteristics of the data-sharing plans are detailed in table 4.
Characteristics of the data-sharing statements for articles with an intention to share individual patient data (including those with unclear intentions)
Of the 77 articles with data-sharing intentions, 7 (9%) mentioned access to other data from the study, besides IPD (eg, data frame for ‘unpublished data’/‘medical coding dataset’/‘non-patient-level data’). Of the 77 articles, 63 (82%) mentioned sharing for the following supplementary documents: study protocol (for 51), statistical analysis plan (for 37), informed consent form (for 15), data dictionary (for 14) and case report form (for 5). Time restriction for IPD was present in 34 of 77 (44%) data sets for either the start date of data-sharing, the end date or both. In two data-sharing statements, it was clear that data were to be available directly after approval of the drug in the European Union and in the USA. In 28 out of 30 other cases there was an embargo: 8 after 2 years, 2 after 18 months, 15 after 1 year, 1 after 9 months, and 2 after 3 months. A restricted access period was specified for 12 data sets: six of these specified restricted access for 2 years, three for 1 year, one for 5 years, one for 10 years, and for one it was stated that the time would be defined by the committee. Of 77 data-sharing statements, 60 (78%) specified that data could only be used for specific reasons: 53 mentioned a scientific aim only, 6 indicated willingness to share data specifically for meta-analyses or individual meta-analyses, and 1 data-sharing statement specified that the aim of the reuse was to be focused on a particular disease (herpes zoster). A specific mechanism was detailed in 68 of 77 (88%) data-sharing statements. Thirty-five only mentioned the need to establish a data-sharing agreement and/or a formal data request, 20 indicated that an email contact was necessary, and 13 mentioned data-sharing platforms.
Twenty-two of 100 (22%) had registered a data-sharing plan on registers such as ClinicalTrials.gov. Of these, 14 specified IPD data-sharing, 6 did not, and for 2 it was unclear.
RCTs published in ICMJE-affiliated journals
The 100 selected RCTs were from 38 different journals (mean number of RCTs per journal=11 (±10)). We found 25 RCTs with data-sharing statements. Seventeen authors/teams (17%; 10% to 26%) declared an intention to share, while the intention was unclear for an additional five (5%; 2% to 12%). The characteristics of the data-sharing plans are detailed in tables 3 and 4. Seven of 22 (32%) articles with a positive (or unclear) intention to share data expressed in a data-sharing statement indicated that other data sets, besides IPD, would be available. Regarding the sharing of any other documents, authors stated they would share study protocols (for seven studies), the statistical analysis plan and the study report (for four studies), and the case report form (for two studies). Time restriction was present for 10 out of 22 (46%) data sets. Three data sets had a limitation for the start date of data-sharing, ranging from 12 months to 18 months and up to 3 years. For the end date of data availability, the following time frames were collected: 5 years in two cases, 3 years for one, 2 years for one, 1 year for three and 3 months for one. For the question as to whether data could be used for any type of purpose, 14 out of 22 (64%) eligible data sets were only available for specific purposes (ie, research). For 10 of these cases the scientific aim was mentioned but not detailed, and in four statements no aim was specified at all. A specific kind of access mechanism was cited in 21 of 22 (96%) statements. Six of them mentioned a data-sharing agreement, one referred to a data platform, and fourteen data sets could be requested by email. For the 13 out of 100 (13%) trials with registration of their data-sharing plan, 2 planned to share the data, 6 did not, and for 5 it was unclear.
Discussion
Statement of principal findings
In our survey we found that 57% of ICMJE-member journals had an explicit data-sharing policy on their website and that approximately a third of the ICMJE-affiliated journals had one. Slightly more than a third of the member journals and most of the affiliated journals (around 56%) referred to the ICMJE guidelines without specifying a specific data-sharing policy. In addition, nearly 60% of the affiliated journals with an explicit policy had a less demanding policy than that required by the ICMJE. In contrast, the former ICMJE policy of trial registration was better implemented, with more than 71% of member journals and 66% of the affiliated journals explicitly requiring it as part of their policies.
For journals with a data-sharing policy, a data-sharing statement was frequent among member journals (98%), with rates of intention to share data of around 77%. These rates are in line with the intention to share previously reported in the Annals of Internal Medicine.14 In contrast, among ICMJE-affiliated journals with a data-sharing policy, data-sharing statements were not frequent (25%), and the intention to share data was only found in 22% of RCTs published in journals with an explicit data-sharing policy. Importantly, the statements often refer to data-sharing on request, and rarely to a specific repository or to fully available data sets. We already know that, even under a strict data-sharing policy such as the policy in place at The BMJ and PLOS Medicine, data availability is suboptimal, even when researchers express an intention to share.15 And indeed, in a recent scoping review16 we found that while the willingness to share data was generally high across trials, actual data-sharing rates were generally lower. In addition, there was considerable heterogeneity in data-sharing statements, with a focus on IPD data, and with very inconsistent information related to statistical codes and other documents (eg, the study protocol or the study report), which are key elements for reproducible research.17 Our results therefore question whether the new ICMJE policy as implemented by journals adequately supports clinical trial data-sharing, and they underline the need for efforts towards more reproducible research. Although data-sharing is only one aspect among others (eg, registration/best practices in reporting), without data-sharing, reproducibility is not possible.
Few characteristics were found to be associated with an explicit data-sharing policy. All were related to the publishers and the World Bank wealth category of the country/region of the editorial offices. As observed in previous research,18 a positive association between the JIF and data-sharing was found in univariate analysis, but it did not survive in the multivariate analysis. While the JIF is often (incorrectly) thought to be a surrogate for journal quality, our study suggests that professionalism and characteristics of the publisher and the editorial office resources could be better markers of quality and the implementation of reproducible research policies.
Findings in relation to other studies
A similar survey conducted in 2019 by our team also identified lack of implementation of basic data-sharing instructions in surgery journals with a JIF over 2. Only 50% of the journals had a data-sharing policy on their website,19 and in general these policies were not as demanding as those required by the ICMJE.
Furthermore, research done by Dal-Ré and Marušić20 found independently an alike number of predatory journals in the list of journals which claim to follow the ICMJE recommendations.
It is important to note here that almost all the ‘big’ publishing houses have different data-sharing policies for their different journals (eg, the BMJ Group has three levels of data-sharing policy, and Taylor & Francis have five different types).21 22 A related analysis was conducted by Mellor23 on the various data-sharing policies of the four big publishing houses, Elsevier, Springer Nature, Taylor & Francis and Wiley. This survey compared the Transparency and Openness (TOP) guidelines with the data-sharing policies of the different journals.24 Similar definitions to the ones we used to define more or less stringent requirements were adopted, and the authors found that most of the basic or level 1 data-sharing policies were not even TOP-compliant. This confirms our impression that even if policies are in place, they are not sufficiently demanding to be liable to change the data-sharing culture. A harmonisation needs to be established, and previous successful experiences should be taken into account.25 26
Limitations of this study
In our study, we had to rely on online information, as it proved difficult to contact editors and ascertain the existence of data-sharing policies. While all included journals had an electronic format, we cannot exclude that some may have implemented a policy without mentioning its enforcement explicitly on its website. In addition, we are studying a moving target in a changing environment, and it is likely that some journals that had no explicit policy when we performed our search have now implemented one. Repeated monitoring of the implementation of data-sharing policies therefore seems necessary. Another limitation was that we did not check specifically whether the data sets were actually made available when the authors indicated availability in the statement, nor did we request any data to ascertain data availability. Data availability rates could indeed be lower than suggested in an intention to share, as observed in The BMJ overall,27 and more particularly for clinical trial data, even after communicating with the study authors.15 Moreover, it would be interesting to see how many funders or academic institutions really share their data after expiry of the time restrictions indicated.
A further limitation was the language filter. Due to lack of resources we were not able to include every language. This might have caused bias, as for instance Russian journals might have presented a different data-sharing policy from journals that publish in English.
A large range of journals were included, especially in terms of quality. We tried to limit the inclusion of ‘predatory journals’ using Beall’s list. In this matter there is no real gold standard, as no exact definition existed when we planned our study. Other lists such as Cabell’s blacklist show an overlap with Beall’s list.28 Recently, a new definition has been proposed29 and it could help to better identify predatory journals. On the one hand, we were surprised by the large number of ICMJE-affiliated journals referenced on the ICMJE website and listed in Beall’s list. On the other hand, the investigators performing the data extraction had the impression that some of the selected journals had very poor editorial standards (in cases where instructions for authors were not clear and information was not given for all the steps of the editorial process) and could also fit the definition of predatory journals.
Finally, our identification of factors associated with an explicit data-sharing policy is only exploratory. Several unmeasured confounders, for instance the journal’s income and/or the numbers of RCTs published by a given journal, could account for some of the associations found. Other unmeasured confounders may exist and great caution is warranted in interpreting these results, which naturally cannot be considered as reflecting any causal relationship.
Perspectives
It appears that data-sharing policies are infrequent and poorly enforced in most ICMJE-affiliated journals. Perhaps the journals do not know how best to implement the policy, or they may be worried they will lose submissions if the policies are implemented. Other explanations could be the costs resulting from the process or the greater labour intensity. It is also possible that some authors, researchers and indeed editors may be opposed to data-sharing policies. In addition, there is no specific enforcement for an affiliated journal to follow the ICMJE guidelines. It can be noted that the ICMJE states on its website that they ‘cannot verify the completeness or accuracy of this list’ and that ‘there may be some listed journals that do not follow all of the many recommendations and policies in the document’.7 Furthermore, the large proportion of presumed predatory journals we found as well as the small proportion of journals enforcing the new policy are of concern for the impact and credibility of the ICMJE. We suggest that journals provide audits and feedback (to readers), especially as the number of ICMJE-affiliated journals is growing very fast, with 4725 in November 201830 and already 5504 in November 20197 (+16% in 1 year). Without such checks, journals with poor editorial practices could present affiliation with the committee as an endorsement of a sort of quality label in biomedical journals, while this is not the case. The ICMJE affiliation could be indeed perceived as a guarantee, since these standards have of course been endorsed by more than three-quarters of the most prominent journals in biomedicine, as illustrated by Shamseer and colleagues in 2016.31
In addition, continuous audits of journal policies and their enforcement could be used as a better indicator of journal quality than the current exclusive focus on the JIF. There is room for development of new responsible metrics in this area, encompassing other aspects of reproducible research practices, such as registration policies and the use of reporting guidelines.32 And indeed, data-sharing is only one facet of reproducible research policies.
Steps in the right direction have already been taken, such as the uniform guidelines for data-sharing in journals that have been developed by the Data Policy Standardisation and Implementation Interest Group of the Research Data Alliance.33 This could help to reach the goal of full transparency and data-sharing for clinical trial results, since the implementation of the current ICMJE policy seems suboptimal.
Acknowledgments
We would like to thank Chloé Rousseau from CHU Rennes for her statistical advice and Clémence Belvèze from the university library Rennes 1 for her help with the search algorithm.
References
Footnotes
Twitter @ReiTheR_RCT, @dmoher
Correction notice Size of the figure 1 has been increased.
Contributors MS, JFG, AD, DM and FN conceived and designed the experiments. MS, JFG, LC and HG performed the experiments. MS and FN analysed the data. MS and FN interpreted the results. MS wrote the first draft of the manuscript. DM and FN contributed to the writing of the manuscript. MS, JFG, LC, HG, AD, DM and FN agreed with the results and conclusions of the manuscript. All authors have read them and confirm that they meet the ICMJE criteria for authorship. All authors had full access to all of the data (including statistical reports and tables) in the study and can take responsibility for the integrity of the data and the accuracy of the data analysis. MS is the guarantor.
Funding FN received funding from the French National Research Agency for the ReiTheR (Reproducibility in Therapeutic Research) project, which this article was part of (project number: ANR-17-CE36-0010).
Competing interests FN, MS and JFG received funding from the French National Research Agency for the submitted work.
Patient consent for publication Not required.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement Data are available in a public, open access repository. All data and all materials are available on the OSF website: https://osf.io/n6whd/.