Article Text
PDF
Abstract
Introduction Systematic reviews evaluating the impact of interventions to improve the quality of peer review for biomedical publications highlighted that interventions were limited and have little impact. This study aims to compare the accuracy of early career peer reviewers who use an innovative online tool to the usual peer reviewer process in evaluating the completeness of reporting and switched primary outcomes in completed reports.
Methods and analysis This is a cross-sectional study of individual two-arm parallel-group randomised controlled trials (RCTs) published in the BioMed Central series medical journals, BMJ, BMJ Open and Annals of Emergency Medicine and indexed with the publication type ‘Randomised Controlled Trial’. First, we will develop an online tool and training module based (a) on the Consolidated Standards of Reporting Trials (CONSORT) 2010 checklist and the Explanation and Elaboration document that would be dedicated to junior peer reviewers for assessing the completeness of reporting of key items and (b) the Centre for Evidence-Based Medicine Outcome Monitoring Project process used to identify switched outcomes in completed reports of the primary results of RCTs when initially submitted. Then, we will compare the performance of early career peer reviewers who use the online tool to the usual peer review process in identifying inadequate reporting and switched outcomes in completed reports of RCTs at initial journal submission. The primary outcome will be the mean number of items accurately classified per manuscript. The secondary outcomes will be the mean number of items accurately classified per manuscript for the CONSORT items and the sensitivity, specificity and likelihood ratio to detect the item as adequately reported and to identify a switch in outcomes. We aim to include 120 RCTs and 120 early career peer reviewers.
Ethics and dissemination The research protocol was approved by the ethics committee of the INSERM Institutional Review Board (21 January 2016). The study is based on voluntary participation and informed written consent.
Trial registration number NCT03119376.
- peer reviewers
- randomized controlled trials
- reporting
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Strengths and limitations of this study
To the best of our knowledge, this will be the first study that will assess the efficacy of using an innovative online tool to help early career peer reviewers to assess the quality of reporting of randomised controlled trials.
A randomised selection of trials from a large panel of biomedical journals.
The absence of background for choosing the a priori effect size.
Introduction
The peer review process is a cornerstone of biomedical research publication.1 The editorial peer review process is described as journal editors relying on the views of independent experts in making decisions on, for example, the publication of submitted manuscripts or presentation of reports at meetings.2 The peer review system is considered the best method for helping scientific editors decide on the acceptability of a manuscript for publication.3 Nevertheless, the effectiveness of the system has been questioned. In 2010, a report commissioned by the UK House of Commons showed that the cost to the UK Higher Education Institutions in terms of staff time was £110 to £165 million per year for peer review and up to £30 million per year for the work done by editors and editorial boards.4 Worldwide, peer review is estimated to cost £1.9 billion annually and accounts for about one-quarter of the overall costs of scholarly publishing and distribution.5 The human resources were estimated to be about 15 million hours by 2010.6
Furthermore, the peer review system may fail to identify important flaws in the quality of submitted manuscripts and published articles.7 8 A recent study by Hopewell et al 9 showed that peer reviewers failed to detect important deficiencies in the reporting of methods and results of randomised trials. There is also considerable indirect evidence of the failure of peer review. Systematic reviews of clinical questions highlight a high prevalence of key information not being provided in peer-reviewed articles. Moreover, there is evidence to suggest that peer reviewers are not able to detect fraud,10 mistakes11 12 or spin.13 14 Systematic reviews evaluating the impact of interventions to improve the quality of peer review for biomedical publications highlighted few randomised controlled trials (RCTs) and that interventions such as training have little or no impact.15 16
How the peer review process is currently organised can explain the difficulties encountered. A number of tasks are expected from peer reviewers when assessing reports of an RCT.17 They have to assess the completeness of reporting, whether outcomes were switched from that reported in the trial registry, the choice of the comparator and intervention, the relevance of outcomes, the risk of bias, the external validity, the presence of spin, the originality, importance and relevance of the RCT to the journal’s readers, etc. It may be unrealistic to expect a peer reviewer to complete all of these tasks.
Furthermore, the assessment of the completeness of reporting should be a prerequisite in the peer review process, because lack of transparency is a barrier to the assessment of important aspects of a trial such as the risk of bias18 and reproducibility.19 Another essential task of peer review is the identification of switched outcomes (ie, a discrepancy between the prespecified outcomes reported in the trial registry or protocol and outcomes reported in the manuscript). To avoid reporting bias(es),20 all prespecified outcomes should be reported in the manuscript, and authors should report any changes to trial outcomes after the trial started, with reasons.21 The Centre for Evidence-Based Medicine Outcome Monitoring Project (COMPare) assessed RCTs published in the five general medical journals with the highest impact factor. They compared the outcomes published to the outcomes recorded in trial registries or protocols. Among the 67 RCTs examined in July 2016 (www.COMPare-Trials.org), only nine had no discrepancies; 354 outcomes were not reported and 357 new outcomes were ‘silently’ added.18
Despite being essential, the assessment of the completeness of the reporting and the identification of switched outcomes are not appropriately performed.9 22 These tasks do not require a high level of expertise and could be performed by researchers early on in their career. Early career peer reviewers could use a simple online tool based on the CONSORT 2010 checklist and Elaboration and Explanation publication20 for reporting parallel-group RCTs developed according to the same principles used for the CobWeb tool.23
CobWeb is a writing aid tool based on CONSORT 2010. It was developed to provide guidance to authors when writing a manuscript of an RCT. The content of the tool was based on the CONSORT 2010 checklist and the Explanation and Elaboration document. For each domain, the tool comprised the corresponding CONSORT checklist item(s), bullet points with the key elements that need to be reported extracted from the Explanation and Elaboration document, as well as (an) example(s) of good reporting.
The peer reviewer tool will use the existing CobWeb elements reworded to be appropriate for peer reviewers and for the items not considered in CobWeb, we will extract the bullet points from the Explanation and Elaboration document of the CONSORT 2010 following the same process.21
The tool will feature bullet points eliciting the meaning of each checklist item. This tool will allow for standardising the completion of this task, be an opportunity to train early career peer reviewers to become more expert peer reviewers and be used to provide standardised and personalised feedback to authors based on the CONSORT items and bullet points; the authors would receive a request to report each item (with associated bullet points) that the early career peer reviewer identified as being inadequately reported.
The objectives of this project are as follows:
Develop an online tool and training module based (a) on the CONSORT 2010 checklist and the Explanation and Elaboration document modified for early career peer reviewers (ie, early stage researchers: masters students, PhD students, residents involved in clinical research during their study and clinicians involved in clinical research who never peer reviewed a manuscript) for assessing the completeness of reporting of 10 key items and (b) the COMPare process used to identify switched in primary outcomes in completed reports of RCTs.
Compare the performance of early career peer reviewers who use the peer reviewer tool with usual peer reviewers in identifying inadequate reporting and switched outcomes in completed reports of RCTs.
The planning, conduct, analysis and reporting of the study were discussed by the study’s scientific committee.
Methods and analysis
Study design
We approached the issue of peer reviewing as if it were a diagnostic test aimed at detecting incomplete reporting. Before evaluating this ‘diagnostic test’ in an RCT, we aimed to evaluate its accuracy in a cross-sectional study. For this purpose, we defined a gold standard (ie, assessment by systematic reviewers) and two diagnostic tests to be compared: usual peer reviewers and early career peer reviewers trained and using the online tool.
This is a cross-sectional diagnostic study comparing the accuracy of early career peer reviewers who use the tool to that of usual peer reviewers when evaluating the completeness of reporting and a switched in primary outcome(s) in completed reports of RCTs at the first submission (figure 1). All procedures in the study will be consistent with ethical practice. This study received ethical approval from the Institutional Review Board of the INSERM ethics committee (IRB 17–335). The protocol was written following the Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) guidelines.
The study design.
The tool: COBPeer
Principles of the tool
The online peer review tool provides guidance to early career peer reviewers when they are assessing the completeness of reporting of RCTs based on the relevant CONSORT 2010 checklist items, switched in primary outcomes from trial registry records based on the process used in COMPare and the associated CONSORT 2010 Elaboration and Explanation document.
Before using the tool, the early career peer reviewer will undergo an online training exercise consisting of evaluating extracts of manuscripts for each CONSORT item and corresponding trial registration records with specific feedback according to their answers.
The content of the tool and training will be tested and validated by the scientific committee of the study.
Completeness of the reporting of RCTs section
To assess the completeness of reporting of RCTs, the tool will be developed according to the same principles used for developing the CobWeb tool.23
We will focus on the CONSORT items that are least often reported (ie, reported in less than 50% of the reports assessed as shown in the study published by Hopewell et al 9) and are necessary to conduct a meta-analysis (ie, items of the risk of bias tool, outcomes data, safety data).
The selected items are:
five items in the methods section (outcomes (6a), randomisation/sequence generation (8a), allocation concealment mechanism (9), blinding (11 a, 11b));
four items in the results section (participant flow (13a, 13b), outcomes and estimation (17a), harms (19));
one item (23) dedicated to trial registration (reporting registration number)
We decided to focus on these items because they are essential for systematic reviewers when evaluating the risk of bias and recording the outcome data.
Of note, we relied on the CONSORT extension for harm24 to develop the bullet point for item 19 and on the SPIRIT guidelines explanatory document25 to develop the bullet point for item 6a.
An item will be considered adequately reported if all bullet points related to this item are indicated as being reported. For example, item 11 ‘Blinding’ will be considered adequately reported if the authors reported all the information below:
Who (ie, participants, healthcare providers, data collectors, outcome adjudicators, and data analysts) was blinded to treatment assignments.
How the blinding was performed (eg, used of placebo, intervention by physician unaware of the study).
The similarities of the characteristics of the interventions (eg, appearance, taste, smell, method of administration).
The content of the tool is available in table 1.
The most important items of the Consolidated Standards of Reporting Trials (CONSORT) 2010 statement and bullet points that early career peer reviewers will assess
Switched outcomes section
The switched outcomes section will be based on the process used in COMPare that was simplified. The early career peer reviewer will have to:
find the trial’s registry entry, by using http://google.com/;
copy the link to the online registry;
copy the date of the first registration;
copy the date of the beginning of the study;
copy the prespecified original primary outcome(s) from the registry;
compare the primary outcome(s) reported in the manuscript to the primary outcome(s) reported in the registry and determine whether there was a switch in outcome (ie, primary outcome added, deleted or changed);
indicate if authors justified the switched outcome(s) in the manuscript.
Overall the RCT manuscript will be classified as
no switch in outcome;
presence of switch in outcomes (at least one outcome was switched);
unable to assess.
Of note, the early career peer reviewer will not search for the protocol or assess the protocol as was done in COMPare because of time constraints.
Online training module
The online training module will start with a short reminder about the importance of the completeness of reporting and the assessment of switched outcomes from the trial register. For each item, the tool will provide bullet points of the item with explanations. Then, early career reviewers will assess passages from RCTs that are adequately or inadequately reported by using the tool. For each item assessed, participants will receive personalised feedback with a detailed explanation of the reported and missing bullet points.
Participants
Inclusion criteria
Early career peer reviewers are defined as early stage researchers: master students, PhD students, residents involved in clinical research during their study and clinicians who have never reviewed a manuscript.
Exclusion criteria
Participants will be excluded if they have already peer reviewed one RCT.
Setting and recruitment
We will use a large strategy of participants’ recruitment. Participants will be identified from:
editors of biomedical journals. We will contact specifically contact editors of journals interested in research in peer review field (eg, BMC Medicine, Trials, BMC Peer Review and Research Integrity);
learnt societies (eg, European Society of Emergency Medicine);
network of international students (eg, Students 4 Best Evidence);
University Paris Descartes and Paris Diderot students.
We will send a standardised email to their contacts for promoting our study. The email will invite people to participate in an online training course on peer review. They will be informed of the inclusion criteria and about the different steps of the study (to complete an online training module and then peer review one manuscript randomly selected from our sample). If they agree to take part, they will then log onto the system.
Blinding
Evaluators (systematic reviewers) will be blinded to the peer reviewers’ assessment, early career peer reviewers’ assessments and the content of the tool. They will have access to only one manuscript to assess. They will be instructed to base their assessment on only the content of the manuscript.
The evaluation of peer reviewers was blinded as performed before the study.
Early career peer reviewers will be blinded to the aim of the study, systematic reviewers’ assessments and peer reviewers’ comments. Systematic reviewers’ assessments will not be accessible. Peer reviewers’ comments in theory are accessible for articles published in BioMed Central series medical journals, BMJ and BMJ Open; however, the journal name, authors’ names and study title will be masked to early career peer reviewers.
Manuscript selection
We searched the US National Library of Medicine’s PubMed database to identify all primary reports of two-arm parallel-group RCTs with an individual randomisation unit, whatever the treatment evaluated, that were published in the BioMed Central series medical journals, BMJ, BMJ Open and Annals of Emergency Medicine and indexed with the publication type ‘Randomised Controlled Trial’. We included journals from BioMed Central series medical journals publishing at least 5 RCT reports per year between 1 January 2015 and 13 December 2016. We retrieved 1600 citations. One researcher screened all titles and abstracts and included all primary reports of RCTs assessing interventions in human participants who were randomly allocated to intervention groups. We excluded cluster RCTs, cross-over trials, equivalence and non-inferiority trials, feasibility studies, cost-effectiveness studies, phase I trials, study protocols, non-randomised controlled trials, secondary publications or analyses, pilot studies, systematic reviews, methodology studies and early phase studies. The screening process identified 222 eligible reports, among which we randomly selected 120 reports of RCTs. For each RCT, we retrieved the submitted manuscript and the first-round peer reviewers’ comments (available online or requested from editors/publishers). Citations for 17 RCTs with documents not available were replaced by 17 other randomly selected RCTs. Overall, we identified 120 manuscripts from 24 different journals.
Interventions
Gold standard
The gold standard will be the assessment of the completeness of reporting by systematic reviewers of the 10 CONSORT items for the first manuscript submitted, (ie, before any changes were made as part of the peer review process). This assessment corresponds to the assessment of the risk of bias (random sequence generation, allocation concealment, blinding of participants and personnel, blinding of outcome assessment, selective reporting of outcomes) and the extraction of efficacy and harm outcome data. Pairs of systematic reviewers will independently extract data from eligible reports; any differences between reviewers will be resolved by discussion, with the involvement of an arbitrator if necessary. Reviewers involved in the data extraction will have expertise in the conduct of systematic reviews and will assess the completeness of reporting from the systematic-reviewer perspective. They will not have access to the tool to avoid being influenced by the tool. Consequently, the gold standard will rate items as inadequately reported only if the reporting is a real barrier to the conduct of a systematic review. This situation will ensure that the assessment of the early career peer reviewer who use the tool is identifying inadequate reporting that is not relevant.
The systematic reviewers will also systematically compare the outcomes reported in the manuscript and the outcomes reported in the registry and will document any discrepancies.
Overall, for the first manuscript, before any changes were made as part of the peer review process, each item will be classified as adequately reported (yes/no) for the gold standard.
Online tool used by early career peer reviewers
If early career peer reviewers agree to participate, they will start the training module. If they successfully complete the training module, they will then download one randomly selected manuscript and assess at the first version submitted at the first round by using the online tool. They will be asked to perform this assessment in 30 min maximum.
There will be one early career peer reviewer per manuscript.
Usual peer reviewer
The usual peer review process assessment of the completeness of reporting and switched outcomes will be assessed from the peer reviewers’ reports. Two researchers will read all the peer review reports for the first round. The researchers will determine whether the peer reviewers raised some concerns about the completeness of reporting of the 10 CONSORT items considered and identified a switch in outcomes between the manuscript and the register. The assessment of all peer review reports for each manuscript will be combined (ie, the item will be rated as incompletely reported if at least one peer reviewer rated it as such). The two researchers will be blinded to the gold standard assessment.
Outcomes
Main outcome measure
The primary outcome will be the mean number of items accurately classified per manuscript initially submitted to the journal.
For the completeness of the reporting of RCTs section; 10 CONSORT items will be considered: Outcomes (6a), Randomisation/sequence generation (8a), Allocation concealment mechanism (9), Blinding (11 a, 11b), Participant flow (13a, 13b), Outcomes and estimation (17a), Harms (19) and one item (23) dedicated to trial registration.
Each item will be classified as ‘adequately reported’ (yes/no).
For the switched outcomes section, the manuscript will be classified as:
no switch in primary outcome;
presence of switch in primary outcomes (at least one outcome was switched);
unable to assess.
Secondary outcome measure
The secondary outcomes will be the mean number of items accurately classified per manuscript for the 10 CONSORT items and the sensitivity, specificity and likelihood ratio to detect the item as adequately reported and to identify switch in outcomes.
Because the gold standard may be less accurate than the tool used by early career peer reviewers in detecting inadequate reporting, we will systematically submit apparent false-positive results to experts identified by the scientific committee, who will indicate whether they believe the item is adequately reported (table 2).
Planned tabulations
Statistics
Sample size calculation
We will allow for detecting an effect size of 0.3 for the mean number of items accurately classified per manuscript with a power of 90% and a two-sided alpha-level of 5%.
Each manuscript will be assessed by one early career peer reviewer; each early career peer reviewer will assess one manuscript. So, we will randomly select 120 reports of RCTs to be assessed in the study. We do not have any data on the mean (SD) number of items accurately classified by usual peer reviewers. For each manuscript, we will have a single assessment related to the usual peer review process that will combine the first assessment of all peer reviewers for that manuscript.
Statistical analysis
The primary outcome will be the mean number of items accurately classified per manuscript by usual peer reviewers and by early career peer reviewers, the gold standard being the systematic reviewers’ assessment. We will compare the mean number of items accurately classified per manuscript by early career peer reviewers versus usual peer reviewers. This will be performed using a paired t-test. The sensitivity and specificity comparing the early career peer reviewers and the usual peer-reviewers’ assessment will be determined by a test for pair proportion.
Data management and monitoring
Data will be entered on at a secure, password-protected, website. As part of the data protection, confidentiality and anonymity requirements, all participants will be assigned a unique individual identifier. The database will be managed by the main investigators (CA and BI). Access will be limited to individuals deemed appropriate. No interim analyses will be performed.
Ethics and dissemination
Authorisation by the Commission Nationale de l’Informatique et des Libertés (file number 2021376) whose purpose is to protect participants’ personal data and the institutional review board of the INSERM ethics committee (IRB 17–335) (see Appendix 1 in the online supplementary file 1) was obtained and the study protocol was registered at ClinicalTrials.gov (http://clinicaltrials.gov, NCT03119376). Any changes or additions to the protocol will be registered on ClinicalTrials.gov. Participation will be voluntary. All participants will provide informed consent and may withdraw at any time (see Appendix 2 in the online supplementary file 1). Following their participation, they will receive one Continued Medical Education credit.
Supplementary file 1
The results will be presented at international congresses, published in a peer-reviewed journal and summarised on ClinicalTrials.gov. In addition, the findings will be disseminated via an open access website so that they can be used by the wider community. Study results will also be disseminated directly to participants.
Any changes or additions to the protocol will be registered on Clinicaltrials.gov. Authorship of papers arising from this study will be based on contributions to the study including intellectual content as recommended by the International Committee of Medical Journal Editors.
The data set will be the property of the scientific committee. The results will be presented at international congresses, published in a peer-reviewed journal and summarised on ClinicalTrials.gov
Discussion
Several researchers have highlighted that the quality of reporting of peer-reviewed and editor-approved clinical research is poor.26–28 However, because biomedical journals remains the best method for disseminating biomedical knowledge, it is fundamental to assess the implementation of novel interventions and must be assessed in different actors of the publication system to improve its efficiency and the quality of reporting. In this study, we limited our research to interventions to improve quality of reporting through peer review. To the best of our knowledge, this will be the first study of an innovative online tool to help early career peer reviewers to assess the quality of reporting of RCTs.
References
Footnotes
↵i This CONSORT item was slightly modified to focus only on the primary outcome.
Contributors AC and IB were responsible for the concept of the study and conceived and designed the study. DM, DGA, DS, SA, SH, DRS and PR helped to write the draft protocol and made a critical contribution to the content. All authors read and approved the final protocol.
Competing interests None declared.
Patient consent Obtained.
Ethics approval The Institutional Review Board of INSERM ethics committee (IRB 17-335).
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement All data will be published.