Article Text
Abstract
Objectives The objective of this study was to explore whether reducing the material supplied to external experts during peer review and decreasing the burden of response would maintain review quality into prioritising research questions for a major research funder.
Methods and analysis Clinical experts who agreed to review documents outlining research for potential commissioning were screened for eligibility and randomised in a factorial design to two types of review materials (long document versus short document) and response modes (structured review form versus free text email response). Previous and current members of the funder’s programme groups were excluded. Response quality was assessed by use of a four-point scoring tool and analysed by intention to treat.
Results 554 consecutive experts were screened for eligibility and 460 were randomised (232 and 228 to long document or short document, respectively; 230 each to structured response or free text). 356 participants provided reviews, 90 did not respond and 14 were excluded after randomisation as not eligible.
The pooled mean quality score was 2.4 (SD=0.95). The short document scored 0.037 (Cohen’s d=0.039) extra quality points over the long document arm, and the structured response scored 0.335 (Cohen’s d=0.353) over free text. The allocation did not appear to have any effect on the experts' willingness to engage with the task.
Conclusions Neither providing a short or a long document outlining suggested research was shown to be superior. However, providing a structured form to guide the expert response provided more useful information than allowing free text. The funder should continue to use a structured form to gather responses. It would be acceptable to provide shorter documents to reviewers, if there were reasons to do so.
Trial registration number ANZCTR12614000167662.
- peer review
- health technology assessment
- clinicians
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Footnotes
Twitter Andrew Cook: @ajcook
Contributors AC designed the study, analysed and interpreted the data, drafted the article and approved the final version for publication. ES designed the study, collected the study data, interpreted the data, critically reviewed the article and approved the final version for publication. GD collected the study data, critically reviewed the article and approved the final version for publication.
Disclaimer The views and opinions expressed herein are those of the authors.
Competing interests All authors are employed by the University of Southampton to contribute to the National Institute for Health Research (NIHR). Their continuing employment may to some extent depend on the continued funding of NIHR.
Ethics approval University of Southampton Faculty of Medicine Ethics Committee #8192.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement Anonymised data may be requested from the corresponding author.