Article Text
PDF
Abstract
Objective Many journals permit authors to submit supplementary material for publication alongside the article. We explore the value, use and role of this material in biomedical journal articles from the perspectives of authors, peer reviewers and readers.
Design and setting We conducted online surveys (November–December 2016) of corresponding authors and peer reviewers at 17 BMJ Publishing Group journals in a range of specialities.
Participants Participants were asked to respond to one of three surveys: as authors, peer reviewers or readers.
Results We received 2872/20340 (14%) responses: authors 819/6892 (12%), peer reviewers 1142/6682 (17%) and readers 911/6766 (14%). Most authors submitted (711/819, 87%) and 80% (724/911) of readers reported reading supplementary material with their last article, while 95% (1086/1142) of reviewers reported seeing these materials sometimes. Additional data tables were the most common supplementary material reported (authors: 74%; reviewers: 89%; readers: 67%). A majority in each group indicated additional tables were most useful to readers (61%–77%); 20%–36% and 3%–4% indicated they were most useful to peer reviewers and journal editors, respectively. Checklists and reporting guidelines showed the opposite: higher proportions of each group regarded these as most useful to journal editors. All three groups favoured the publication of additional tables and figures on the journal’s website (80%–83%), with <4% of each group responding that these do not need to be available. Approximately one-fifth (16%–23%) responded that raw study data should be available on the journal’s website, while 24%–33% said that these materials should not be made available anywhere.
Conclusions Authors, peer reviewers and readers agree that supplementary materials are useful. Supplementary tables and figures were favoured over reporting checklists or raw data for reading but not for study replication. Journals should consider the roles, resource costs and strategic placement of supplementary materials to ensure optimal usage and minimise waste.
Trial registration number NCT02961036.
- supplementary materials
- survey
- peer review
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
Our large sample from a diverse group of active international authors and reviewers from 17 different journals provide evidence for stakeholder views on supplementary materials within the biomedical literature.
The response rate is comparable to response rates for other electronic surveys of researchers.
Participants were asked to respond in the assigned role/perspective of a reader, peer reviewer or author, although these are not mutually exclusive categories, as academics often engage in all three activities.
Background
Many journals allow or require authors to submit supplementary material along with their manuscript. These materials might help in deciding about the publication of the article (such as completed checklists for reporting guidelines) or provide additional information for readers who wish to delve deeper into the findings, replicate the research or use it for secondary analysis. The materials might also help improve access in the context of initiatives such as the Findability, Accessibility, Interoperability and Reusability Data Principles for the automatic finding and use of scientific data,1 and the wish to facilitate automation in the systematic review process.2
The volume of supplementary materials is accelerating in step with research complexity and multidisciplinary alliances. For example, Schriger et al show the percentage of articles containing supplementary materials increasing from 7% in 2003 to 25% in 2009 with web-only supplementary materials doubling in the same time period.3 Scientific journals report challenges in keeping up, citing reviewer fatigue, publishing delays, bloated publishing repositories and confusion, as it is not unusual for articles that occupy 5–7 pages in the journal to present with over 140 pages of supplementary data or for systematic reviews or trial reports to include several hundred pages of information that would be needed to replicate, but not to report the findings of the research.4–7 Supplementary materials might provide additional results from a study or the detail needed to replicate the methods or present formulas, statistical models, intervention details or algorithms. Some journals refuse the materials as excessive, while others allow ‘reasonable use’ which each journal defines individually.4–7 This is set within the backdrop of an increasing demand for research transparency through the sharing of all findings and corresponding data.8 Although standards for supplementary materials were suggested in 2012 by the National Information Standards Organisation and the National Federation of Advanced Information Services,9 the concerns of medical journals were not specifically considered and any policy adopted by medical journal editors will have implications for readers, editors, reviewers and the general public.
Clinicians and researchers struggle to keep up with reading the literature. Nearly a decade ago, Bastian et al reported the publication of 75 trials and 11 systematic reviews per day and asked ‘how will we ever keep up?’ 10 The numbers have continued to increase since then and the challenges have been compounded by the burgeoning supplementary material and problems with incompatible file systems, bandwidth restrictions and broken weblinks.11 The increasing volume of supplementary materials submitted to journals puts more pressure on journal editors and peer reviewers to retrieve relevant information from multiple sources.7 Schaffer and Jackson12 make recommendations on how access to supplementary material can be improved. There is concern that the excessive volume of supplementary materials can influence decisions made during peer review and skew the integrity of the scientific record.6 A recent study of research manuscripts submitted to JAMA, JAMA Internal Medicine and JAMA Pediatrics found that manuscripts with supplements were more likely to be peer reviewed and accepted than those without supplements.13 The requirements and practices of journals around supplementary materials vary12 14 and journals’ expectations of peer reviewers in terms of supplementary material are often not made clear in guidance to reviewers.6 For example, some journals explicitly state that supplementary material will not be peer reviewed, while others only mention that it will not be typeset. This variety of approaches forces authors, reviewers and readers to place different degrees of prioritisation and importance on supplementary material when including, reading or using them.
The use of supplementary materials during and after submission and publication is patchy, and the perceived value to stakeholders of the work involved in producing, assessing and using them is unclear.13 15 We conducted a survey to help resolve these uncertainties and to investigate the role of supplementary material in biomedical journal articles from the perspective of authors, peer reviewers and readers.
Methods
This survey is registered at ClinicalTrials.gov.
Sampling
Journal sampling
Participants were drawn from a sample of 17 of BMJ Publishing Group’s biomedical journals (online supplementary appendix 1). Journals varied in size and impact factor but each has a website and publishes supplementary material.
Supplementary file 1
Participant sampling
One author (SS) downloaded contact details of all corresponding authors who submitted a full-length original research submission to one of the 17 journals in 2013 and all peer reviewers who had completed a review of a research submission for one of the journals in 2014 from the journal manuscript tracking systems. She used Microsoft Excel to remove duplicates from within each journal subsample and then across author/reviewer samples for all journals based on the person’s email address. We sent each sampled email address an invitation to just one of the three surveys, but it is possible some duplicates remained if an individual had more than one email address in the manuscript tracking systems. We excluded potential participants if they had previously opted out of receiving BMJ communications or had participated in a BMJ research survey within the previous 6 months.
Two-thirds of the authors were randomly assigned to receive the Author Survey, two-thirds of the peer reviewers were randomly assigned to receive the Reviewer Survey and one-third of each sample was randomised to receive the Reader Survey, under the assumption that all participants were likely to be readers of journal articles.
Questionnaire administration
The surveys were developed by the researchers and piloted with 45 volunteers to check for ambiguous questions. The surveys were revised based on this feedback before launching.
Participants were sent an email invitation in November 2016 to complete an online survey administered using SurveyMonkey. Non-respondents were sent up to two reminders. Participants were asked to complete the survey from the perspective of their allocated role to provide information about their use of specific types of supplementary material (study protocol, data collection or extraction forms, data tables and figures, completed reporting guideline checklists and flow diagrams, interview transcripts and raw study data). Survey questions asked who the material is most useful to; the expected use of materials by authors, reviewers and readers; the preferred option for accessing supplementary material and if and where supplementary material should be published. The questions and response categories for each of the survey instruments are shown in online supplementary appendices 2–4.
Statistical analysis
Data were exported into Excel, cleaned and anonymised prior to analysis. All statistical analyses were conducted in SPSS V.22. Descriptive and summary statistics of interval scale variables were calculated using mean and standard deviation (or median and IQR for skewed data), and categorical data as frequency and percentages. Data have been reported from the individual perspectives of authors, readers and reviewers, as well as the aggregated overall perspective.
Public and patient involvement
Forty-five volunteers piloted the surveys and shared valuable feedback to make the questions clear and unambiguous. These volunteers were community members, physicians, researchers, patients and teachers.
Results
Online supplementary appendix 5 shows which questions in the surveys pertain to our findings presented below and in the tables and appendices.
Supplementary file 5
Respondent characteristics
We sent the survey by email to 20 340 people and received 2872 (14%) responses (819 (12%) from authors, 1142 (17%) from peer reviewers and 911 (14%) from those responding as readers); see table 1. The numbers of years as an active researcher were comparable across respondents with a mean of 4.4 years (SD 1.96) for authors, 4.6 years (SD 1.98) for readers and 5.3 years (SD 2.89) for reviewers. The approximate number of research papers reported as published by respondents was a median of 46 overall (36 for authors, 41 for readers, 51 for reviewers, which are statistically different across the groups: independent samples Kruskal-Wallis test p<0.001) but with a spread of experience (IQR: 81 research papers). More than 87% of respondents read articles in medical journals either frequently or very frequently. Respondents are from an international sample, with authors from 65 countries, reviewers from 57 and readers from 53 countries.
Characteristics of respondents
Respondent’s interaction with supplementary material
When recalling what supplementary material was contained in their last article submitted, authors stated including additional tables of data (74%) or additional figures (57%) most frequently, followed by checklists for relevant reporting guidelines (39%). Readers recalled reading additional tables of data (67%) or additional figures (53%), followed by study protocol (23%). Over 80% of reviewers recalled the use of additional figures and tables of data in articles they peer reviewed sometimes or often, in contrast to more than 80% reporting rarely seeing raw study data or interview transcripts (online supplementary appendix 6).
Preferred option for accessing supplementary material
Overall (n=2872) respondents’ preferred option for accessing tables of data and additional figures was as supplementary files alongside the article (60% and 59%, respectively), while 50% chose this as their preferred option for data collection forms and completed checklists for relevant reporting guidelines. In contrast, 40% of respondents preferred that interview transcripts and raw study data would not be made available (see figure 1 for overall data and online supplementary appendix 7 for responses by group).
Overall views of preferred option for providing/reading/receiving supplementary material (n=2872).
The open-text responses to accessing supplementary materials also showed common sentiment across readers, reviewers and authors; as illustrated by this quote ‘It depends on the type of research and my purpose for accessing it. If I am only reading for enjoyment or for an overview of the topic I seldom look at supplementary materials but to replicate the research or to further verify the authors findings or methods, the supplementary materials provide nuances the paper does not.’
Who the material is most useful to
Figure 2 shows the overall views of who each type of supplementary material is most useful to, from the total of 2872 respondents. Additional tables of data and additional figures were deemed to be most useful to readers (>65%), while the study protocol and data collection/extraction forms were deemed most useful to peer reviewers (>40%), in contrast to the completed checklists which were deemed most relevant to journal editors (40%).
Overall views on who each type of supplementary material are most useful to (n=2872).
Table 2 (and online supplementary appendix 8) further stratifies these opinions by allocated group, which reveals similar trends to those given overall. For instance, additional tables of data were regarded as most useful to readers (58%–72%) by all groups (authors, reviewers and readers), while checklists were perceived as more useful to journal editors or peer reviewers rather than readers (36%–45% versus 12%–16%).
Author, reviewer and reader perspectives on the value of additional tables of data, completed checklists for reporting guidelines and raw study data by group*†
If and where supplementary material should be published
Figure 3 depicts the overall views on where (each type of) supplementary material should be published, be this on the website alongside the article, on another website, available directly from the authors, or that it does not need to be available. The responses are not mutually exclusive, but more than 81% preferred to see additional tables of data and figures on a website along with the article. In contrast, respondents preferred interview transcripts (37%) and raw study data (39%) to be available by contacting the article’s corresponding author, with a further 30% and 27% respondents indicating these materials did not need to be made available, respectively. Other forms of supplementary material, for example, checklists, were perceived variably with responses of either availability on the website along with the article (45%) or of no need to be available (23%). Online supplementary appendix 9 shows that the responses were similar by group.
Overall views on where supplementary material should be published (n=2872).
In the open-text responses, there were multiple requests for inclusion and publication of replicable software codes, dynamic models with the modelling results, statistical models, videos and models for imaging and genetics while others saw no need for supplementary materials stating that the responsibility of the authors was to deliver clear and concise reporting that would fit within the given word limits of a paper. An important consideration noted by some respondents was that some data were restricted and could not be shared without compromising the identities of participants particularly in data linkage sets. Respondents stressed the need for improved navigation both of the website to access the materials and of the materials themselves in terms of labelling, ordering and readability. It was suggested that supplementary materials for an article should be downloadable as a single zipped file.
Expected use of materials by authors, reviewers and readers
Almost half the authors who responded expect that peer reviewers should routinely read all supplementary material. But on asking reviewers what they do with supplementary material, 8%–16% ignored completed checklists, flow diagrams, interview transcripts and raw study data, with 11%–26% saying it depended on the manuscript. We found that only additional tables of data and additional figures were being routinely read entirely, at approximately 60%, with other categories below 36%. In response to the question about what they usually do with supplementary materials, no more than 27% of readers responded that they routinely read all of any type of supplementary material, with 30%–40% ignoring completed checklists, flow diagrams, interview transcripts and raw study data (see online supplementary appendix 10–12).
Discussion
In general, authors, reviewers and readers expressed a preference for supplementary material that provided additional tables over completed reporting checklists or raw data when reading research articles. This may highlight a greater desire among these users of research to have access to information that has been analysed or summarised by the original researchers. A recurring theme in free-text comments was how the importance and value of supplementary materials depended on the purpose for which they were accessed. For example, respondents noted that as interested readers they might not access any supplementary materials but that they would want to be able to access supplementary materials for analysis, replication, secondary research or teaching purposes. The respondents also expressed concerns about data accessibility, security and the persistence of all data, as well as concerns about protecting the trustworthiness and viability of permissions for raw data (particularly when made available to third parties). Considering these findings, our survey adds impetus to calls to improve the quality of reporting, the use of reporting guidelines15–17 and the evaluation of the impact of initiatives intended to improve the quality of the literature and decisions based on it. The survey also revealed uncertainty about the use and placement of supplementary materials, as illustrated by the following representative open-text comment:
A manuscript to be published should be able to stand on its own. Journals are making a mistake by making article word counts shorter, then having supplementary material. If more data are needed to understand the study, they should be in the article
In 2009–2011, the journals Cell, The Journal of Neuroscience and Science announced that they would not allow authors to include supplemental material on submission or host supplemental material on their websites. Instead, authors were given the option of including a URL to direct readers to the supplementary material on a website maintained by the authors, along with a short description of the supplementary material.4 5 18 However, we found little support from our respondents for including a weblink within the published paper or for requesting supplementary material directly from investigators by email. Although journals and researchers may feel a social responsibility to make data publicly and permanently available,18 they often lack the necessary tools or collaborators to build and maintain persistent repositories. Private web pages and email are not persistent over time and may be vulnerable to corruption. Hofner and colleagues recommend the use of recognised repositories where digital object identifiers are supplied as good practice for data preservation and to preserve the options to replicate the findings.19 There is considerable debate over how to make research more transparent and reproducible.20 As supplementary material often contains content that helps make research more reproducible, it is important for it to be accessible in the long term to help improve research efficiency. Others argue that the supplementary material needs to be better structured to avoid computational errors and to enable machine reading, particularly in the fields of genomics, neuroscience, chemistry and other basic sciences.21 Pop and Salzberg proposed that specific sections of the supplementary material should be directly hyperlinked within the text of the article to improve the utility of published scientific articles and to increase the likelihood that this material is adequately peer reviewed.6
Study limitations
Our response rate of 14% is typical of current response rates for electronic surveys to researchers,22 but still allowed us to achieve a large sample, with nearly 3000 responses from a diverse group of international authors and reviewers from 17 biomedical journals. As such, our findings make a substantial contribution to the evidence on stakeholder views on the value of supplementary materials within the peer-reviewed biomedical literature. Participants were asked to respond in the assigned role/perspective of a reader, peer reviewer or author, and these are not mutually exclusive categories, as academics often engage in all three activities. Participants gave general perceptions and were not asked to report on specific cases or the purpose of accessing the article and this may have influenced responses.
Remaining uncertainties and future research
Some respondents expressed a preference in open-text comments for standardised, well-organised supplementary materials that could be combined into a single zipped file for downloading or offered as a persistent link. However, others commented that data protection standards and ethical oversight might not be explicitly extended to making supplementary materials publicly available. These concerns were not directly addressed within the survey questions and so it is not known how representative or widespread these opinions might be. However, the views expressed could be the target of further investigation. It may also be worth investigating the relationship between the value of supplementary material and the cost of production and publication to researchers should journals take on the responsibility for the state of supplementary materials in terms of perpetual availability, typesetting and compatibility.
Conclusions
Our findings provide evidence that should help journals, researchers and funders to consider the roles, costs and benefits of supplementary materials. The findings highlight, for example, a greater desire among users of research to have access to information that has already been analysed or summarised by the original researchers, rather than their raw material. It may be helpful for journals to expand file types to allow storage of, and access to a variety of file types, including multimedia, computer models and working software prototypes. Our survey should also add impetus to calls to improve the quality of reporting and the use of reporting guidelines,15–17 and we hope that it will stimulate greater emphasis on the need for evaluation of the impact of all initiatives intended to improve the quality of health research and the decisions that will subsequently be based on this literature.
Supplementary file 2
Supplementary file 3
Supplementary file 4
Acknowledgments
The authors thank the 45 volunteers who piloted this research and all the researchers who completed the surveys and especially those who shared open-text comments. Their perspectives have increased our understanding.
References
Footnotes
Contributors AP, SS and MC designed the study and drafted the questionnaires. AP drafted the protocol with input from SS and MC. SS extracted the samples of authors and reviewers from the journals’ manuscript tracking systems and managed the surveys on SurveyMonkey. MC randomised participants to their allocated roles. HM analysed the anonymised data. All authors interpreted the results, wrote this manuscript and approved its final version.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests AP is the patient editor (Research and Evaluation) at the BMJ, and SS is a full-time employee of the BMJ. MC reports involvement in many clinical trials and systematic reviews and has prepared and used supplementary material widely. He seeks funding for these trials and reviews, as well as for research into methodology, including dissemination and accessibility. HM has no conflicts of interest.
Patient consent Not required.
Ethics approval The research was reviewed by, and received ethics clearance through, the University of Oxford Central University Research Ethics Committee (MS-IDREC-C1-2013-174).
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement The datasets used and analysed during the current study are available from the corresponding author on reasonable request.