Objective To estimate the financial costs paid by individual medical researchers from meeting the article processing charges (APCs) levied by open access journals in 2019.
Design Cross-sectional analysis.
Data sources Scopus was used to generate two random samples of researchers, the first with a senior author article indexed in the ‘Medicine’ subject area (general researchers) and the second with an article published in the ten highest-impact factor general clinical medicine journals (high-impact researchers) in 2019. For each researcher, Scopus was used to identify all first and senior author original research or review articles published in 2019. Data were obtained from Scopus, institutional profiles, Journal Citation Reports, publisher databases, the Directory of Open Access Journals, and individual journal websites.
Main outcome measures Median APCs paid by general and high-impact researchers for all first and senior author research and review articles published in 2019.
Results There were 241 general and 246 high-impact researchers identified as eligible for our study. In 2019, the general and high-impact researchers published a total of 914 (median 2, IQR 1–5) and 1471 (4, 2–8) first or senior author research or review articles, respectively. 42% (384/914) of the articles from the general researchers and 29% (428/1471) of the articles from the high-impact medical researchers were published in fully open access journals. The median total APCs paid by general researchers in 2019 was US$191 (US$0–US$2500) and the median total paid by high-impact researchers was US$2900 (US$0–US$5465); the maximum paid by a single researcher in total APCs was US$30115 and US$34676, respectively.
Conclusions Medical researchers in 2019 were found to have paid between US$0 and US$34676 in total APCs. As journals with APCs become more common, it is important to continue to evaluate the potential cost to researchers, especially on individuals who may not have the funding or institutional resources to cover these costs.
- health policy
- health economics
- medical journalism
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
This cross-sectional analysis estimated the financial costs paid by a large (n=487) randomly selected sample of individual medical researchers from meeting the article processing charges (APCs) levied by open access journals in 2019.
This analysis used a large number of sources to identify author and journal data, including Scopus, author institutional profiles, Journal Citation Reports, publisher databases on APCs, the Directory of Open Access Journals and individual journal websites.
Secondary and sensitivity analyses were conducted considering author (gender, affiliation, region and training), journal, and APC-related characteristics.
Without access to the financial records from the index researchers and journals in our sample, our estimates do not represent the actual APCs that the index researchers in our sample paid.
Publications in peer-reviewed journals are currency in the academic world, and are often viewed as a proxy for productivity, competency and prestige. With over 15 million publishing scientists across the world,1 2 the pressure to publish has only risen, as has the importance of publications for employment, promotion and tenure.3 Over the past decade, there has been a striking growth in the number of scientific articles published per year, with nearly 2.5 million scientific articles published in 2018 alone.4 The sheer quantity of scientific research being published, the shift to predominantly electronic publishing and a broad movement to make scientific research more transparent has wrought a dramatic change in the landscape of scientific publishing.5 6
Currently, the primary mechanism for the publication of scientific articles is through peer-reviewed journals. For the most part, these journals have operated using a subscription model, generally owned and managed by a professional society or a medical publisher. Under this model, the cost to individual researchers, either to access articles or publish their own research in the journals, is minimal (although they contribute substantial in-kind effort through peer and editorial review). Instead, institutions pay subscription fees, which can reach millions of dollars for larger publishers, to gain access to articles for the institution’s affiliates.7 8
However, over the past 20 years, a new model of scientific publishing emerged in parallel with a rise in digital ‘publication’ rather than print distribution—open access publishing.5 Generally, open access journals forgo subscriptions for their online content and instead make research available to scholars without institutional subscriptions and to the general public. With no revenue from subscriptions, some open access journals established a new business model built primarily around article processing charges (APCs).5 Compared with the subscription journals, the APC model has shifted part of the financial burden of publishing from academic institutions to individual researchers and their funders, who are responsible for APCs that average US$2000 (£1568) to US$3000 (£2352) per article.9 10
With almost 5000 open access journals following the APC business model,11 researchers are increasingly having to consider if and how they can afford to publish their research in open access journals with the limited pool of funds available. While the vast majority of medical researchers are supportive of the concept of open access publishing, over half listed financial barriers as the primary reason they would choose not to publish in open access journals.12 Although APCs can be covered with funds from research grants or by funders directly, not all research is grant funded, the structure and amount of funding that comes from grants can vary by field, and the ability or willingness of funders to cover APCs differs by region.10 13 Additionally, early career researchers or under-represented minority researchers may have more limited access to grant funding or institutional funds to cover APCs, as do researchers in less lucrative clinical fields like primary care and public health.14–16 While fee waivers are sometimes granted to researchers in low-income and middle-income countries or without funding, discounted APCs may still be prohibitive for many researchers.17 18
If financial barriers play such a substantial role in scientists’ decisions on where to publish, it is important to investigate the potential financial costs of publishing on individual medical researchers. Therefore, we aimed to estimate how much individual medical researchers spend on APCs over the course of a year for both a general sample of medical researchers as well a sample of researchers who published in the highest impact factor clinical medicine journals in 2019.
Study design and sample
We conducted a cross-sectional analysis of two random samples of medical researchers to obtain estimates of the average amount of money that individual researchers spend on APCs each year. We used Scopus to identify 250 general medical researchers and 250 medical researchers who published at least one article in one of the ten highest impact factor general clinical medicine journals.
Generating a sample of general and high-impact researchers
First, we downloaded the first 20000 English language research or review articles published in a journal indexed in the Scopus subject area of ‘Medicine’ in 2019 (figure 1), the maximum data export permitted through the Scopus portal. Next, we used Scopus to identify all research or review articles published in the top ten highest impact factor clinical medicine journals in 2019 (according to the Journal Citation Report (JCR)19: New England Journal of Medicine (NEJM), Lancet, Journal of the American Medical Association (JAMA), British Medical Journal, JAMA Internal Medicine, Annals of Internal Medicine, PLOS Medicine, BMC Medicine, Mayo Clinic Proceedings and Canadian Medical Association Journal. For each sample, we used a random number generator to select 500 articles. However, given the broad nature of the search used, some articles randomly selected did not fall under the category of ‘general clinical medicine,’ or were other article types misclassified as research or review articles. Therefore, the first 250 articles determined to be eligible were retained for each sample (i.e., 250 general medicine articles and 250 high-impact medicine articles).
Next, we identified the senior-most (i.e., last) author of each research or review article contained in the sample (hereafter, index researcher). Potential duplicate researchers were verified and removed through a Scopus and/or Google Scholar search of the researcher’s name. If authorship was listed as a group, without any designated individuals, the manuscript was excluded from the sample. If a group authorship was listed as the senior author, the senior-most individual author on the article was used.
Three investigators (MKE, XS and JJS) independently abstracted data, and to ensure data quality, approximately 20% of each sample was abstracted in duplicate to verify consistency. All uncertainties were discussed with a fourth investigator (JDW). All data abstraction and validation were conducted between 22 April and 22 July 2020.
For the index researchers in both samples, we used the researcher’s Scopus profile to collect the researcher’s name, affiliation, geographical region (based on the six WHO regions),20 H-index, year of first publication and research field. Research field was collected from the subject area tags listed on the researcher Scopus profile. For each index researcher, we conducted a Google search of the researcher’s name and screened the first 10 pages to identify an institutional researcher profile. If a researcher profile was available, we also abstracted researcher gender, if clearly indicated in the profile or through identification of gender pronouns, and training (a doctor of medicine (MD), with or without other degrees; a doctor of philosophy (PhD), with or without other degrees (excluding MD); or any other degrees).
Identification of first and senior author publications in 2019
Using the Scopus profile of each index researcher, we identified all of the articles published in 2019 where the index researcher was listed as either the first or senior author. Articles on which the index researcher was a middle author (no matter if 3 authors were listed or 20) were excluded, as we assumed the index researcher would not have paid any associated APC as a middle author. For each article, we abstracted the corresponding journal’s title and determined whether the article was marked as open access in Scopus.21
Journal characteristics and APCs
We used JCR to determine the 2018 journal impact factor for each unique journal. Next, we identified the journal publishing model (open access, hybrid or subscription based) and the APCs for each journal. A hybrid journal was defined as a traditional subscription-based journal with a fee-based open access publication option.22 To ascertain APCs, we first used publisher-specific databases,23–37 which provide lists of open access and hybrid journals from selected publishers and their associated APCs. If a journal could not be identified through a publisher database, we used the Directory of Open Access Journals (DOAJ)11 to identify if the journal was open access. If listed on DOAJ, the journal was considered open access and the corresponding APC was collected from the provided link to the journal website on DOAJ. If the publishing model of a journal could not be determined from those two sources, we relied on the information provided on individual journal websites. Journals without a clear open access policy (either full open access or a hybrid approach) were considered subscription-based.
We defined the standard APC for an open access journal as the fee associated with publishing in that journal. For hybrid journals, we defined the APC as the fee associated with optional open access publication. We did not include additional fees not associated with open access publication, such as charges for colour printing or reprints, as part of the APCs. If an APC for a given journal was based on word count or page limits, we approximated the standard APC using an average article (3500 words) or page count (8 pages).38 In addition to the standard APC, we collected the minimum APC for any journal with multiple APC options. The minimum APC was defined as the lowest APC a researcher could pay given any discounts publicly listed by the journal on the journal website or in the publishing database or different licensing options (e.g., institutional or author membership discounts or commercial vs non-commercial licenses).
Using descriptive statistics, we characterised the sample of both the general researchers and high-impact researchers, including gender, affiliation, training, geographical region and seniority (based on H-Index and length of the researcher’s career). Length of the researcher’s career was approximated by subtracting the year of the index researcher’s first publication from 2020.
Next, we calculated the median (IQR) APCs paid by index researchers in 2019 for both groups. To do this, for each index researcher, we calculated the maximum total APCs paid in 2019 by assuming that an APC was paid by the index researcher if they were the first or senior author of an article. If an index researcher’s article was published in an open access journal with an APC listed, we assumed that the APC was paid without any discounts or waivers. For any of the index researcher’s articles published in either hybrid or subscription-based journals, we assumed no APC was paid. Lastly, we also calculated the proportion of articles published in open access journals.
We used the Mann-Whitney U test or Mood’s test as appropriate to compare median APCs paid per index researcher by the characteristics noted above. For comparisons of APCs paid by H-Index and length of the researcher’s career, we compared researchers above and below the median H-Index and across quartiles in each sample, respectively. Any unknown values were considered as missing. US dollar amounts were converted to British pound sterling using the average exchange rate for 2019.39 All data analyses were conducted in R (version 3.6.1; The R Project for Statistical Computing) and used a threshold for statistical significance of 0.05.
We repeated the analyses above assuming that index researchers paid: (1) the minimum publicly listed APCs, (2) the APCs for articles published in hybrid journals and classified as ‘Open Access’ on Scopus, (3) APCs for only their first author articles, and (4) APCs for only their senior author articles.
Patient and public involvement
This study was an analysis of publicly available, non-clinical data. There was no patient or public involvement in any of the phases of the study, although we expect the public to be broadly supportive of open access publishing since it permits access to information and does not require expensive subscriptions through medical libraries.
After accounting for duplicate index researchers and non-English publications, our sample included 241 general and 246 high-impact researchers. Among the 241 general researchers, 239 (99.2%) had ‘Medicine’ listed as one of their subject areas on Scopus; all 246 high-impact researchers had ‘Medicine’ listed as one of their subject areas.
Nearly all of the general researchers were affiliated with academic centres or hospitals (236/241, 97.9%); 62 (25.7%) were based in the Americas, 76 (31.5%) in Europe and 69 (28.6%) in the Western Pacific region (table 1). An institutional profile could not be identified for approximately one-quarter of the researchers (62/241, 26.5%). Among the 179 researchers with an institutional profile, two-thirds had an MD (120/179, 67.0%) and 70 (70/179, 39.1%) were women. On average, general researchers had published at least two (median: 2, IQR 0–4) first or senior author articles in 2019, had an H-Index of 11 (median: 11.0, IQR 3.0–23.0), and had at least a 15-year publication history.
The vast majority of high-impact researchers were affiliated with academic centres or hospitals (213/246, 87.6%); 85.8% (211/246) were primarily based in the Americas or Europe (table 1). An institutional profile was identified for almost all (241/246, 98.0%) of the high-impact researchers. The majority of those with an institutional profile (166/241, 68.9%) had an MD and one-third (82/241, 34.0%) of the researchers were women. High-impact researchers had, on average, a publication history of greater than 20 years, an H-Index of 38.5 (median: 38.5, IQR 22.0–64.0), and had published at least four first or senior author manuscripts in 2019 (median: 4, IQR 2–8).
In 2019, the 241 general researchers published 914 first or senior author research or review articles in 598 unique journals. The most common journals were Medicine (15/914, 1.6%) and the International Journal of Molecular Sciences (9/914, 1.0%). Among the 462 journals with a 2018 JCR impact factor, the median impact factor among their articles was 2.65 (IQR 1.69–3.86). The 246 high-impact researchers published 1471 original first or senior author research or review articles in 604 unique journals. The most common journals were NEJM (60/1471, 4.1%), The Lancet (43/1471, 2.9%) and PLOS Medicine (36/1471, 2.5%). Among the 537 journals with a 2018 JCR impact factor, the median impact factor was 5.05 (IQR 3.18–11.05).
Of the 914 first or senior author research or review articles published by the general researchers, 414 (414/914, 45.3%) were indexed in Scopus as open access. There were 384 (41.6%) articles published in an APC-based journal (table 2). Of the 457 (457/914, 50.7%) articles published in journals with a hybrid funding model, 72 (72/457, 15.8%) were indexed as open access. Among the high-impact researchers, 726 (726/1471, 49.4%) of the articles were indexed in Scopus as open access. Just under one-third of all articles were published in an APC-based journal (426/1471, 28.9%). Among the 870 (870/1471, 59.1%) articles published in journals with a hybrid funding model, less than one-third (255/870, 29.3%) were open access.
Article processing charges
The journal funding model and any associated APCs could be identified for 94.1% (860/914) of the first or senior research or review articles published by the general researchers and 97.8% (1439/1471) of the articles published by the high-impact researchers. In 2019, the 241 general and 246 high-impact researchers paid an estimated total of US$497716 (£390209) and US$1067869 (£837209) in APCs, respectively, for their first and senior author articles. Although the median APCs paid by general clinical medical researchers was US$191 (IQR US$0–US$2500) (£150, £0–£1960), one researcher was estimated as having paid US$30115 (£23610) in APCs (table 3). The median total APCs per researcher in the high-impact sample was US$2900 (IQR US$0–US$5465) (£2274, £0–£4285); one researcher was estimated as having paid as much as US$34676 (£27186) in APCs.
In sensitivity analyses, after including potential discounts on standard APCs, the minimum listed APCs general researchers could have paid for their first and senior author publications in 2019 was US$0 (IQR: US$0–US$2500) (£0, £0–£1960) (table 3). However, researchers in the high-impact sample would have paid approximately US$300 less on average (median: US$2600, IQR US$0–US$5465) (£2038, £0–£4285). If all researchers paid the APCs for their first and senior open access published in hybrid journals (as opposed to the articles being made available through delayed open access due to funder requirements, at the discretion of the journal, or through other mechanisms such as self-archiving) the median total APCs paid by the general and high-impact researchers would have been US$739 (IQR US$0–US$3950) (£579, £0–£3097) and US$5000 (IQR US$0–US$10879) (£3920, £0–£8529), respectively.
The estimated median total APCs paid did not vary across index researcher gender, training, H-index and years since first publication (table 4). However, high-impact researchers in the Region of the Americas did have lower median total APCs per researcher than those in other regions of the world (Region of the Americas: US$1695, IQR US$0–US$3935 (£1329, £0–£3085) vs Other regions: US$4800, IQR US$1888–US$8290 (£3763, £1480–£6500); p<0.001).
In this cross-sectional study, 241 and 246 randomly selected general and high-impact medical researchers published a median of 2 and 4 first or senior author research or review articles in 2019, respectively. Approximately one-third of the articles across both samples were published in journals that required an APC. The median total APCs per general and high-impact researcher in 2019 was US$191 (£150) and US$2900 (£2274), respectively, with one researcher who may have incurred as much as US$34676 in APCs (£27186). Across both samples, there were no meaningful differences in APCs paid by gender, affiliation or training. However, in the high-impact sample, researchers from the Region of the Americas had a lower median total APCs paid (US$1695) (£1329) than researchers from all other regions (US$4800) (£3763). As open access publishing with APCs becomes increasingly common, it is important to consider the financial implications for individual researchers across different fields, settings and levels of seniority.
Our study suggests that many general and high-impact researchers could have paid thousands of dollars in APCs to publish their first and senior research and review articles in 2019. Across the 487 index researchers in both samples, which represents only a fraction of all biomedical researchers actively publishing in 2019, the total estimated APCs was approximately US$1500000 (£1 176 000). Given that general researchers published a median of 2 first or senior articles per year (potentially in lower impact factor journals with smaller APCs)40 41 it may not be surprising that the median total APCs per researcher was relatively low (US$191) (£150). However, among high-impact researchers, who published a median of 4 first or senior research or review articles in 2019, the median total APCs per researcher was US$2900 (£2274). This suggests that these researchers paid an APC for one of every four of their first or senior author articles. Moreover, if we extrapolate our findings, individual researchers could spend a total of US$116000 (£90 944) on publication costs over a 40-year career.
It is important to note that there are numerous benefits to open access publishing. Limiting the amount of science that exists behind a paywall can have clear advantages for individual researchers and the public.42 43 Open access publishing can enhance equity by improving the ability of researchers, either working in low-resource settings or at institutions that cannot support the hefty cost of journal subscriptions, to access publications.44 45 Articles published open access can receive a citation boost compared with those behind paywalls, a boon for researchers looking to increase the audience and impact of their work.46 Furthermore, APCs often serve an important purpose in the publication process. APCs can be used to pay the salaries of journal editors, who are often responsible for screening a large number of manuscript submissions, identifying and soliciting appropriate peer-reviewers (and performing their own peer-review), and helping improve the quality of studies as they transition from submission to eventual publication. Moreover, APCs could be used to pay peer reviewers for their efforts—a service currently provided by researchers for free even in cases where researchers are paying thousands of dollars to publish an article.47 However, if APCs continue to increase, questions will continue to be raised about journals’ potential profit motives, predatory journals and hybrid journals that receive payments from both institutions and researchers.
However, the rise of the APC-centred open access publishing model poses a number of challenges for researchers.18 48 Approximately one-third of the first or senior research and review articles published by the general and high-impact researchers were published in an open access journal that required an APC. Although not all open access journals charge APCs, approximately 50% of all articles that are published open access are published in journals that do.42 When grant money or institutional discretionary funds are used to cover APCs, as is the case for approximately 80% researchers in the health, biological and life sciences,10 fewer resources are available for other research-related expenses.48 For instance, the US$2900 (£2274) median amount spent by researchers in our high-impact sample could support the attendance of multiple individuals at a conference or a critical piece of research equipment. Moreover, for researchers spending tens of thousands of dollars a year on APCs, these funds could have covered the tuition of a graduate student or the partial salary of a postdoctoral fellow. Second, the amount of APCs has risen dramatically in recent years—at a rate nearly three times that of the expected inflation rate.13 49 These increases have raised questions about whether APCs actually reflect the cost of publishing or if publishers are driven by primarily financial motives.9 48 While there does not appear to be a quality difference between subscription-based and open access journals,5 50 there is some evidence that journals with higher APCs are perceived to be higher impact.41 50
Lastly, the amount of APCs can enhance existing inequities in publishing by creating an additional barrier to many researchers based on field,10 seniority,14 disparities in research funding15 16 or setting.18 48 51 For instance, evidence suggests that researchers from countries with gross domestic products (GDPs) lower than US$25000 (£19600) are more likely to pay APCs out of personal funds compared with researchers from countries with GDPs higher than US$25000 (£19600).10 It is important to note that certain journals grant fee waivers to researchers from low-income and middle-income countries or to researchers without funding to support publication. However, many researchers may be unaware of the specific journals that do provide waivers.17 Furthermore, journal waivers do not necessarily address all of the inequities imposed by APCs. For early career researchers with no established grant funding or accumulated discretionary funds, even discounted APCs can be beyond available resources.
As open access publishing becomes the norm, numerous opportunities exist to address the disadvantages that may prevent many researchers from paying for APCs. At the journal level, increased transparency may be necessary to inform researchers from low-income and middle-income countries or at early stages of their careers about the waivers that are available. It is also critical that funders and institutions leverage their influence to restrain the hyperinflation of APCs. In 2018, cOAlition S, an international consortium of research funders, launched ‘Plan S’. This initiative, which aims to make all scientific publications resulting from publicly funded research immediately available open access,52 has proposed an APC fee cap.49 52 As more scientific research is available open access, institutions can shift resources from subscriptions to a pool of funds to support the expenses for early career researchers. Among universities in the UK, there is an ongoing commitment to promoting open access publishing by encouraging submission to open access repositories and by assisting researchers in the payment of APCs for immediate open access publication.13 At the funder level, more agencies could embrace the Gates Foundation or the Charity Open Access Fund model used by the Wellcome Trust, where researchers supported by these funders can request coverage of any associated APCs.53 54 Individual researchers can also increasingly choose to release their research open access through venues such as pre-print servers, like medRxiv, without undermining their ability to publish their findings in peer-reviewed journals.55 Furthermore, so-called ‘Green Open Access’ policies, where researchers can elect to post peer-reviewed papers in open access repositories, are available for many journals, although most researchers do not use this option.44 56 57 Scientific publishing is changing and it will be necessary for all stakeholders to adapt.
Limitations of this study
This study is subject to certain limitations. First, we recognise the limitations of classifying authors as ‘general’ or ‘high-impact’ based on one senior author research or review article published in one of the 10 highest impact factor medical journals. Second, our estimates do not represent the actual APCs that the index researchers in our sample paid. Without access to the financial records from the index researchers and journals in our sample, we had to make several assumptions about the nature of APC payments, most fundamentally that it was the index author who paid the APCs, rather than a funder or other organisation. In particular, articles for which the index researcher was a middle author were excluded, as we assumed index researchers are less likely to pay associated APCs as a middle author. We also did not account for situations in which APCs may have been paid by coprimary or cosenior authors. Additionally, we used the most recent APCs listed on journal websites, which may not represent the APCs paid in 2019. For our primary analysis, we assumed that researchers in our sample did not pay the optional APCs for open access publications in hybrid journals. Using publicly available information, it is difficult to determine if open access publications in hybrid journals were paid for by researchers or were available open access due to funder requirements or journal discretion. Furthermore, we did not account for any unlisted discounts or fee waivers provided by journals to researcher institutions in our analyses. Although the true minimum APCs per researcher may be lower than our estimate, our results did not change substantially when analyses were repeated using the lowest APCs listed by journals (excluding waivers). Overall, our sensitivity analyses provide a range of what researchers are likely to have paid.
Third, although Scopus provides a comprehensive accounting of a given researcher’s publication history, not all manuscripts published by a researcher may be indexed on Scopus. Furthermore, Scopus may create multiple researcher profiles for the same researcher, due to changing institutions or different permutations of the researcher’s name. However, we attempted to identify and include all researcher profiles for each index researcher. Second, we relied on articles classified as ‘articles’ or ‘reviews’ on Scopus. Although this method allowed us to objectively screen and classify index researcher articles, it is possible that we may have included or excluded articles that were incorrectly classified by Scopus. Lastly, due to the cross-sectional design of our study, we are unable to establish causal relationships between author characteristics (e.g., region) and potential APCs paid.
This cross-sectional analysis suggests that clinical medical researchers could have paid as much as US$34676 (£27186) in total APCs for their first and senior author research and review articles in 2019. Although the total APCs in this study are estimates, it is important to understand the potential cost of open-access publishing to researchers as journals with APCs become more common. In particular, future studies should evaluate the impact of APCs on individuals who may not have the funding or institutional resources to cover these costs.
Contributors JDW and JSR first conceived the study idea when arguing about who would have to pay the APC for one of their previous manuscripts. MKE, KN, JSR, and JDW designed this study. MKE, XS and JJS acquired the author, journal, and APC data. MKE conducted the statistical analysis. MKE, JSR and JDW drafted the manuscript. MKE, XS, JJS, KN, RL, JSR and JDW participated in the interpretation of the data and critically revised the manuscript for important intellectual content. MKE and JDW had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. MKE and JDW are guarantors. JDW provided supervision, and despite being the senior author, begged JSR to pay the APCs.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: In the past 36 months, XS received a scholarship from China Scholarship Council. JSR is a former Associate Editor of JAMA Internal Medicine, a current Research Editor at BMJ, and has received research support through Yale from Johnson & Johnson to develop methods of clinical trial data sharing, from the FDA to establish a Center for Excellence in Regulatory Science and Innovation (CERSI) at Yale University and the Mayo Clinic (U01FD005938), from the Medical Device Innovation Consortium as part of the National Evaluation System for Health Technology (NEST), from the Agency for Healthcare Research and Quality (R01HS022882), from the National Heart, Lung and Blood Institute of the National Institutes of Health (NIH) (R01HS025164, R01HL144644), and from the Laura and John Arnold Foundation. JDW received research support through the Collaboration for Research Integrity and Transparency from the Laura and John Arnold Foundation and through the Center for Excellence in Regulatory Science and Innovation (CERSI) at Yale University and the Mayo Clinic (U01FD005938).
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Patient consent for publication Not required.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement The dataset will be made available via a publicly accessible repository on publication. https://osf.io/f6cwu/
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.