Article Text
PDF
Abstract
Introduction Many predatory journals fail to follow best publication practices. Studies assessing the impact of predatory journals have focused on how these articles are cited in reputable academic journals. However, it is possible that research from predatory journals is cited beyond the academic literature in policy documents and guidelines. Given that research used to inform public policy or government guidelines has the potential for widespread impact, we will examine whether predatory journals have penetrated public policy.
Methods and analysis This is a descriptive study with no hypothesis testing. Policy documents that cite work from the known predatory publisher OMICS will be downloaded from the Overton database. Overton collects policy documents from over 1200 sources worldwide. Policy documents will be evaluated to determine how the predatory journal article is used. We will also extract epidemiological details of the policy documents, including: who funded their development, the discipline the work is relevant to and the name of the organisations producing the policy. The record of scholarly citations of the identified predatory articles will also be examined. Findings will be reported with descriptive statistics using counts and percentages.
Ethics and dissemination No ethical approval was required for this study since it does not involve human or animal research. Study findings will be discussed at workshops on journalology and predatory publishing and will be disseminated through preprint, peer-reviewed literature and conference presentations.
- Health policy
- ETHICS (see Medical Ethics)
- Protocols & guidelines
- EPIDEMIOLOGY
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
We use Overton, a new database that is the most comprehensive source of international policy documents, to assess the penetration of predatory journals.
We employ rigorous study design and data extraction standards to ensure reliability of data.
Use of the single largest predatory publisher, OMICS, to determine penetration of predatory journals in policy literature may not be representative of broader predatory publishers/journals.
Introduction
Predatory journals and publishers are defined as ‘entities that prioritise self-interest at the expense of scholarship and are characterised by false or misleading information, deviation from best editorial and publication practices, a lack of transparency, and/or the use of aggressive and indiscriminate solicitation practices’.1 In contrast, legitimate journals generally publish according to a set of best practices.2 Predatory, or illegitimate journals, often do not perform the expected quality checks that legitimate journals conduct. For instance, predatory journals typically forego the peer-review process and publish low-quality work.3 Predatory journals may also accept research that has not received ethical approval.4 Additionally, these publications are increasingly finding their way into reputable databases,5–7 which is problematic.
Despite growing evidence for the negative impact of predatory journals, the extent to which these journals are being incorporated into the scholarly record through citation has only recently been examined.4 8 9 To date, this research has focused on the citation of articles published in predatory journals by authors publishing in legitimate ones.10–12 It is also possible that predatory journals are cited beyond the traditional academic literature in policy guidelines or guidance documents. In the context of healthcare, an individual practitioner reading a flawed article may have a relatively limited impact, but policy guidelines have the potential to influence clinical practice on a much larger scale. Flawed guidelines, like those historically encouraging widespread opioid prescribing, can stem from low-quality untenable and misleading research.13–16 These guidelines can cause serious systemic issues, such as for patient safety, resource use and economic burden.13–15
Pilot work from our group identified multiple policy documents or guidelines citing randomised controlled trials published in predatory journals; similarly, these findings raised the possibility that science published in predatory journals influences policy and guidelines more generally. Here we present a protocol to broadly evaluate the impact of predatory journals on policy and guidance documents.
Objective
The objectives of this study are to: (1) determine whether predatory journal articles are cited in policy and guidance documents; and (2) if so, describe how predatory journal articles are cited and used in these documents; and (3) assess whether or not correlations or patterns exist between how predatory journal articles are cited and used in these policy documents, and how these same predatory journal articles are cited and used in the traditional scientific literature.
Methods
This protocol will be registered on the Open Science Framework and all study materials and data will be shared openly. Our final report will use elements of the Strengthening the Reporting of Observational Studies in Epidemiology reporting guideline17 that are applicable to our study design, where policy documents are analogous to participants. This study does not involve patient or community partners. This is a descriptive study with no hypothesis testing.
Policy sampling strategy
We will use the Overton database to identify policy and/or guidance documents which have cited articles published in predatory journals.18 The Overton database is the largest searchable index of policy documents, guidelines, think tank publications and working papers. Launched in 2019, it collects data from 182 countries and more than 1200 sources worldwide including governments, intergovernmental organisations, and some think tanks and non-governmental organisations. As of November 2021, it contains over 5.1 million policy documents. Overton defines policy documents broadly as ‘documents written primarily for or by policymakers’, and aims to collect working papers, reports, case studies, policy briefs, testimony, clinical guidelines and government documents that are not behind a paywall or clearly in the scholarly record (eg, journal articles). Overton organises policy documents by topic and identifies relationships with other policy documents, academic research and media. This database allows researchers, think tanks, funding agencies and other users to search these documents to determine where research is being cited and how it is shaping policy.
Predatory journal sampling strategy
At present, there is no broadly accepted standard practice to identify predatory journals. The nature of predatory journals is that they are not systematically indexed, meaning they can be hard to identify and retrieve. Lists of journals deemed predatory that are available online are not transparently or systematically curated,19 or are not publicly available.20 To keep our research to a manageable and feasible scope, we will use journals published by OMICS Group, an established predatory publisher. OMICS Group was fined US$50 million in 2019 by the US Federal Trade Commission for deceptive business practices including falsely claiming peer review, listing scientists as journal editors without their knowledge, using fake impact factors, and unauthorised use of logos implying that journals were indexed in the US National Library of Medicine, PubMed Central and Medline.21 22 Currently, OMICS Group has over 700 open-access journals organised by subject area. Overton allows for identification of policy documents that have cited work by the OMICS Group. We will export all policy documents that have cited OMICS articles into Distiller SR (Evidence Partners, Ottawa, Canada), which is cloud-based and audit-ready software that will allow for screening and data extraction.
Eligibility criteria
The Overton database includes some documents which do not meet our definition of policy documents. As a result, we will screen exported documents to ensure they meet our definition of policy documents, which we have based off definitions from Overton and the US Department of Commerce. We will include statements from any agency or organisation that put forward a policy on a statutory, regulatory, or technical issue, or interpretation thereof, or documents otherwise primarily intended for policymakers.23 This definition will include: working papers, briefs, clinical guidelines, regulatory submissions, and other documents released by these agencies that report on their policies or are otherwise obviously intended for policymakers. We will exclude: original scientific research (articles, journals and conference proceedings, with the exception of clinical guidelines published as research articles), and documents unrelated to policy and their audience. We will not exclude documents based on language of publication; however, for those not in English, we will take note of which language they were written in. If two team members are not fluent in the language used in the policy, manually extracted questions for that document will be marked as unable to assess. We will include policy documents published since 2012 when the concept of predatory journals was first widely disseminated.24
Data extraction
Once we have identified our sample of policy/guidance documents citing articles from the OMICS Group journals, we will extract all available meta-data from the Overton database on these documents. This includes both information on the policy/guidance document and the OMICS journal article that was cited. For the policy/guidance document, these data include: the title of the document, the URL, the source (eg, WHO, Center for Disease Control, including categorisation of the source and country if applicable), unique policy document ID number, citations by other Overton policy documents and the date of publication of the document. For the journal article cited, this information includes but is not limited to: the title of the article, the DOI, the journal it was published in and the publication year. From the policy documents, we will manually extract and code the funding source, and discipline of the policy (eg, health, non-health). For the articles, we will extract information including: article type (eg, clinical trial, review, cohort study, etc), affiliations of corresponding authors, number of authors, funding, ethical approval, population (eg, adults), discipline of article (eg, health economics), total number of references, sample size and how the predatory article is used in the policy document. Details on how the predatory article is being cited represent a key component of our study. This will provide a basic understanding around whether articles are being cited simply as background information, or if they are being used in a manner that may directly inform policy recommendations. Data automatically extracted from Overton will be manually reviewed to screen for irregularities (ie, inconsistencies or oddities in the data). Should irregularities be identified, we will confirm the automatically extracted information manually. For example, in pilot searches of Overton, we noted that some automatically generated article DOIs are incorrect. The full-text article will be reviewed to manually check the DOI for accuracy. If more than one version of a policy document is found, the most recent version (or the English version if published in multiple languages) will be retained. In order to reduce bias and between extractor variation, we have created standardised and objective data extraction forms to collect the above-mentioned data. These are available in online supplemental appendix 1. We have also created a ‘codebook’ with objective question and answer descriptions, and will require all extractors to pilot test the extraction forms on the same 10 articles, to ensure consistency, as part of their training. Once the data extraction forms have been pilot-tested and agreed on by the study team, data extraction will be performed by one reviewer and audited by a second reviewer. If necessary, any conflicts will be resolved by discussion or by senior investigators—KDC and MML.
Supplemental material
Once the cited predatory articles have been identified using the above-mentioned method and workflow, we will use their DOI to obtain information on whether the articles have been cited, and if so, by whom. To do so, at least two sources would be used among Google Scholar, Scite and The Lens. We will determine the choice of these sources on the basis of two criteria: (1) the possibility to freely reproduce data without infringing on possible copyright issues and (2) the possibility of implementing automated or semiautomated procedures for extracting scholarly citations. We will describe the scope and limitations of the retained sources for this extraction. Other types of data including self-citations and altmetrics might be collected depending on the tools that are retained for data extraction and potential copyrighted information. We will make sure that the scholarly citations extracted on the basis of DOIs match the records of predatory journal articles found in Overton by reviewing their associated metadata (journal title, article title, publisher, etc). As citation counts are dynamic, we will record the dates when data were extracted for individual entries. We will use the label ‘unworkable’ for DOIs that are not found in the chosen sources for scholarly citation extraction. These data will allow us to determine potential correlations between the number of citations of predatory journal articles in policy documents and the number of scholarly citations of these same articles. We anticipate that the data will not be normally distributed; therefore, an inverse transformation will be applied to the data. A Pearson correlation test will then be used to test for an association between the number of times an OMICS article is cited in Overton and the number of times the article was cited in the academic literature.
Data analysis
Both characteristics of the citing policy documents as well as the articles published in an OMICS journal, and their records of scholarly citations, will be summarised with descriptive statistics using count data and percentages. There will be no hypothesis testing performed; the study will exclusively be descriptive.
Study timeline
The Overton database search was conducted on 18 June 2021. Exporting Overton data, preparing data for extraction, and recruiting and training extractors will be completed in February 2022. Extraction will follow with completion expected by June 2022. Data analysis is expected to be complete by August 2022.
Patient and public involvement
This work does not involve any patients or members of the public.
Ethics and dissemination
Ethics
No ethical approval is required for this study since it does not involve human or animal research.
Dissemination
We aim to make all study results readily accessible to researchers, policymakers, and the community at large to raise awareness around predatory publishing and associated risks. Study findings will be discussed at workshops on journalology and predatory publishing and will be disseminated through preprint, peer-reviewed publications and conference presentations.
Discussion
Results from this study will provide critical insight into the citation of predatory journals in policy and guidance documents. Determining whether predatory journals are cited in policy/guidelines is important for several reasons. First, if predatory journals are cited in policy/guidelines, low-quality research may influence policymaker decisions that could have repercussions in broad disciplines such as healthcare delivery or economic and climate reform. Second, citation of these journals in policy documents increases the credibility of these journals, which may sow confusion for the public, researchers and clinicians. This risk of confusion may be amplified or vary according to the record of scholarly citations of these predatory journal articles.
The proposed study is not without its limitations. Although Overton is the largest database of policy documents and guidelines currently available, it is not a complete list of all policy documents. Overton tracks over 1200 various policy sources, but we may not be able to generalise our findings to policy documents it does not capture. To obtain our sample of predatory journals, we selected journals from the known predatory publisher, OMICS. OMICS is a very large predatory publisher—a search of The Lens estimates OMICS published 92 662 journal articles from 2012 to 2019—but it is not known what proportion of predatory journals OMICS represents; journals from this publisher may not be representative of predatory journals more broadly. Due to the nature of predatory publishing, it is a challenge to estimate the true scale of the problem, with the last key study in 2015 unlikely to reflect the current landscape.25 This means we will only identify a portion of the predatory journal articles cited in policy documents, and cannot completely generalise the citation patterns observed among OMICS journals to a broader sample of predatory journals/publishers. Finally, we are not evaluating the quality of cited OMICS articles; certainly high-quality work (that should be cited) may find its way into OMICS journals; however, we would note that evaluations to date have demonstrated that articles published in potentially predatory journals are of extremely low quality.4 26 An evaluation of these articles may be done in a future study. Despite these limitations, the proposed work is the first to address this question, and will thus provide initial evidence of the penetration of predatory journals in public policy.
Ethics statements
Patient consent for publication
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Twitter @manojlalu, @glbryson, @dmoher, @mallory_pitts
Contributors The following list of contributions is made using CRediT taxonomy: OB-L—methodology, writing (original draft), writing (review and editing). MML—conceptualisation, methodology, writing (original draft), writing (review and editing), project administration. EAA—conceptualisation, methodology, writing (review and editing). MAA—methodology, writing (review and editing). EA—methodology, writing (review and editing). FA—methodology, writing (review and editing). GLB—methodology, writing (review and editing). CD—methodology, writing (review and editing). DD—methodology, writing (review and editing). AG—methodology, writing (review and editing). CL—methodology, writing (review and editing). RL—methodology, writing (review and editing). DM—methodology, writing (review and editing). JM—methodology, writing (original draft), writing (review and editing). MP—methodology, writing (review and editing). AR—methodology, writing (original draft), writing (review and editing). PT—methodology, writing (review and editing). KDC—conceptualisation, methodology, writing (original draft), writing (review and editing), project administration.
Funding Article processing charges were paid by the Dalhousie University Anesthesia Research Fund. No specific project funding was received. MML is supported by the Ottawa Hospital Anesthesia Alternate Funds Association and holds a University of Ottawa Junior Research Chair in Innovative Translational Research. No award or grant numbers are applicable.
Competing interests EAA is the founder and CEO of Open Policy Limited, which creates the Overton database.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.