Article Text
PDF
Abstract
Objectives To describe and compare the characteristics of scholars who reviewed for predatory or legitimate journals in terms of their sociodemographic characteristics and reviewing and publishing behaviour.
Design Linkage of random samples of predatory journals and legitimate journals of the Cabells Scholarly Analytics’ journal lists with the Publons database, employing the Jaro-Winkler string metric. Descriptive analysis of sociodemographic characteristics and reviewing and publishing behaviour of scholars for whom reviews were found in the Publons database.
Setting Peer review of journal articles.
Participants Reviewers who submitted peer review reports to Publons.
Measurements Numbers of reviews for predatory journals and legitimate journals per reviewer. Academic age of reviewers, the total number of reviews, number of publications and number of reviews and publications per year.
Results Analyses included 183 743 unique reviews submitted to Publons by 19 598 reviewers. Six thousand and seventy-seven reviews were for 1160 predatory journals (3.31% of all reviews) and 177 666 reviews for 6403 legitimate journals (96.69%). Most scholars never submitted reviews for predatory journals (90.0% of all scholars); few scholars (7.6%) reviewed occasionally or rarely (1.9%) for predatory journals. Very few scholars submitted reviews predominantly or exclusively for predatory journals (0.26% and 0.35%, respectively). The latter groups of scholars were of younger academic age and had fewer publications and reviews than the first groups. Regions with the highest shares of predatory reviews were sub-Saharan Africa (21.8% reviews for predatory journals), Middle East and North Africa (13.9%) and South Asia (7.0%), followed by North America (2.1%), Latin America and the Caribbean (2.1%), Europe and Central Asia (1.9%) and East Asia and the Pacific (1.5%).
Conclusion To tackle predatory journals, universities, funders and publishers need to consider the entire research workflow and educate reviewers on concepts of quality and legitimacy in scholarly publishing.
- medical ethics
- medical education & training
- medical journalism
Data availability statement
No data are available. The data sets used for and generated in the course of this study are not publicly available in order to protect Cabell's International’s intellectual property rights and to respect the reviewer privacy policies imposed by Publons on behalf of the publishers.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
This study was based on a large sample of peer review reports submitted to Publons.
It is the first study to describe and compare the characteristics of scholars who have reviewed for predatory or legitimate journals, thereby providing insights that can help in educating potential reviewers.
Classifying journals as predatory or legitimate suggests a dichotomy, yet journals can vary in the number and severity of violations of best practices and hence operate in a grey zone.
Selection biases in claiming reviews on Publons might have caused this study to underestimate shares of predatory reviews as reviewers might be more likely to provide reviews for legitimate or high-quality journals to Publons than for predatory or low-quality journals.
Whether a review was written for a predatory journal or a legitimate journal does not allow conclusions on its quality or the expenditure of time required to write it.
Introduction
Scholars spend a considerable amount of their time reviewing the work of their peers. However, reviewing for so-called predatory journals might be a waste of valuable time and effort. There have been several attempts to define the characteristics of predatory journals.1 2 One recent community-led definition describes predatory journals and publishers as entities that prioritise self-interest at the expense of scholarship. They are characterised by false or misleading information, deviation from best editorial and publication practices, a lack of transparency or the use of aggressive and indiscriminate solicitation practices.3
A recent study suggests that 13.7 million reviews are carried out every year and that writing a review takes a median of 5 hours.4 When assessing a manuscript for its soundness, originality, validity and possible impact, reviewers serve a critical gatekeeping function: they help ensure that only papers that pass a certain quality threshold enter the scholarly record.5–10 Such gatekeeping is particularly important in fields where practitioners and policymakers rely on evidence in the form of published journal articles. Without rigorous peer review, invalid studies could influence policies and practices and potentially cause harm to the population.11
In contrast, reviewing for so-called predatory journals might be a waste of valuable time and effort. As one of many deficiencies, predatory journals do not guarantee archiving and long-term access to their contents.2 12 Their publications generate limited readership, are rarely cited and might eventually be lost.13 Even though providing feedback on manuscripts submitted to predatory journals might help authors improve their work, reviewing a manuscript that will have hardly any scientific impact is unlikely to advance science. Also, there have been anecdotal reports that peer review in predatory journals lacks scientific integrity. To highlight some of the predatory journals’ problems, a researcher submitted purposely flawed manuscripts and showed that where reviewers spotted severe scientific problems, editors accepted papers nevertheless.14 Just as publishing research in predatory journals is likely to be a waste of resources, reviewing for these outlets should be viewed in the same critical light, and efforts are needed to stop this practice.15
Little is known about the scholars who contribute to the peer review of predatory journals. Like most other forms of academic gatekeeping, peer review is generally not publicly observable.16 This is particularly true for predatory journals, which lack transparency of publishing processes and policies.1 Understanding who reviews for predatory journals might help gain insights into these outlets’ peer review processes and thus assist in combating them. We aimed to analyse and compare the characteristics of scholars who have reviewed for predatory or legitimate journals, as defined by Cabells Scholarly Analytics’ journal lists. We analysed their sociodemographic characteristics as well as their reviewing and publishing behaviour.
Methods
To study the characteristics of scholars who submitted peer reviews for predatory and legitimate journals to Publons, we linked random samples of journals indexed in Cabells Scholarly Analytics’ (hereafter Cabells) journal lists with the Publons database of peer review reports. We analysed publishing and reviewing behaviour and sociodemographic characteristics for the scholars for whom we identified reviews in the Publons database.
Data sources
We used Cabells’ lists of predatory and legitimate journals to identify reviews for predatory journals and legitimate journals. Cabells is a scholarly, for-profit service that hosts and maintains lists of legitimate and predatory journals. The list of legitimate journals is intended to provide academics with information on reputable outlets for publication.17 The list of predatory journals aims to warn against journals with problematic practices. To be indexed in either list, journals need to fulfil several criteria, which cover their business practices (eg, marketing practices), policies (eg, prepublication policies), peer review or editorial services (eg, the qualification of reviewers).18 We chose to use Cabells’ list of predatory journals because it is the only comprehensive and up-to-date source for identifying such journals across academic disciplines. The only other list of predatory journals has not been operational since 2017, even though an archived version can be accessed online.19 We purchased access and downloaded the lists of predatory journals and legitimate journals in December 2018. The list of predatory journals contained 10 671 journal titles, and the list of legitimate journals included 11 057 journal titles.
Publons is a platform for researchers to track their scholarly contributions and receive recognition for their peer reviews. In October 2019, it contained data on more than 5 million reviews, spanning more than 500 000 reviewers and approximately 40 000 journals.20 Researchers can provide Publons with reviews that they performed for any journal: Publons is not an indexing service and does not endorse or ban journals based on their quality or legitimacy. It provides access to reviews to promote transparency and recognises reviewing engagement regardless of the reputation of journals. Publons verifies that reviewers performed the reviews they claim. When researchers review for partnered journals and opt in for recognition, Publons receives review data directly from the journal. For non-partnered journals, authors can submit their reviews by forwarding the ‘Thank you…’ emails from the journals to Publons. Reviewers can also send a screenshot of the review in the journal’s peer review submission system. Publons randomly audits emails by contacting editors or journal administrators.21 22
As of July 2020, a total of 704 126 reviewers had submitted reviews to Publons. Information on publishing behaviour was available for 291 792 (41.4%) reviewers. For all Publons users, the mean number of reviews per reviewer was 17.7 (SD 47.1), and the mean number of publications per reviewer was 41.7 (SD 71.2). For 195 160 reviewers on Publons (27.7%), information was available on both the country of their primary institution and their publishing record. Most reviewers were affiliated with an institution in Europe and Central Asia (45.5%), followed by North America (14.6%), East Asia and Pacific (17.4%), Latin America and Caribbean (9.0%), Middle East and North Africa (6.4%), South Asia (5.6%) and sub-Saharan Africa (1.5%).
Study sample
Due to intellectual property rights restrictions, we could not match the full Cabells’ list of predatory and legitimate journals with the Publons database. Instead, Cabells permitted us to use samples from either list. In the first step, we drew random samples of 1000 journal titles each from the list of predatory and legitimate journals using the =RAND () formula in MS Excel. We assigned a random number to each row in the full data set for each separate journal list, sorted the rows based on the random numbers from largest to smallest and selected the first 1000 journal titles. In the second step, we linked the samples with the Publons database to identify reviews for predatory and legitimate journals. Following an approach used by Strinzel et al,18 we based the matching of journal titles on the similarity of journal names using the Jaro-Winkler algorithm in R package RecordLinkage.18 23 The Jaro-Winkler metric, which ranges from 0 (no similarity) to 1 (exact match), was calculated for all possible pairs of journals between the two Cabells’ journal lists and the Publons database.24 We selected all journal titles for which the Jaro-Winkler metric equalled 1 (exact match). Further, we manually inspected all pairs with a Jaro-Winkler metric smaller than 1 to identify pairs where, due to orthographical differences between lists, no exact match was found. Only journals for which we found at least one verified review in the Publons database were analysed further. In the third step, we created an extended network of journals. For scholars who had reviewed for linked journals, we selected all other journals for which they claimed one or more reviews on Publons. In the fourth step, we linked the extended network with the Cabells’ lists to classify reviews as either for predatory or legitimate journals. We again calculated the Jaro-Winkler metric for all journal pairs, selected journal titles for which the Jaro-Winkler metric equalled 1 (exact match) and performed a manual assessment of all journal pairs with a Jaro-Winkler metric smaller than 1. The different steps which were performed from July to August 2019 are shown in figure 1.
Process of matching of Publons database of reviewers with journals from Cabells’ lists of legitimate and predatory journals.
We retrieved data from the Publons database on sociodemographic characteristics and publishing and reviewing behaviour for reviewers of matched reviews. The data covered institutional affiliation, dates of first publication and last publication, number of publications and number of reviews in Publons database. Based on these variables, we calculated academic age (in years, calculated based on the date of the last publication minus the date of the first publication), publishing productivity (as the average number of articles published per year) and reviewing productivity (as the average number of reviews written per year). We further linked the country of the primary institutional affiliation with World Bank data25 to classify countries according to region and income. Analyses were performed from August to September 2019.
Patient and public involvement
This study was based on peer review reports submitted to Publons. No patients were involved in developing the research question, outcome measures, overall design of the study, participant recruitment or conduct of the study. Because of the anonymous nature of our data, we were not able to share results directly with peer reviewers.
Results
The sampling and linking procedure resulted in 183 743 unique Publons reviews that 19 598 reviewers submitted. Most of these reviews were for legitimate journals: only 6077 reviews were done for 1160 predatory journals (3.3% of all reviews) compared with 177 666 for 6403 legitimate journals (96.7% of all reviews).
Sociodemographic characteristics and publishing and reviewing behaviour
This information was available for 86 298 reviews submitted to Publons by 7349 individual reviewers (37.5% of all reviewers). Based on each reviewer’s share of reviews for predatory journals, we created five groups of reviewers (table 1; online supplemental table S1). Most reviewers (90%) had never submitted a review for a predatory journal. This group had a relatively old academic age and a high number of publications but had submitted relatively few reviews to Publons. A smaller group of reviewers (8%) had occasionally reviewed for predatory journals, with shares of predatory reviews ranging from 1% to 25% of all reviews. Reviewers in this group were experienced and productive: they had the oldest academic age, published the largest number of articles and submitted the second highest number of reviews on Publons (table 1; online supplemental table S1). About 2% of reviewers regularly reviewed for predatory journals (share 26%–75%). They occupied an intermediate position in terms of academic age and publication and review activity. Very few scholars had shares of predatory reviews between 76% and 99% (0.26% of all reviewers). This group had the youngest academic age and fewer publications than the first three groups but submitted a high number of reviews to Publons. A similarly small number (0.35%) exclusively reviewed for predatory journals. They had a young academic age, published few articles and submitted few reviews to Publons.
Supplemental material
Reviewer subgroups and their characteristics
Geographic distribution of reviews for predatory and legitimate journals
Based on the reviewer’s primary institutional affiliation, we analysed the geographic distribution of reviews. This information was available for 10 950 reviewers (55.9% of all reviewers) affiliated with institutions in 117 different countries, who submitted 107 722 reviews. To interpret reviewing behaviour in the context of reviewing productivity per region, we calculated proportions of reviews for predatory journals for each region and each World Bank country income group.25 Regions with the highest shares of predatory reviews were sub-Saharan Africa (21.8% predatory reviews), Middle East and North Africa (13.9%) and South Asia (7.0%), followed by North America (2.1%), Latin America and the Caribbean (2.1%), Europe and Central Asia (1.9%) and East Asia and the Pacific (1.5%) (figure 2). In economic terms, low-income countries had the highest share of predatory reviews (27.0%), followed by lower middle-income countries (17.5%), upper middle-income countries (4.9%) and high-income countries (1.5%) (figure 3).
Proportions of predatory reviews by World Bank region.
Proportions of predatory reviews by World Bank income class.
Discussion
This study investigated the profiles of scholars who submitted reviews for predatory and legitimate journals to Publons. It examined the reviewing patterns for different subgroups of scholars and their publishing behaviour, scientific productivity and academic age. It further investigated the geographic distribution of reviews for predatory journals and legitimate journals. The analysis showed that scholars who reviewed predominantly or exclusively for predatory journals were less experienced than reviewers with no or only a few predatory reviews: the former had a younger academic age, fewer publications and fewer reviews. Their characteristics resemble the authors publishing in predatory journals, who are young and inexperienced scholars.26 The geographical distribution of reviews submitted for predatory journals showed that low-income and lower middle-income countries have particularly high shares of predatory reviews submitted to Publons. In contrast, upper middle-income and high-income countries have low shares of predatory reviews.
Early-career researchers and scholars in low-income and lower middle-income countries may have limited access to the mentoring and educational resources available in high-income countries. In addition, early-career researchers may have less experience in scholarly publishing and reviewing than senior-career researchers. They might, therefore, be more exposed to the deceptive practices of predatory journals. Predatory journals may provide an opportunity for marginalised members of the global academic community to survive in the ‘publish or perish’ culture of today’s academia. The assessment of a scholar’s publication record and the ability to secure funding far outweigh peer review contributions when it comes to academic promotion.27 Nevertheless, scholars now increasingly use platforms that help them showcase their reviewing activities on their applications or curricula vitae.28 29 In an academic system characterised by pressure to be productive and distinguish oneself,30 early-career researchers and researchers from developing countries, in particular, might be eager to expand the list of journals for which they review. Some authors from high-income countries also publish regularly in predatory journals, indicating that extending the publication list in this way can also be attractive for scholars working in these countries.15
Cabells’ lists used in this study suffer from several limitations. First, the stability of the lists is limited given ever-changing publishing practices of journals11; second, classifying journals as predatory or legitimate suggests that predation is a dichotomous category; yet journals can vary in the number and severity of violations of best practices and hence operate in a grey zone16; third, the intention to deceive is a crucial criterion31 but difficult to assess; fourth, Cabells’ lists give most weight to criteria that are easy to verify, for example, the quality of a journal’s website, but neglect important criteria that are difficult to verify, like peer review and editorial services.18
Due to intellectual property restrictions, we could not link Cabells’ entire lists against the Publons database. Instead of simply identifying journal titles from the lists in the Publons database, we had to draw random samples of journals, link them with the Publons database and analyse the matched reviews. The restrictions imposed by Cabells made our methodological approach more complicated and less transparent. Despite these drawbacks, we believe that Cabells’ journal lists provide a more objective and rigorous approach for identifying predatory and legitimate journals than any other currently available journal lists.
Whether a review was written for a predatory journal or a legitimate journal does not allow us to draw any conclusions regarding its quality or the expenditure of time required to write it. Reviewers may provide thorough peer reviews for predatory journals, thereby improving the work published in these journals. Further, it is unclear how the quality of peer review differs across groups of reviewers, for example, junior and senior scholars. Our study was not designed to evaluate the quality of individual reviews or whether editors followed the reviewers’ recommendation. However, others have shown that peer review in predatory journals was superficial, for example, by focusing on a manuscript’s layout rather than on its scientific content.14 16 Also, there is evidence that where reviewers identified scientific problems, editors accepted papers nevertheless.14 In the future, we plan to evaluate whether the quality of peer review differs across different types of journals and groups of reviewers.
Our study is limited by the relatively small number of scholars who reviewed for predatory journals and a lack of available data. The small number of cases in some cells in table 1 limited our estimates’ precision regarding the sociodemographic characteristics of reviewers. We chose not to analyse the data on when a review was submitted to Publons, which hampered conclusions on the correlation between academic age and reviewing behaviour. It would have been interesting to examine whether predatory journals ask past authors to review new manuscripts. Unfortunately, we did not have access to data on the journals in which reviewers published before they reviewed for predatory or legitimate journals.
Finally, the use of the Publons database also comes with limitations. Reviewers might be more likely to provide reviews for legitimate or high-quality journals to Publons than for predatory or low-quality journals. Such bias will probably have led us to underestimate shares of predatory reviews. Although the proportion of predatory journals seeking peer review is unknown, it is likely that the number of predatory journal reviews submitted to Publons represents a fraction of those conducted and that the ones submitted to Publons will differ from the overall group. Further, although Publons verifies reviews, there is potential for misconduct. By claiming made-up reviews on Publons, predatory journals could create a false impression of scientific integrity and quality. However, the fact that we looked at Publons users with a record of both reviewing and publishing provides reassurance that these are real scholars. To our knowledge, the Publons database is the only available database of peer reviews that is not limited to individual publishers or journals, making it a unique resource for studies such as this one.
Conclusions
To our knowledge, this is the first study to investigate the profiles of scholars reviewing for predatory journals. Our results show that relatively few such reviews were submitted to Publons. Still, writing these reviews probably involved a considerable expenditure of time and resources. The profiles of scholars reviewing for predatory journals resemble the scholars who publish in these outlets: they tend to be young and inexperienced researchers and affiliated with institutions in lower income countries. Predatory journals have gained relevance for scholars for publishing their own work and for reviewing the work of others. A holistic approach to combating deceptive journals is, therefore, needed, taking into account the entire research workflow.
Most initiatives addressing the problem of predatory journals focus on reducing the submissions of manuscripts by warning authors not to publish their work in these outlets.11 To prevent scholars from reviewing for predatory journals, research institutions, funders and publishers ought to invest in the training of reviewers. Such training should enable them to make informed decisions on what journals they review for. Reviewers should be educated on concepts of quality and legitimacy in scholarly publishing and the characteristics of predatory journals to help them avoid reviewing for the latter. Particularly early-career scholars and scholars located in developing countries should be provided with educational resources and services. An adapted version of the Think, Check, Submit guidance, which provides researchers with a checklist intended to help them identify trustworthy journals, could be a useful resource for reviewers.32 Funders and universities should monitor for which journals their grantees and faculty members review and warn against committing time to review for predatory journals. When evaluating applications for funding or promotion, they might check peer review records for predatory titles. Services that help researchers get credit for their reviewing activities should have a clear policy for predatory journals. Providers of lists of predatory journals could check peer-reviewing activities and assess the contents of submitted reviews for quality and rigour. Research institutions, funders and publishers should boost their efforts to discourage scholars from reviewing for these outlets. Finally, more research is needed to investigate the quality of peer review across different journals, including legitimate and potentially predatory journals.
Data availability statement
No data are available. The data sets used for and generated in the course of this study are not publicly available in order to protect Cabell's International’s intellectual property rights and to respect the reviewer privacy policies imposed by Publons on behalf of the publishers.
Ethics statements
Ethics approval
Under Swiss law, no ethics approval is required for this type of study. No peer reviewer or peer review report can be identified from this report.
Acknowledgments
We would like to thank Katrin Milzow for her useful feedback on an earlier draft of this manuscript, Matt Hodkinson for his helpful comment on the preprint and Jennifer Frost for her critical reading of the final manuscript.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Contributors AS: conceptualisation, methodology, formal analysis, investigation, data curation, visualisation, writing–original draft preparation, writing–review and editing. MS: investigation, visualisation, writing–review and editing. ME: methodology, supervision, writing–review and editing. MD: data curation, investigation, writing–review and editing. TB: conceptualisation, methodology, investigation, data curation, supervision, writing–review and editing.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests AS and MS work for the SNSF. ME is president of the Research Council of the SNSF. MD works for Publons. TB was product lead at Publons at the time of the study.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.