Article Text
PDF
Abstract
Objective We investigated characteristics of systematic reviews (SRs) assessing measures to prevent COVID-19 by (1) identifying SR registrations in Prospective Register of Systematic Reviews (PROSPERO), (2) identifying published SRs in COVID-19 Living Overview of the Evidence (L-OVE) and (3) estimating the proportion of PROSPERO registrations published as full SR between 8 and 16 months after registration.
Study design This meta-research study is part of the German CEOsys project, www.covid-evidenz.de. We searched PROSPERO entries registered between 1 January 2020 and 31 August 2020, and we searched COVID-19 L-OVE for published SRs (search date: 5 May 2021) focusing on measures to prevent COVID-19 and SARS-CoV-2 transmission. The two samples were screened for eligibility and key characteristics were extracted and summarised.
Results Of 612 PROSPERO registrations, 47 focused on prevention and were included. The preventive measures included public health interventions (20), followed by personal protective equipment (10), vaccinations (9) and others (8). In total, 13 of 47 (28%) PROSPERO registrations had been published as full SR (as preprint only (6), as peer-reviewed article only (6), as preprint and peer-reviewed article (1)). Median time between PROSPERO registration and publication was 5 months for peer-reviewed SRs and 2 months for preprints.
Of the 2182 entries identified in COVID-19 L-OVE, 51 published SRs focused on prevention and were included. Similar to the PROSPERO sample, most published SRs focused on public health interventions (21). The number of included primary studies ranged between 0 and 64 (median: 7). Nine published SRs did not include any studies because of a lack of primary studies.
Conclusion Considering the urgent information needs of policymakers and the public, our findings reveal the high-speed publication of preprints and lack of primary studies in the beginning of the COVID-19 crisis. Further meta-research on COVID-19 SRs is important to improve research efficiency among researchers across the world.
PROSPERO registration number CRD42021240423.
- COVID-19
- PREVENTIVE MEDICINE
- PUBLIC HEALTH
Data availability statement
All data relevant to the study are included in the article or uploaded as supplementary information.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
We systematically described Prospective Register of Systematic Reviews (PROSPERO) registrations and published systematic reviews identified in COVID-19 Living Overview of the Evidence (L-OVE) focusing on COVID-19 prevention (the most important intervention at the beginning of the pandemic).
Although the focus of this research was on the beginning of the pandemic, the findings and methodological approaches are important regarding pandemic preparedness to next disease outbreaks.
We considered systematic reviews which are important to guide evidence-based clinical and health policy decision-making at different stages (as PROSPERO registration and published as full systematic review, including preprints and peer-reviewed articles).
We estimated the proportion of PROSPERO registrations published as full systematic review within 8 to 16 months after registration.
Introduction
The COVID-19 pandemic is still impacting almost all countries worldwide. By 21 March 2022, approximately 6 100 000 (https://www.worldometers.info/coronavirus/) people died due to COVID-19. Since the beginning of the pandemic in early 2020, researchers are responding to the virus by conducting a wide range of research from basic research to clinical studies and systematic reviews—to identify both the most effective prevention and treatment strategies.
In the COVID-19 pandemic and beyond, the synthesis of clinical studies within systematic reviews is essential to guide evidence-based clinical and health policy decision-making. Prior to 2011, only a few organisations, including Cochrane and the Joanna Briggs Institute, disseminated protocols (to define the research question and methods) for the planned or ongoing systematic reviews and the majority of reviews have become ‘public’ only at the time when the review was completed, peer-reviewed and published.1 To facilitate the transparency, reproducibility and usability of conducted systematic reviews, the International Prospective Register of Systematic Reviews (PROSPERO) was launched in February 2011 and it is recommended that each systematic review is registered before conducting the full systematic review.1 2 A PROSPERO registration can be updated once the review is completed and the full citation for the final report should be provided (including the uniform resource locator (URL)). Besides the increasing transparency regarding the conduct of systematic reviews, PROSPERO is also a valuable source to investigate the quality of current research (ie, for meta-research, research on research).
There have been several investigations (meta-research projects) on PROSPERO registrations3–5 and also on published systematic reviews of COVID-19.6–10 These meta-research studies focused on different methodological aspects, including the external validity of the research questions. Moreover, they often revealed poor reporting in COVID-19 research, both at the protocol stage and of the published systematic review. However, to our knowledge, there has been no investigation on PROSPERO registrations focusing on epidemiological and methodological characteristics and publication rates of prevention research during the beginning of the COVID-19 pandemic in 2020.
We conducted a meta-research study to investigate the number and characteristics of PROSPERO registrations and published systematic reviews identified in COVID-19 Living Overview of the Evidence (L-OVE) (a web-based app which aims to capture the entirety of all published research addressing COVID-19) focusing on measures to prevent COVID-19 and SARS-CoV-2 transmission during the beginning of the pandemic. Moreover, we determined the proportion of PROSPERO registrations (registered up to 31 August 2020) that have been published either as preprint or peer-reviewed systematic review (by 5 May 2021) and we piloted an approach to evaluate whether there are methodological differences between the PROSPERO registration and the corresponding published systematic review.
Methods
This meta-research study followed the methods of a systematic review and is reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 2020 guideline.11 This meta-research project was prospectively registered in PROSPERO and has been part of a larger research project (CEOsys, https://covid-evidenz.de/) funded by the German Federal Ministry of Education and Research (BMBF, grant number 01K×2021). The registered protocol for the current project refers to (1) research on preventive measures for COVID-19 and (2) research on treatment measures for COVID-19 (on the date of April 2022, the research focusing on treatment is still ‘ongoing’). We decided to present both research questions in separate publications taking into account that prevention research has been the first response to COVID-19 and SARS-CoV-2 transmission in the early phase of the pandemic. We believe that evaluating both samples (prevention and treatment) together would not allow us to point out the importance of prevention research under pandemic circumstances in detail. Furthermore, stakeholders related to prevention research and research on treatment measures may (partly) differ: while epidemiologists, the general population and different stakeholders involved in public health decisions are more interested in preventive measures, clinicians and/or stakeholders involved in clinical guideline development may be more interested in COVID-19 treatment.
Systematic literature searches
First, we searched the PROSPERO registry (https://www.crd.york.ac.uk/prospero/) for entries registered between 1 January 2020 and 31 August 2020 (including the beginning of the pandemic) focusing on measures to prevent COVID-19 and SARS-CoV-2 transmission. The PROSPERO COVID-19 filter was applied and the search was restricted to specific fields (prevention, treatment). The automatic search was supplemented by manual searches. The keywords used are displayed in online supplemental material S1. The search strategy used for the automatic search is displayed in online supplemental material S2.
Supplemental material
Supplemental material
Second, we searched for published systematic reviews in COVID-19 L-OVE (https://app.iloveevidence.com/). COVID-19 L-OVE contains entries from over 40 medical databases (including Medline, Embase, Cochrane Library, CINAHL and others) and registries (including different national trial registries, medRxiv, bioRxiv, Research Square and others). Our search made use of filters as implemented in COVID-19 L-OVE (‘Prevention and Treatment’ and ‘Systematic Review’ filters, see online supplemental material S3). We did not further restrict our search by using keywords. The search for published reviews in COVID-19 L-OVE was performed on 5 May 2021. Additionally, we performed manual searches in Google Scholar that included the Center for Reviews and Dissemination (CRD) numbers of eligible PROSPERO registrations to make sure that we did not miss any PROSPERO registration published as full systematic review by 5 May 2021.
Supplemental material
Eligibility criteria and study selection
We included PROSPERO registrations and full published systematic reviews, which addressed any preventive measure in any human population confronted with the COVID-19 pandemic reporting at least one health-related outcome. Preventive measures were defined as any intervention to prevent the transmission of the virus or to prevent an infection and/or the outbreak of the disease. We did not apply any restrictions regarding the comparators. We excluded PROSPERO registrations and full systematic reviews if measures were evaluated in relation to other viruses (eg, influenza). Moreover, interventions to prevent aggravation of clinical symptoms were not considered.
Considering the fact that PROSPERO registrations do not provide abstracts, screening for eligibility was based on the full registration entry. The records identified in COVID-19 L-OVE, on the other hand, were screened using a two-step approach: (1) title and abstract screening and (2) full-text screening. The screening process in PROSPERO was conducted by two reviewers independently (JN and JS). The screening process in COVID-19 L-OVE was conducted by one reviewer (JN) and checked by another reviewer (JS). Disagreements were resolved by discussion between both reviewers or by consulting a third reviewer (CS) to reach consensus.
Data extraction
The following main characteristics of the PROSPERO registrations and published systematic reviews (identified in COVID-19 L-OVE) were extracted: reference (eg, registration ID, CRD number or DOI), corresponding author, institutional affiliation, review type (eg, network meta-analysis, living systematic review, rapid systematic review), population, intervention and primary outcomes (as defined in the inclusion criteria both in the PROPERO registration and the published systematic review). Furthermore, we collected information on study types predefined in the PROSPERO registration and study types included in the published systematic reviews.
For PROSPERO registrations, we additionally extracted the registration date and anticipated completion date. Furthermore, to assess deviations between the PROSPERO registration and the corresponding published systematic review, we additionally extracted methodological key data, including information on the database search, the risk of bias assessment and the outcomes of interest. These key data were compared between the PROSPERO registration and the published systematic review to explore possible deviations, which may impact the methodological quality of systematic reviews.
For published systematic reviews identified in COVID-19 L-OVE, we additionally extracted the number of included studies, type of publication (preprint, journal publication) and whether a published protocol was available (either additionally to the PROSPERO registration or only as publication (journal publication or published on a platform other than PROSPERO)). Data extraction was performed by one reviewer (JN) and checked by a second reviewer (JS). Any disagreements were resolved by discussion or by involving a third reviewer (CS) if no agreement could be reached.
Outcomes
Our main outcomes were (1) the number and characteristics of COVID-19 PROSPERO registrations with focus on prevention that were registered during the first pandemic wave (between 1 January 2020 and 31 August 2020), (2) the number and characteristics of published COVID-19 systematic reviews with focus on prevention identified in COVID-19 L-OVE up to 5 May 2021 and (3) the proportion of PROSPERO registrations that have been completed and published as full systematic review by 5 May 2021 (including the time between registration and publication).
Data synthesis
Data analysis involved a combination of qualitative synthesis and descriptive statistics for the identified PROSPERO registrations and also for the published systematic reviews. To estimate the proportion of PROSPERO registrations that were published as full systematic reviews, we matched the PROSPERO registrations and the published systematic reviews based on (1) key characteristics (population, intervention, study design), (2) URLs provided in the PROSPERO registration, (3) registration (CRD) numbers provided in the published systematic reviews and (4) comparing corresponding and/or first authors in PROSPERO registration with corresponding and/or first authors in the published systematic review.
Within our sample of PROSPERO registrations, we calculated the overall proportion of those registrations that were published as full systematic review by 5 May 2021. This analysis was based on dichotomous data (published vs not published). Additionally, we calculated the median time in months between the registration in PROSPERO and the publication of the systematic review (for PROSPERO registrations that were published by 5 May 2021) and stratified the published systematic reviews after publication type (preprint or peer-reviewed article or both).
Deviations from the PROSPERO registration and the corresponding published full systematic review were summarised descriptively.
Patient and public involvement
No patient involved.
Results
Results of the literature searches
PROSPERO registrations
The searches in PROSPERO identified 612 registrations (figure 1, PRISMA flowchart). After screening (applying the eligibility criteria), 47 PROSPERO registrations were considered eligible.
PRISMA flowchart11 of PROSPERO registrations between 1 January 2020 and 31 August 2020. PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses; PROSPERO, Prospective Register of Systematic Reviews.
Published systematic reviews identified in COVID-19 L-OVE
The search in COVID-19 L-OVE identified 2179 records (figure 2, PRISMA flowchart). From these, 25 were automatically identified as duplicates by the software Endnote, 1114 were excluded during title and abstract screening and 982 during full-text screening. In total, 58 records corresponding to 48 unique published systematic reviews met our inclusion criteria. Our manual searches identified three more systematic reviews. Finally, we included 51 published systematic reviews.
PRISMA flowchart11 of published systematic reviews identified in COVID-19 L-OVE (search on 5 May 2021). L-OVE, Living Overview of the Evidence; PICO, Population Intervention Comparison Outcome; PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses.
Characteristics of PROSPERO registrations
The key characteristics of the 47 PROSPERO registrations are presented in table 1. Detailed characteristics are provided in online supplemental material S4.
Supplemental material
Characteristics of PROSPERO registrations
Affiliation of main investigator
The institutional affiliation of the main investigator was based in Asia (13/47), Latin America (12/47), Europe (10/47), North America (6/47) and others.
Population, intervention and outcomes
Most PROSPERO registrations (38/47) set their focus on the general population and/or on high-risk populations (elderly, populations with morbidities, healthcare workers or others having contact with patients with COVID-19). Almost half of the PROSPERO registrations (20/47) focused on public health interventions (eg, mass screening, quarantine, boarder restrictions, hygiene or distance, or a combination of different strategies). The most frequently predefined outcomes were incidence or prevalence of COVID-19 and/or SARS-CoV-2 transmission (47/47), mortality rates (20/47) and disease severity (defined, eg, as severity of symptoms or hospitalisation, 16/47).
PROSPERO registrations published as full systematic reviews
Between 8 and 16 months after registration, 13/47 of the PROSPERO registrations were published as full systematic review. Therefrom, 6/13 were peer-reviewed articles (including n=1 Cochrane review), 6/13 were preprints and 1/13 was published as preprint and peer-reviewed article (online supplemental material S4).
When comparing the PROSPERO registrations with the corresponding published systematic review, we identified concerns regarding the methodology in 5/13 reviews. The concerns mainly refer to (1) the selection of the reported results (ie, predefined outcomes in the PROSPERO registration and reported outcomes in the published systematic review showed major differences (3/5 reviews)) and (2) the predefined risk of bias assessment, which was finally not conducted in the published systematic review (2/5 reviews). The remaining systematic reviews (8/13) showed no or only (very) minor deviations—mainly related to the fact that the systematic review authors searched less databases to identify primary studies than indicated in PROSPERO (see also online supplemental material S5).
Supplemental material
Median time between PROSPERO registration and the date of the publication was 5 months for peer-reviewed articles (n=6, first quartile: 2.5, third quartile: 7.5, range: 1–9 months) and 2 months for preprints (n=7, first quartile: 1.5, third quartile: 4, range: 1–7 months).
Characteristics of published systematic reviews identified in COVID-19 L-OVE
The main characteristics of the 51 published systematic reviews identified in COVID-19 L-OVE are presented in table 2. Detailed characteristics are provided in Online supplemental material S6).
Supplemental material
Characteristics of published systematic reviews identified in COVID-19 L-OVE
Affiliation of main investigator
The institutional affiliation of the investigating groups was mostly based in Europe (19/51), Asia (15/51), North America (9/51) and others (8/51).
Population, intervention and outcomes
The focus in terms of population and intervention was comparable to the PROSPERO registrations. Most published systematic reviews set their focus on the general population and/or high-risk populations (50/51) and almost half of the publication (21/51) evaluated public health interventions (eg, wearing masks, social distancing, handwashing, screening for the virus). The most frequent-reported outcomes focused on the incidence or prevalence of COVID-19 or SARS-CoV-2 transmission (36/51), effectiveness (with a wide range of definitions, 20/51) and safety (16/51).
Publication status
In total, 13/51 systematic reviews were published as preprints, both as preprint and peer-reviewed publication (10/51) or as peer-reviewed publication (28/51).
Studies identified in systematic reviews
The published systematic reviews included mostly non-randomised studies of interventions (29/51) and/or randomised controlled trials (12/51) and/or modelling studies (12/51). In 9/51 systematic reviews, no clinical studies were included. Overall, the total number of included studies ranged between 0 and 64 (figure 3).
Boxplot for number of included studies within published systematic reviews identified in COVID-19 L-OVE. Upper whisker (40) and lower whisker (0) extend 1.5 times the IQR from the third and first quartile. Box height represents third quartile (75%)=18 and first quartile (25%)=3; IQR=15; Centre line inside the box represents median=7. Each dot represents one sample point. n=50 sample points. Boxplot was created with http://shiny.chemgrid.org/boxplotr/ based on R statistics software. L-OVE, Living Overview of the Evidence
Protocol published prior to publication of systematic review
In total, 17/51 published systematic reviews were registered in PROSPERO and 3/17 of the registered PROSPERO registries were additionally published in a peer-reviewed journal, including one Cochrane protocol. Furthermore, for 5/51 systematic reviews, we identified a protocol on a platform other than PROSPERO (on the Open Science Framework platform (OSF.io) or on the website of the affiliated institution). For the remaining 29/51 published systematic reviews, neither a PROSPERO entry nor a published protocol exists.
Discussion
Main findings
In contrast to the PROSPERO registrations where most prevention research was initiated in Asia (mainly India and China), followed by Latin America (mainly Brazil) and Europe (mainly UK), published systematic reviews identified in COVID-19 L-OVE were affiliated with European countries (mainly UK) followed by Asia (mainly India and China) and North America (USA and Canada). Similar to the PROSPERO registrations, most of the published systematic reviews focused on public health interventions and on the general or high-risk population. Outstanding, at the beginning of the pandemic preventive measures particularly in school populations were not adequately considered (n=2 PROSPERO registrations, n=1 published systematic review).
Approximately, 20% (9/51) of the published systematic reviews identified in COVID-19 L-OVE did not include any primary clinical study addressing the research question of interest. The reason for these ‘empty reviews’ may be associated with restrictions regarding the predefined eligible study design such as randomised controlled trials or a general lack of studies. While it is obvious that these ‘empty reviews’ may not be useful for decision-making12 they reveal important research gaps to initiate primary studies.
Of the systematic reviews identified in COVID-19 L-OVE, 45% (23/51) were published as preprints. Preprints have played an important role in the COVID-19 pandemic. For example, postings on the preprint server MedRxiv have increased by over 400%: from over 580 in the last 4 months in 2019 to over 2500 in the first 4 months in 2020. Additionally, views of preprint have increased by 100%.13 Besides bypassing the often-delaying peer-review process, preprint studies also benefit from immediate open access dissemination and facilitate collaborations between researchers worldwide. However, the increasing adoption of preprint studies is also associated with pitfalls: for example, even before the pandemic, up to 85% of research was ‘wasted’ and/or biased due to poor research questions, poor study designs and study methodology, poor reporting and selective publication.14–16 Taking into account the time pressure and often an inadequate research infrastructure, many of these problems have been amplified in COVID-19 research.6–10 Moreover, a lacking critically scientific validation by peer review may particularly impact the large number of preprint publications.17
Comparison with other meta-research
To our knowledge, there is no other meta-research project published or ongoing that describes the characteristics of systematic reviews of COVID-19 prevention measures at the protocol stage (ie, when registered in PROSPERO) and for published systematic reviews. For example, Andersen et al 18 searched for published meta-analyses indexed in the PubMed database (https://pubmed.ncbi.nlm.nih.gov/) and for corresponding a priori registered PROSPERO entries focusing on any clinical intervention—but before the COVID-19 crisis. In this meta-research, one-third of the 2475 meta-analyses identified were affiliated with institutions in England or China, followed by another third from USA, Canada, Australia and Brazil. These are basically the same countries we also identified as the ‘leaders in the pandemic’—except for Australia, where the COVID-19 incidence was very low during the early phase of the pandemic.19 Furthermore, approximately 20% of the meta-analyses in Andersen et al 18 were published within 0 to 9 months after registration in PROSPERO. Considering that our meta-research focused on preventive measures in COVID-19 and also included preprint publications, it is difficult to compare our findings with the results of Andersen et al. It further remains unclear how many of the PROSPERO registrations in our sample have been stopped before completion or rejected for publication or where still under ‘peer review’, whereas in Andersen et al,18 the study sample consisted only of meta-analyses that have all been published successfully.
Strengths and limitations
The current meta-research study represents characteristics of PROSPERO registrations and published systematic reviews on preventive measures—the most important intervention—at the beginning of the COVID-19 crisis. Although this meta-research project is based on small samples (owing to the first phase of the COVID-19 crisis and the fact that adequate prevention research had to be ‘established’ at this stage) and a limited external validity (owing to the dynamic of the COVID-19 pandemic) concealing these findings would be contra productive regarding pandemic preparedness to next disease outbreaks, particularly by ‘other’ viruses causing epidemics or pandemics. We also believe that the established methodology related to meta-research studies within this publication will be beneficial for other researchers, epidemiologists and/or different stake holders when conducting research on research (meta-research) to investigate the quality and characteristics of current evidence synthesis and to support evidence-based clinical and health policy decision-making.
Conclusion
Most research on preventive measures in form of evidence synthesis was conducted in Asia, Europe and South America and addressed public health interventions. Furthermore, we found that almost 20% (9/51) of the published systematic reviews on prevention identified in COVID-19 L-OVE were empty, implicating the lack of primary studies at this early stage of the pandemic.
To improve cooperation strategies among research groups and also between policymakers worldwide, our meta-research indicated that it is important to investigate reasons for not publishing initiated research projects in more detail. Moreover, we would like to stress the importance of investigating deviations between what was originally predefined and planned by using, for example, data from the PROSPERO register and/or other platforms or published review protocols and what was finally reported in the published systematic review (ie, investigating reporting bias and dissemination bias) using larger samples.
Although preprints allow fast dissemination particularly in pandemic situations, such form of publication need to be handled cautiously owing to the lacking peer review and authors need to be strongly encouraged to publish their findings also in a peer-reviewed journal.
Overall, PROSPERO is a platform that enables communication among authors of systematic reviews worldwide. Therefore, we feel it is important to encourage authors of systematic reviews to register their research in PROSPERO, to keep the PROSPERO entry updated and clearly describe potential deviations between the PROSPERO entry (or any other published protocol) and the published full systematic review.
Data availability statement
All data relevant to the study are included in the article or uploaded as supplementary information.
Ethics statements
Patient consent for publication
Ethics approval
This study did not receive nor require ethics approval, as it does not involve human & animal participants.
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Contributors JN, JJM and CS conceptualised the aim of this meta-research project and designed the methodology. JN, JS and CS were involved in screening, data extraction and data synthesis. JN prepared the original draft. JS, WS, JJM and CS contributed to the refinement (review and editing) of the draft. CS supervised all steps of this meta-research project. JJM obtained the financial support supporting this study. All the authors reviewed and agreed to the final version being submitted. CS is guarantor.
Funding This work was supported as part of the CEOsys project by Federal Ministry of Education and Research (BMBF), Germany, grant number 01KX2021.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.