Article Text

Original research
Comparative-effectiveness research of COVID-19 treatment: a rapid scoping review
  1. Ba Pham1,
  2. Patricia Rios1,
  3. Amruta Radhakrishnan1,
  4. Nazia Darvesh1,
  5. Jesmin Antony1,
  6. Chantal Williams1,
  7. Naveeta Ramkissoon1,
  8. Gordon V Cormack2,
  9. Maura R Grossman2,
  10. Melissa Kampman3,
  11. Milan Patel4,
  12. Fatemeh Yazdi1,
  13. Reid Robson1,
  14. Marco Ghassemi1,
  15. Erin Macdonald1,
  16. Rachel Warren1,
  17. Matthew P Muller1,5,
  18. Sharon E Straus1,6,
  19. Andrea C Tricco1,7
  1. 1Li Ka Shing Knowledge Institute, St Michael's Hospital, Unity Health Toronto, Toronto, Ontario, Canada
  2. 2David R. Cheriton School of Computer Science, University of Waterloo, Waterloo, Ontario, Canada
  3. 3Epidemiology and Evidence Evaluation for Safety and Effectiveness Section, Health Canada, Ottawa, Ontario, Canada
  4. 4Vaccine Supply and Assurance, Public Health Agency of Canada, Ottawa, Ontario, Canada
  5. 5Department of Medicine, University of Toronto, Toronto, Ontario, Canada
  6. 6Department of Geriatric Medicine, University of Toronto, Toronto, Ontario, Canada
  7. 7Epidemiology Division and Institute of Health Policy, Management, and Evaluation, Dalla Lana School of Public Health, University of Toronto, Toronto, Ontario, Canada
  1. Correspondence to Andrea C Tricco; Andrea.tricco{at}


Objectives The COVID-19 pandemic has stimulated growing research on treatment options. We aim to provide an overview of the characteristics of studies evaluating COVID-19 treatment.

Design Rapid scoping review

Data sources Medline, Embase and biorxiv/medrxiv from inception to 15 May 2021.

Setting Hospital and community care.

Participants COVID-19 patients of all ages.

Interventions COVID-19 treatment.

Results The literature search identified 616 relevant primary studies of which 188 were randomised controlled trials and 299 relevant evidence syntheses. The studies and evidence syntheses were conducted in 51 and 39 countries, respectively.

Most studies enrolled patients admitted to acute care hospitals (84%), included on average 169 participants, with an average age of 60 years, study duration of 28 days, number of effect outcomes of four and number of harm outcomes of one. The most common primary outcome was death (32%).

The included studies evaluated 214 treatment options. The most common treatments were tocilizumab (11%), hydroxychloroquine (9%) and convalescent plasma (7%). The most common therapeutic categories were non-steroidal immunosuppressants (18%), steroids (15%) and antivirals (14%). The most common therapeutic categories involving multiple drugs were antimalarials/antibiotics (16%), steroids/non-steroidal immunosuppressants (9%) and antimalarials/antivirals/antivirals (7%). The most common treatments evaluated in systematic reviews were hydroxychloroquine (11%), remdesivir (8%), tocilizumab (7%) and steroids (7%).

The evaluated treatment was in favour 50% and 36% of the evaluations, according to the conclusion of the authors of primary studies and evidence syntheses, respectively.

Conclusions This rapid scoping review characterised a growing body of comparative-effectiveness primary studies and evidence syntheses. The results suggest future studies should focus on children, elderly ≥65 years of age, patients with mild symptoms, outpatient treatment, multimechanism therapies, harms and active comparators. The results also suggest that future living evidence synthesis and network meta-analysis would provide additional information for decision-makers on managing COVID-19.

  • COVID-19
  • Clinical trials
  • scoping review
  • knowledge synthesis
  • evidence synthesis

Data availability statement

Data sharing not applicable as no datasets generated and/or analysed for this study. Data sharing is described in the Data repository paragraph at the end of the Methods section.

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See:

Statistics from

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Strengths and limitations of this study

  • Broad literature search and study selection yielded 915 study reports, including 616 relevant studies (188 randomised controlled trials) and 299 evidence syntheses.

  • Detailed charting of study populations, interventions and outcomes of included studies and reviews were conducted to analyse characteristics and trends in the included literature and to elucidate lessons for future research.

  • Practical implications for future research with respect to study design, populations, interventions, comparators, outcomes and methodological approaches were identified.

  • Semiautomation approach to study selection, allowing for a very broad literature search and screening approximately 290 000 titles/abstracts in about 40 person-hours over 2.3 weeks.

  • This is a scoping review and as such, we did not assess the risk of bias of the included studies and evidence syntheses.


The current global pandemic of COVID-19 has resulted in a high burden of disease and mortality worldwide.1 2 The lack of effective treatments for COVID-19 has resulted in the almost constant production of studies and evidence syntheses evaluating potential treatment options, as illustrated by thousands of study protocols in clinical trial registries and hundreds of review protocols in systematic review registries.3 4 Attempts to synthesise this evidence thus far have resulted in various scoping reviews focusing on single drugs or isolated drug classes.5–9 Better understanding of the characteristics of study populations, treatments and outcomes of this research is a prerequisite to the design and conduct of future comparative-effectiveness research.

The objective of this rapid scoping review was to provide an overview of the characteristics of studies examining COVID-19 treatment.


The conduct of the rapid scoping review was guided by the JBI (formerly Joanna Briggs Institute) guide for scoping reviews, alongside the World Health Organization (WHO) guide to rapid reviews.10 11 Compared with a scoping review, we used streamlined methods in this rapid scoping review (eg, single reviewers conducted study selection). An integrated knowledge translation approach was used to engage with the knowledge users from Health Canada (MK) and Public Health Agency of Canada (MP) throughout the conduct of the rapid scoping review, including during: research question development, literature search, study inclusion, interpretation of results and draft report. The protocol for the review was registered using the Open Science Framework ( The discussion section includes minor amendments that occurred to the conduct of the review from the original protocol. Reporting of results was guided using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension to Scoping Reviews statement.12 Our research question was ‘What evidence exists on the treatments for COVID-19 in primary studies and reviews’, which is appropriate for the scoping review methodology.13

Patient and public involvement

Since this work was carried out as part of a rapid response to the COVID-19 pandemic project, timelines did not allow for participation of any patients or members of the public in this rapid scoping review.

Literature search

Comprehensive literature searches and citation screening were used in combination to gather relevant evidence from MEDLINE, EMBASE and preprint servers (biorxiv/medrxiv).14 The literature was initially searched from inception to 21 May 2020 and subsequently updated to 15 May 2021. Titles/abstracts were identified for screening using the Continuous Active Learning (CAL) tool, which uses supervised machine learning (see online supplemental appendix 1 for the description and performance of the tool).14 For archives that could be retrieved in their entirety (eg, MEDLINE, preprint servers), the CAL tool applied broad relevant search terms (online supplemental appendix 1). This search was supplemented by a literature search conducted by an experienced librarian in EMBASE (online supplemental appendix 2). The literature search was not restrict by language or publication status.

Eligibility criteria

The eligibility criteria followed the PICOS framework and consisted of:

  • Population: Individuals of any age who were clinically and/or laboratory diagnosed with COVID-19.

  • Intervention: Any compounds under investigation in human clinical trials as potential COVID-19 therapies (online supplemental appendix 3). Chinese medicine and complementary and alternative medicine—either alone or in combination with these medications—were excluded.

  • Comparator: Any of the interventions listed above, no intervention or placebo.

  • Outcomes: Any reported outcome.

  • Study designs: Primary studies of any design with a comparator group. Evidence syntheses of such studies were included, including systematic reviews, scoping reviews, rapid reviews, meta-analysis and overviews of reviews.

Study selection

A streamlined approach to study selection was used for the rapid scoping review. In combination with manual screening by reviewers, the CAL tool was used to identify and rank the titles and abstracts most likely to meet the inclusion criteria. This process continued iteratively until none of the identified articles met the inclusion criteria. For manual screening, a screening form based on the eligibility criteria was prepared for reviewers to aid in making consistent judgements on article relevance. A pilot-test was conducted using a random sample of 10 titles/abstracts until reviewers reached at least 75% agreement. Subsequently, screening was completed by single reviewers.

Data charting and coding

A charting form was developed and calibrated among the entire review team using two randomly selected full-text articles to ensure a standard approach to data collection. Following successful completion of the pilot-test, included studies were charted by single reviewers and verified by a second reviewer to ensure accuracy. Methodological quality or risk of bias appraisal of included studies was not conducted since this is a scoping review.10

The items collected included study characteristics (eg, study duration, study design, country of conduct), patient characteristics (eg, type of diagnosis, mean age), intervention and comparator details (eg, type of intervention, dose, frequency, duration) and outcome measures details (eg, mortality, viral clearance and hospital admission).

Pharmacological agents were grouped by their therapeutic category.15 Study primary outcomes were grouped together to reflect the clinical, virology, respiratory, inflammatory, cardiology and olfactory status and measures of COVID-19.16 17 The numbers of effect and harm measures were derived by counting the outcomes from the description of study outcomes. Authors’ conclusions were coded into the following categories: favour treatment, favour control, indeterminate and other.18 Pairs of reviewers conducted the data coding independently, with discrepancies reviewed and resolved through discussion by a pair of reviewers.


The charted and coded data were summarised descriptively for all patient population, interventions, comparators, outcomes and conclusion statements. The data were stratified by study design (randomised controlled trials vs non-RCT) and review type (review conducted according to a review protocol or otherwise).

Data repository

All material related to this review, including EndNote databases, extracted data in MS Excel, coding categories and analysis procedures written in the statistical software R are available at


Literature search

Figure 1 displays the literature search results. The semi-automation process with CAL and human reviewers allowed for the screening of approximately 290 000 titles/abstracts in about 40 person-hours over 2.3 weeks. Specifically, CAL identified 289 844 COVID-19 records and 4183 potentially relevant titles/abstracts. Title/abstract screening by reviewers resulted in 1542 potentially relevant reports. Report screening by reviewers resulted in 915 relevant reports, including 616 studies and 299 evidence syntheses. The list of included primary studies and evidence syntheses is in online supplemental appendix 4 and 5, respectively.

Figure 1

Flow diagram of included studies. Notes: *Estimated number of unique titles/abstracts based on: Medline (Ovid) includes preprints on COVID-19 from Medrxiv and Biorxiv, and large overlapping records between Medline and Embase. The flow chart was modified from the PRISMA 2020 statement.25

Characteristics of included studies

Figure 2 displays the timing when the studies were available online; on average 48 primary studies per month were published from July 2020 to April 2021. Table 1 displays the characteristics of the 616 included studies of varying design, including randomised controlled trials (188 studies (31%)), retrospective cohort studies (304 (49%)) and prospective cohort studies (70 (11%)), among others. The median study duration was 28 days and the median sample size was 169 participants. Public sources provided funding for about one-third of the studies; RCTs were funded often by private funding sources (27% relative to 3% for non-RCT). The primary studies were conducted in 51 countries, including the USA (26%), China (17%), Italy (8%), Spain (7%), France (6%), India (4%), Iran (3%), UK (3%) and Brazil (3%), among others (online supplemental table A1, online supplemental appendix 6).

Figure 2

Timing of available online of included studies*. The numbers of primary studies and systematic reviews for May 21 are higher because the literature search ended at 15 May 2021.

Table 1

Study, participant and outcome characteristics

Most studies were conducted with participants admitted to acute care hospital (84%). Participants were on average 60 years of age, including 61% male, and mostly with confirmed COVID-19 via PCR test (table 1). About one-third of the included studies enrolled participants with severe or critical COVID-19 conditions. Few studies (0.3%) enrolled children (eg,<16 years of age) or the elderly (eg,≥65 years of age, 2%). Figure A1 displays the cloud of words often used to describe the participants (online supplemental appendix 6). Typical words used were COVID-19, COVID-19 patients, hospitalised, severe, pneumonia, ICU, outpatient, respiratory distress, invasive mechanical ventilation, critically ill and supplemental oxygen, among others.

The median number of effect outcomes was four, and the corresponding number of harm outcomes was one (table 1). Common primary outcomes included death/survival (32% of the included studies), clinical status/measures (19%), virology status/measures (10%), respiratory status/measures (9%), safety/adverse events excluding death (7%) and composite outcomes involving death (6%, for example, intubation and death, or intensive care admission and death), among others.

The included studies evaluated 827 treatment arms (711 single-drug and 116 multiple-drug treatment arms) against 616 control arms, of which 106 (17%) control arms involved active comparators (table 2). The treatment arms consisted of 215 unique treatment options (online supplemental table A2, online supplemental appendix 6). The most common treatments were tocilizumab (11%), hydroxychloroquine (9%), convalescent plasma (7%), steroid (4%), lopinavir combined with ritonavir (4%), methylprednisolone (3%), remdesivir (3%), enoxaparin (2%), hydroxychloroquine combine with azithromycin (2%) and anakinra (2%), among others.

Table 2

Treatment options frequently evaluated in included studies

Table 2 also displays the common therapeutic categories of the evaluated treatment. The most common therapeutic categories were non-steroidal immunosuppressant (18%), steroid (15%), antiviral (14%), antimalarial (12%), anticoagulant (5%), convalescent plasma (8%), antibiotic (4%), anti-inflammatory (3%), interferon therapy (2%), antiparasitic (2%) and immunomodulatory (2%), among others (details in online supplemental table A3, online supplemental appendix 6). Common therapeutic categories involving multiple drugs were the combination of antimalarial/antibiotic (16%), steroid/non-steroidal immunosuppressant (9%), antimalarial/antiviral/antiviral (7%), 2-antivirals (4%) and antiviral/interferon (4%), among others (online supplemental table A4, online supplemental appendix 6).

Characteristics of included evidence syntheses

Figure 2 displays the timing when the evidence syntheses were available online, on average 22 reviews appeared each month from May 2020 to April 2021. Table 3 displays characteristics of the 299 included evidence syntheses, including 88 (29%) evidence syntheses and 211 (71%) evidence syntheses conducted with and without a review protocol, respectively. Commonly conducted evidence syntheses included systematic review with meta-analysis (63%), systematic review (24%), meta-analysis (4%, none mentioned the use of a review protocol), scoping review (3%) and rapid review (3%), among others. Most reviews (83%) included RCT and non-RCT studies. The median number of data sources was 5 and the median number of included studies was 14. The evidence syntheses were conducted in 39 countries, including the USA (19%), China (14%), India (11%), Iran (6%) and the UK (6%), among others (online supplemental table A5, online supplemental appendix 6).

Table 3

Evidence synthesis characteristics

The evidence syntheses evaluated 518 treatment arms against 299 control arms (table 4). The treatment arms consisted of 115 unique treatment options (online supplemental table A6, online supplemental appendix 6). The most common treatment options were hydroxychloroquine (11%), remdesivir (8%), tocilizumab (7%), steroids (7%), convalescent plasma (6%) and lopinavir/ritonavir (5%), among others (table 4 and online supplemental table A6, online supplemental appendix 6).

Table 4

Treatment options evaluated in systematic reviews

Treatment evaluation according to authors’ conclusion

Table 5 displays the results of the treatment evaluation according to authors’ conclusion. Among the included primary studies and evidence syntheses, the conclusion was in favour of treatment in 50% and 36% of the evaluated treatment arms, respectively.

Table 5

Treatment evaluation according to authors’ conclusion


We completed a rapid scoping review for Health Canada and Public Health Agency of Canada to identify pharmacologic treatments for COVID-19. A comprehensive search of electronic databases, trial registries and other grey literature sources from inception to May 2020 identified 9 controlled trials and 19 cohort studies with approximately 8000 participants. Updated to 15 May 2021, the search of electronic databases identified 915 relevant reports, including 616 studies with approximately 15.4 million participants and 299 evidence syntheses.

With respect to study population, existing studies put much emphasis on adult patients admitted to acute care hospitals. Future studies need to focus on children, older adults aged ≥65 years and patients with mild symptoms in community settings. Future study populations will need to reflect a broader range of age groups as the current pandemic evolves to affect younger age groups.19 20

With respect to treatment, many studies and reviews evaluated antimalarial agents. Existing studies emphasised preventing and treating cytokine surge with steroids and non-steroidal immunosuppressants, including interleukin‐6 inhibitors (eg, tocilizumab, sarilumab), interleukin‐1 antagonist (eg, anakinra), anti‐IL‐1β monoclonal antibody (eg, canakinumab), TNF‐alpha inhibitor (eg, adalimumab) and Janus kinase inhibitors (eg, baricitinib, ruxolitinib). Future studies may need to explore treatment for patients not responding to these agents, such as immunomodulators (eg, thymosin-α1). Existing studies put much emphasis on monotherapy; future studies need to evaluate combination therapy that addresses the multiple aspects of COVID-19, such as virology, respiratory, inflammatory and cardiology. Future studies may also need to explore outpatient treatment for patients with mild symptoms, and treatment options not frequently evaluated in existing studies, such as therapeutic anticoagulants.

With respect to comparators, most existing randomised controlled trials used placebo comparators while most observational studies used standard of care as comparator; future studies may consider active treatment as comparators, especially when evaluating treatments aiming to produce incremental improvement against effective treatments. Methodological issues related to the selection and delineation of comparators in studies evaluating combination therapies deserve attention. For example, a study evaluated multimechanism approach with medications targeting early immunomodulation, anticoagulation, and viral suppression to prevent catastrophic cytokine release syndrome encountered large variation in clinical characteristics of study participants and standard-of-care comparators in the five participant hospitals in two countries, including differences in disease severity and different doses of colchicine and types of steroids used across comparative groups.17

With respect to outcomes, about one-third of the included studies used mortality as the primary outcome. Tracking this outcome may require sufficiently long study duration, perhaps longer than the median duration of less than a month observed among existing studies, especially in patients with prolonged respiratory problems, suggesting longer follow-up duration for future studies. Of note, few existing studies used composite endpoints involving death, including endpoints such as intubation and intensive care admission. This use seems to be particularly suitable to capture the respiratory, immunology and cardiovascular aspects of COVID-19, as well as mortality. Few existing studies focused on harms due to treatment and among those that evaluated benefits and harms, the median number of reported harms was only one; future studies need to put more emphasis on harm evaluation. Existing RCTs put much emphasis on the use of clinical status/measures as primary outcome measures. Future trials may consider other primary outcomes that are relevant to patients, such as pneumonia, acute respiratory distress syndrome, multiorgan failure and septic shock, among others.

With respect to study design, our results showed a breakdown of 30% and 70% for RCTs and observational studies, respectively. Future trials are needed for evaluating combination therapies. Observational studies will remain pertinent in the evaluation of combination therapies, especially when rich data becomes available with their use in practice. Our review excluded qualitative studies, but we wish to emphasise the importance of these studies in elucidating the experience of COVID-19 patients.

With respect to evidence synthesis, we identified a small number of meta-analyses conducted without the associated systematic review and review protocol (n=13). This practice needs to be scrutinised because of the associated high risk of bias in the results, which could be wrong, but appeared to be convincingly precise.21 Existing evidence syntheses mostly evaluated monotherapy; future evidence syntheses will need to include data from the evaluation of combination therapy. The number of existing network meta-analyses was low (n=4); future network meta-analyses are needed to identify effective treatment given a plethora of treatment options, as well as to identify effective component treatment options addressing multiple aspects of COVID-19.22 Given the growing literature, there is a definitive need for living evidence synthesis, in which the synthesis is updated regularly as new studies become available.23 The results suggest that monthly updates may become necessary.

With respect to the growing literature, the use of automation tools like CAL for study selection will become essential to ensure a highly sensitive yield of relevant studies, responsive timelines for decision-making and reduced workload for reviewers. In this rapid scoping review, we used a continuous active learning approach that integrates machine learning with feedback instructions from reviewers. This approach allowed the screening of approximately 290 000 titles/abstracts in about 40 person-hours over 2.3 weeks. We believe this approach is indispensable for future reviews involving large body of literature. This approach called for slight changes in our review conduct and reporting, of note the reported number of the titles/abstracts excluded by the automation tool in the flow chart (see figure 1).

There are several limitations of this review. This is a scoping review, and as such, we did not assess the risk of bias in the included studies and reviews. Initially, the review protocol called for a borrowing strength of evidence approach, including studies evaluating treatment for SARS and MERS. The initial literature search in May 2020 included electronic databases, trial registries, Cochrane Library and other grey literature sources. Given the growing literature on COVID-19 by May 2021, the current review was focused only on COVID-19 treatment, with relevant studies identified from MEDLINE, EMBASE and preprint servers.

In this scoping review, the evaluated treatment options appeared to attain a reasonable chance of being more effective than their comparators, approximately 50% and 30% according to the authors’ conclusions from the included studies and reviews, respectively. However, we did not extract outcome data or combine them to verify the authors’ conclusions. To provide a broad overview of the comparative effectiveness research on COVID-19 treatment, we included reports from preprint servers, but these reports had not gone through peer review. Despite these limitations, the methods used in this review were carefully selected to address the needs of our knowledge users from Health Canada and Public Health Agency of Canada. In addition, we made the material from this rapid scoping review available in an online data repository as the data may be useful for conducting systematic reviews of specific therapies or for updating the current review.24


This rapid scoping review characterised a growing body of comparative-effectiveness studies and evidence syntheses evaluating hundreds of monotherapy and combination therapy options addressing the multiple sequelae of COVID-19. The results suggest future studies in children, elderly (eg, ≥65 years of age) and patients with mild symptoms, with additional data on outpatient treatment, multimechanism therapy, harms and active comparators. The results also suggest that future living evidence synthesis and network meta-analysis would provide additional information for decision-makers on managing COVID-19.

Data availability statement

Data sharing not applicable as no datasets generated and/or analysed for this study. Data sharing is described in the Data repository paragraph at the end of the Methods section.

Ethics statements

Patient consent for publication

Ethics approval

This research is exempt from ethics approval because the work is carried out on published documents.


The authors would like to thank Jesse McGowan for her assistance in developing literature searches, Alissa Epworth for her assistance executing searches and retrieving articles, and Krystle Amog and Navjot Mann for their assistance in formatting this manuscript.


Supplementary materials

  • Supplementary Data

    This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.


  • Contributors PR and BP analysed the data, interpreted the results and drafted the original and revised manuscript, respectively. ACT and SES conceived and designed the study, helped obtain funding, interpreted the results and helped write sections of the manuscript. GVC and MRG provided methodological and technical support and edited the manuscript. AR, ND, JA and FY coordinated the review, screened citations and full-text articles, abstracted data, resolved discrepancies and edited the manuscript. MK, MP and MPM helped conceive the study, provided methodological support and content expertise and edited the manuscript. RR and MG provided methodological support, screened citations and full-text articles and assisted with drafting the manuscript. CW, NR, EM and RW screened citations and full-text articles, abstracted data and assisted with data analysis. ACT acted as a guarantor of the work, accepted full responsibility for the work and/or the conduct of the study, had access to the data, and controlled the decision to publish the work. All authors read and approved the final manuscript.

  • Funding This work was supported through the Drug Safety and Effectiveness Network funded by the Canadian Institutes of Health Research (DMC-166263). SES is funded by a Tier 1 Canada Research Chair in Knowledge Translation (17-0245-SUB) and the Mary Trimmer Chair in Geriatric Medicine (award number is not applicable); ACT is funded by a Tier 2 Canada Research Chair in Knowledge Synthesis (17-0126-AWA).

  • Disclaimer The funders had no involvement in the design, conduct, or publication of this study.

  • Competing interests None declared.

  • Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.