Article Text

Original research
How missing evidence-based medicine indicators can inform COVID-19 vaccine distribution policies: a scoping review and calculation of indicators from data in randomised controlled trials
  1. Andrew Larkin1,
  2. Howard Waitzkin1,2,
  3. Ella Fassler1,
  4. Kesavan Rajasekharan Nayar3,4
  1. 1 Allende Program in Social Medicine, Albuquerque, New Mexico, USA
  2. 2 Locum Tenens Program, Health Sciences Center, and Department of Sociology, University of New Mexico, Albuquerque, New Mexico, USA
  3. 3 Santhigiri Research Foundation, Thiruvananthapuram, Kerala, India
  4. 4 Global Institute of Public Health, Thiruvananthapuram, Kerala, India
  1. Correspondence to Dr Howard Waitzkin; waitzkin{at}unm.edu

Abstract

Objective Reports of efficacy, effectiveness and harms of COVID-19 vaccines have not used key indicators from evidence-based medicine (EBM) that can inform policies about vaccine distribution. This study aims to clarify EBM indicators that consider baseline risks when assessing vaccines’ benefits versus harms: absolute risk reduction (ARR) and number needed to be vaccinated (NNV), versus absolute risk of the intervention (ARI) and number needed to harm (NNH).

Methods We used a multimethod approach, including a scoping review of the literature; calculation of risk reductions and harms from data concerning five major vaccines; analysis of risk reductions in population subgroups with varying baseline risks; and comparisons with prior vaccines.

Findings The scoping review showed few reports regarding ARR, NNV, ARI and NNH; comparisons of benefits versus harms using these EBM methods; or analyses of varying baseline risks. Calculated ARRs for symptomatic infection and hospitalisation were approximately 1% and 0.1%, respectively, as compared with relative risk reduction of 50%–95% and 58%–100%. NNV to prevent one symptomatic infection and one hospitalisation was in the range of 80–500 and 500–4000. Based on available data, ARI and NNH as measures of harm were difficult to calculate, and the balance between benefits and harms using EBM measures remained uncertain. The effectiveness of COVID-19 vaccines as measured by ARR and NNV was substantially higher in population subgroups with high versus low baseline risks.

Conclusions Priorities for vaccine distribution should target subpopulations with higher baseline risks. Similar analyses using ARR/NNV and ARI/NNH would strengthen evaluations of vaccines’ benefits versus harms. An EBM perspective on vaccine distribution that emphasises baseline risks becomes especially important as the world’s population continues to face major barriers to vaccine access—sometimes termed ‘vaccine apartheid’.

  • COVID-19
  • public health
  • health policy

Data availability statement

Data are available upon reasonable request. All the data used in this study are publicly available online and are properly cited. More detailed information about obtaining access to the data will be provided on request.

http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

Statistics from Altmetric.com

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

STRENGTHS AND LIMITATIONS OF THIS STUDY

  • The study uses a scoping review methodology to determine the extent to which publications about COVID-19 vaccines have analysed indicators from evidence-based medicine (EBM)–absolute risk reduction (ARR), number needed to be vaccinated (NNV), absolute risk of the intervention (ARI) and number needed to harm (NNH)–to assess and compare benefits and harms; the study also presents calculations of these rarely discussed EBM indicators based on data from major randomised controlled trials.

  • The methods of calculating these key EBM indicators facilitate a reproducible technique to compare benefits and harms for assessment of vaccines previously developed and those to be developed in the future.

  • Because sensitivity analyses permit calculation of ARR and NNV in subpopulations with different baseline risks and with varying assumptions about vaccine effectiveness, the approach can supplement consideration of other contextual risk factors such as age, immunocompromised status and comorbidities in policy decisions about priorities for vaccine distribution.

  • Measures of harm and comparisons of harms versus benefits by calculation of ARI and NNH were challenging to achieve based on publications or accessible unpublished data, so we were not yet able to resolve the question of benefits versus harms through these EBM indicators.

  • Partly because we could analyse only published data from the major publications but not the inaccessible raw data, we could not determine if there was a rationale for not reporting ARR and NNV in addition to relative risk reduction, or for not reporting quantitative EBM measures of harm such as NNH.

Introduction

Among its challenges, the COVID-19 pandemic has worried some people who have tried to promote evidence-based medicine (EBM) and analysis of complex systems.1 For instance, EBM teaches that absolute risk reduction (ARR) and the number needed to be treated or vaccinated (NNT/NNV), in addition to relative risk reduction (RRR), are fundamental measures in evaluating new clinical interventions such as vaccines. Moreover, to compare benefits and harms from an intervention, the number needed to harm one person (NNH), calculated from the absolute risk of an intervention (ARI), can be compared with the NNT or NNV. In a classic statement of this principle, leaders of EBM argued:

… we conclude that the absolute risk reduction is superior to the relative risk reduction because it incorporates both the base-line risk and the magnitude of the risk reduction. Its reciprocal, the number needed to be treated, expresses the absolute risk reduction in a manner that is easily understood by clinicians, and can be used to describe the harm as well as the benefits of therapy and other clinical maneuvers.2

A widely used textbook on EBM, in its 2019 edition, similarly favours the reporting of ARR: ‘Thus, the ARR is a more meaningful measure of treatment effects compared with the RRR’.3 Likewise, a publication of the US Food and Drug Administration recommends reporting both ARR and RRR in assessments of treatment or vaccine efficacy.4 Medical and public health journals often encourage or require reporting absolute differences.

Despite these recommendations, publications of the major COVID-19 vaccine evaluations, both randomised controlled trials (RCTs) and population-based assessments, have reported findings about RRR but not ARR, NNV, ARI or NNH. These publications have not compared benefits and harms using NNV and NNH. This lack remains puzzling because evaluations of other vaccines such as influenza have emphasised these indicators.5 6

Certain concerns about absolute measures have arisen. Some modelling studies show that the targets for NNV in prior vaccination programmes have varied and that this indicator does not fully capture the effects of vaccination on transmission and herd immunity.7–9 Reducing the potential for transmission, even without achieving herd immunity, can inform policy decisions if this more limited effect flattens the epidemic curve and reduces pressure on the health system. Absolute measures of risk reduction and harms also are context dependent. That is, these measures may vary in relation to subpopulations’ characteristics, such as infection incidence and prevalence, background immunity, and proportion of people with advanced age, immunocompromised status, and comorbidities. Such variations linked to context have attracted concerns about the use of absolute measures to analyse benefits and harms in consistent ways across subpopulations.

However, the absolute measures actually may help clarify that vaccines’ effectiveness varies across subpopulations with different characteristics, especially varying baseline risks of infection. Several studies have clarified that vaccines’ effectiveness, estimated by NNV, can vary in geographical areas with different baseline risks and therefore can inform policies about vaccine distribution.10 11 From this perspective, the context-dependent character of absolute measures may create opportunities to use these EBM indicators in clarifying how the distribution of disease and vaccines’ effectiveness differ depending on variability in subpopulations’ contextual conditions.

Such an approach also facilitates assessment of changes in ARR and NNV over the course of epidemics, for instance, when new variants of viruses emerge, as investigators have done with influenza virus.10 11 A similar approach to ARR and NNV could provide important information about changes in absolute measures of risk reduction for COVID-19 occurring with new variants like Delta and Omicron.

These efforts may lead to more helpful recommendations about vaccine distribution. During epidemics and pandemics, situations change quickly, so data and conclusions rapidly become out of date. One goal of this work is to clarify a conceptual and methodological approach to decisions about vaccine distribution, based partly on EBM indicators, that may prove helpful over time. Such recommendations to address subpopulations’ ARR and benefit–harm comparisons supplement but do not replace considerations about the previously identified high-risk factors that warrant emphasis in immunisation campaigns, including age, immunodeficiencies, comorbidities and occupation.

Because of these concerns, we have addressed the following questions:

  1. What are the RRRs and ARRs achieved by available vaccines, and what is the NNV in order to prevent one symptomatic infection?

  2. To assess benefits versus harms, what is the NNH for the vaccines, compared with NNV?

  3. How does the NNV vary in populations with different baseline risks of disease, and what are the implications for global health strategies of vaccine distribution?

Beyond trying to answer these three questions, we aimed to compare results for COVID-19 vaccines with those of prior vaccines. Finally, we hoped to prepare a simple explanation of our findings to assist in public educational efforts, informed consent procedures and global health policy decisions about vaccine distribution.

Materials and methods

Scoping review

To assess prior work on these questions, we initially planned to conduct a full systematic review of the literature. However, two problems impeded a systematic review according to current methodological standards. First, the number of studies that reported the EBM indicators, or that provided data from which we could calculate the indicators, was too small to synthesise quantitatively or through a narrative summary. Second, the data needed to calculate several key EBM measures, especially those measuring harms, were not reported in the major publications about vaccine efficacy and effectiveness, and we could not obtain the raw data from these studies due to restrictions on data access imposed by the investigators and/or sponsoring pharmaceutical corporations. Editors of BMJ have noted the adverse impact of these restrictions limiting open access to data from these RCTs.12 13 After consultations with several leaders in this field, we concluded that a systematic review could not achieve valid scientific conclusions, given deficiencies in available data.

Instead, we reached a decision that a scoping review, with an objective of clarifying the scope of the extant literature using EBM indicators, would be a more appropriate review method at this time, and that a full systematic review might become feasible if and when the above problems were resolved. This decision adheres to editorial recommendations accompanying the initial publication presenting consensus-based recommendations about scoping reviews: ‘The scoping process allows organizations to consider whether a potential topic is already covered by existing reviews, determine whether the evidence is too scarce to allow for a systematic review and where primary research is needed, and identify any areas within the broader topic where a systematic review may be appropriate.’14

For the scoping review, we prepared a protocol using the checklist that is part of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Extension for Scoping Reviews (PRISMA-ScR), based on the PRISMA guideline for systematic reviews.15–19 We registered the protocol (https://osf.io/search/?q=waitzkin&page=1). Figure 1 presents a flow diagram prepared according to PRISMA-ScR recommendations.

Figure 1

PRISMA-ScR flow diagram outlining literature search and results of screening process for scoping reviews. PRISMA-ScR: checklist and explanation.15 PRISMA-ScR, Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews.

Inclusion criteria required that publications contain at least one specified search term and involve human patients of any age, gender or race/ethnicity in any setting. We excluded in vitro studies, animal studies and protocol-only publications. Application of the inclusion and exclusion criteria involved independent assessment by two authors (AL, HW) and resolution of differences through discussion. We introduced no filters into the searches.

During April and May 2021, we searched PubMed for articles combining COVID and SARS-CoV-2 separately with each of the following terms, spelled out and as abbreviations: “ARR”, “RRR”, “NNT”, “NNV”, “NNH”, “risk”, “harm” and “benefit”. Later, during July 2021, we repeated the above search strategy and extended it to include PubMed, Epistemonikos COVID evidence, the Cochrane Library of Systematic Reviews, the Cochrane COVID Study Register, EMBASE, CINAHL and the WHO COVID database. In October 2021 and August 2022, we repeated the search strategy using the same sources. We supplemented these search efforts by using two general search engines, DuckDuckGo and Google, with the same search terms.

To address possible sources of bias, we requested that experienced reference librarians at the University of New Mexico Health Sciences Library and Informatics Center do an independent search of the literature; they found no additional publications beyond those located through our searches. We also corresponded by email with colleagues at two centres of EBM at McMaster University and Oxford University, who provided no information about publications coming from their own or other groups concerning EBM measures in relation to COVID-19 vaccines.

Using notes and a spreadsheet, the same two authors separately read and assessed each article to clarify answers, if any, to the three questions above. To summarise each article, we used the abstract if available; if not, we prepared a summary of the article. After our separate analyses, we communicated about the findings and resolved differences in interpretation of five publications through discussion.

Few articles involved actual research, so ranking all publications with measures to assess research quality would have been inappropriate. (The above editorial on scoping review methods states: ‘Consequently, scoping reviews do not include meta-analysis or assess the strength of evidence across studies. Instead, they chart concepts, themes, and the amount and type of evidence available.’14). We and our consulting colleagues agreed that a predetermined rating scheme for methodological quality, applied in a standardised fashion to all publications, was unlikely to reveal sufficient number of studies or variation in quality to yield valid quantitative analyses. For that reason, we elected to categorise the publications by assessing type of publication as follows: full peer-reviewed articles analysing EBM measures in published RCTs and population-based assessments of COVID-19 vaccines; brief commentaries or perspectives presenting opinions; letters to the editor; blog posts and items in the general online media. This categorisation conveyed an implicit ordering of the publications in terms of comprehensiveness and peer review assessment, rather than ranking the quality of research methodology as would be expected in a systematic review.

Assessment of vaccines

For our assessment of vaccine trials, we applied the definitions in box 1, which were developed in EBM and in systematic reviews of vaccine evaluations.2 3 20–24 We used these EBM measures to analyse reports from three early and influential studies. Two studies, concerning the BNT16262 (Pfizer-BioNTech) and mRNA-1273 (Moderna-NIH) vaccines, were large, multinational RCTs showing vaccine efficacy.25 26 The third study, an effectiveness trial that showed the ‘real-world’ impact of vaccine on population-level outcomes, was implemented at the largest healthcare organisation in Israel.27

Box 1

Definitions and abbreviations of evidence-based medicine used in this study (listed alphabetically)

  • Absolute risk (AR): the rate of an event (for instance, symptomatic COVID-19 infection) in a group.

  • Absolute risk in the control group (ARc): the rate of an event in the group not receiving an intervention.

  • Absolute risk in the intervention group (ARi): the rate of an event in the group receiving an intervention.

  • Absolute risk reduction (ARR): a calculation that shows the difference from a group’s baseline event rate between those receiving and not receiving an intervention (ARc−ARi); that is, it indicates the reduction of risk from the group’s baseline risk that results from the intervention. If c=events in control group, v=events in vaccinated group, Nc=the sample size of the control group and Nv=the sample size of the vaccination group, then ARR=(c/Nc)−(v/Nv).

  • Absolute risk of the intervention (ARI): the rate of the intervention’s adverse effects. ARI is calculated by subtracting the rate of adverse effects in the group not receiving an intervention from that in the group receiving an intervention.

  • Baseline risk: the total number of cases per population in a group not exposed to an intervention (such as vaccination) during a specified time period. In a randomised controlled trial, baseline risk is the proportion or percentage of study participants in the control group for whom a specified event, such as COVID-19 infection, is observed. In a community, baseline risk can be estimated by incidence (number of new cases per population over a defined period of time) among the unvaccinated population.

  • Evidence-based medicine (EBM): ‘the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients.’23

  • Number needed to be vaccinated (NNV): a calculation that shows the number of people needed to be vaccinated in order to prevent a pertinent event in one person, such as a symptomatic infection by COVID-19. NNV is the reciprocal of ARR=1/ARR.

  • Number needed to harm (NNH): a calculation that shows how many people need to be vaccinated to cause harm for one person. NNH is the reciprocal of ARI=1/ARI.

  • Relative risk reduction (RRR): a calculation that shows the difference in event rate between those receiving and not receiving an intervention, expressed as a percentage of the event rate among those not receiving an intervention: (ARR/ARc)=(ARc−ARi)/ARc. If c=events in control group, v=events in vaccinated group, Nc=sample size of the control group and Nv=sample size of the vaccination group, RRR=[(c/Nc)−(v/Nv)]/(c/Nc).

  • Risk–benefit (also known as harm–benefit) analysis: an analysis that compares the risks (harms) of an intervention (for instance, NNH) with the benefits of an intervention (for instance, NNV).

References: 2 3 20–24

Using published data from each study, we calculated ARR, NNV, ARI and NNH, although none of the studies reported these indicators as such. Based on recommendations in the literature, we calculated 95% confidence intervals (CIs) for each of these measures.28–30 Later, as reports for new vaccines became available from RCT efficacy studies for the GamCovidVac (Gamaleya), Ad26.COV2.S (Johnson & Johnson) and ChAdOx1 nCoV-19 (AstraZeneca-Oxford) vaccines,31–33 we repeated this approach to calculate and to analyse the same EBM indicators. We then compared benefits and harms, using NNV and NNH.

To clarify implications for policies about vaccine distribution and accessibility, we performed sensitivity analyses that considered ARR and NNV in regions with differing baseline risks of disease. The geographical areas included counties in the USA and states or territories in India (we have a special interest in these areas due to our work in public health). During May 2021, we obtained the baseline risks for selected US counties from the Johns Hopkins University dashboard and for selected states and territories of India from the New York Times dashboard.34 35

As an endpoint for the effectiveness of vaccines in subpopulations, we initially focused on the prevention of symptomatic COVID-19, which was the endpoint emphasised in the initial publications of the RCTs. The percentage of COVID-19 infections that become symptomatic and how it varies across geographical areas with differing baseline risks, to our knowledge, have not been determined definitively. In our calculations, we first assumed that about 50% of infected persons become symptomatic, with the following rationale. This percentage represents an approximate midpoint in the range of percentages of patients who become symptomatic. This range is from 25% to 80% in published narrative and systemic reviews.36 37 More recent systematic reviews and meta-analyses led to estimates of 64.9% and 67.6%.38 39 These studies varied in data sources and methodological procedures, raising challenges of synthesising the results. A reference librarian confirmed our literature review in finding no other studies that clarified definitively the proportion of infected people who become symptomatic.

Due to these concerns, we performed further sensitivity analyses that used the observed lower and upper limits for percentages of symptomatic infections: 25% and 80%. After the effectiveness of vaccines decreased over the course of the pandemic due to the emergence of the Delta variant, we repeated the sensitivity analyses during September 2021. We anticipated that the ARRs and NNVs would vary substantially under these different assumptions. For that reason, we planned to assess the rank orderings of the calculated ARRs and NNVs in the geographical subpopulations. In the analyses, we considered contextual components for which data were available, such as varying prevalence and vaccine effectiveness. However, because contemporary, accurate data about other important contextual components, such as age, immunodeficiency and comorbidities, were unavailable, multivariate modelling to consider the relative effects of these contextual conditions was not yet feasible.

In addition to preventing infection and symptoms, public health policy sought to avert more serious outcomes that challenge health systems’ resources, so we extended the calculations to include hospitalisation. Like the probability of symptom development, to our knowledge, the probability that infection will lead to hospitalisation remains uncertain, overall and for varying at-risk subpopulations. Many studies of hospitalised patients have determined the relative risks of hospitalisation for vaccinated versus unvaccinated patients. On the other hand, we and the reference librarians could find only two recent studies using population-based methods that involve random sampling with serological testing to assess risk for hospitalisation. In these studies, the infection hospitalisation rate (IHR) was 2.10% for unvaccinated people in Indiana and 6.86% for those in Connecticut.40 41 With these data, we calculated ARRs and NNVs for hospitalisation in the same geographical subpopulations and time periods as in the assessment of symptom development.

To place COVID-19 vaccines in a historical context, we compared available data with those achieved by prior widely used vaccines that previously achieved high levels of effectiveness. We focused on benefits as measured by NNV, calculated from publications concerning vaccines for COVID-19 and for several other viral diseases. For assessment of ARR/NNV, we searched for the most recent research studies that reported RCTs or systematic reviews. To reduce the likelihood of biased comparisons, we asked the reference librarians to confirm that other studies did not report findings about ARR or NNV that differed substantially from those that we reported.

For the study’s components other than the scoping review, we prepared a protocol using the Strengthening the Reporting of Observational Studies in Epidemiology checklist, which pertains to observational studies.42 We also registered this checklist (https://osf.io/search/?q=waitzkin&page=1).

Patient and public involvement

In designing the study, we considered questions that we have received frequently from patients, colleagues and members of our communities. We also shared and requested feedback about the preliminary design with participants of two large list serves targeting people in public health and social medicine. From the beginning, we tried to illustrate how these concepts, findings and policy implications can be explained simply to patients, policymakers and the public. We plan to disseminate the findings widely to the public and policymakers through press releases, media contacts, institutional websites, blogs, list serves, personal communications and social media tools.

Results

Scoping review

We found few publications about COVID-19 vaccines that referred to key EBM indicators of benefits and harms (ARR, NNV, ARI and NNH). In our searches through August 2022, after duplicates were removed, a total of 23 citations were identified from searches of electronic databases. Consistent with our goal of inclusiveness in determining the overall scope of publications on EBM analyses of COVID-19 vaccines, after resolution of three disagreements about eligibility between the two reviewers, we did not exclude any of these citations. After screening, 18 full-length texts remained for qualitative synthesis. The searches using PubMed identified all citations found in other databases.

The scoping review led to several findings. All 18 publications considered ARR and NNV. Most publications commented in general terms about the lack of reporting about these EBM measures of risk reduction. Two commentary articles criticised the deficit in reporting and presented limited calculations about ARR and/or NNV based on data in published reports of RCTs.43 44 Four brief commentaries also presented limited calculations of ARR and NNV based on published reports and further interpreted these calculations.45–48 Two blog posts by a journal editor referred similarly to the deficit in reporting of key EBM indicators in the leading publications about the RCTs.49 50 A few letters to editors of medical journals (for instance,51–54) conveyed similar points. Two full peer-reviewed articles (with sections on background, methods, results and discussion) considered these issues further55 56; one of these was retracted and later again published after a new peer review in a different journal.55 In the general online media, a handful of critiques raised similar concerns. We found no publications that reported ARI or NNH, and no publications that attempted to compare benefits measured by NNV with harms measured by NNH. Likewise, no publications attempted to analyse how ARR and NNV vary in populations with different baseline risks of disease, or to discuss the implications of such analyses for strategies of vaccine distribution.

Assessment of vaccines for COVID-19

Risk reduction with vaccines

Table 1 shows data from the RCTs of the five major vaccines and from the Israeli population-based study, including both point estimates and 95% CIs. The RRRs for the Pfizer-BioNTech and Moderna-NIH vaccines were in the range of 95%. This means that, among all those who became ill, 95% were in the unvaccinated groups. The publications for these studies did not report ARRs, but data published in the studies permitted their calculation. Calculated ARRs for vaccinated subjects compared with the baseline risk for unvaccinated subjects were in the range of 1%. The NNV to prevent one symptomatic infection, calculated from the ARR, was 141 for mild COVID-19 and 2714 for severe COVID-19 (requiring hospitalisation) in the Pfizer-BioNTech study, vs 87 and 502 in the Moderna-NIH study. In the later reports about the Gamaleya, Johnson & Johnson and AstraZeneca-Oxford RCTs, although the endpoints varied somewhat, the RRRs, ARRs and NNVs showed approximately the same range of results as in the earlier Pfizer-BioNTech and Moderna-NIH findings.

Table 1

Risk reduction with major vaccines in major randomised controlled trials

In the study from Israel, RRRs were somewhat less impressive than in the RCTs (calculations based on figure 2 in the published report27): 34% for mild symptomatic disease, 58% for hospitalisation, 68% for severe symptomatic disease and 72% for death. The ARR from baseline risk was less than 1%. The NNV ranged from 490 for mild symptomatic infection, to 4004 for hospitalisation and 25 940 for death.

Harms versus benefits

Calculations of NNH, as well as comparisons of harms versus benefits using NNH and NNV, were challenging with available data. These studies did not report rates of harmful events consistently for vaccinated and control groups, and we were not able to ascertain denominators with numerical counts reported for some harmful events. The Pfizer-BioNTech study listed numerical counts for types of harm but provided comparative rates for vaccinated and control groups only for a few outcomes. In the Moderna-NIH study, the text presented numerical counts and percentages for some but not all adverse events. Methods of reporting harms differed across the RCTs of the Gamaleya, Johnson & Johnson and AstraZeneca-Oxford vaccines. The population-based study in Israel did not present data on measures of harm.

Despite these limitations, we were able to calculate ARI and NNH for some data, as shown in table 2. For minor adverse events, the ARIs in the Pfizer-BioNTech and Moderna-NIH studies were substantial, and the NNHs were less than 10. In the Pfizer-BioNTech study, the calculated ARI for a ‘serious adverse event’ was 0.1%, and the NNH was 1000, although the CI indicated that the difference between vaccination and control groups was not statistically significant. Considering ‘adverse events that were deemed by the trial team to be related to the vaccine or placebo’, the Moderna-NIH data showed a calculated ARI of 3.7% and an NNH of 27. For ‘treatment-related severe adverse events’ in the Moderna-NIH study, the ARI was 0.3% and the NNH was 333, again without a significant difference between vaccination and control groups. For data from the Gamaleya, Johnson & Johnson and AstraZeneca-Oxford vaccines, we were able to do these calculations only for serious adverse events. However, the Gamaleya investigators reported that no serious adverse events were ‘considered associated with vaccination’. In the Johnson & Johnson report, there was no significant difference in such events between vaccination and control groups. The AstraZeneca-Oxford publication reported only three severe adverse events, and no significant difference between groups.

Table 2

Measures of harm with major vaccines in major randomised controlled trials

Based on the NNVs and NNHs from these studies, the comparisons of benefits versus harms using EBM measures were equivocal. As expected, NNHs for minor adverse effects appeared substantial, but NNHs for serious adverse effects were inconclusive. Notably, some CIs, especially for serious harms, indicated that the point estimates were uncertain and that differences between harms and benefits were not statistically significant.28 29

Vaccine benefits in populations with differing baseline risks

From the sensitivity analyses for symptomatic infections, the ARRs and NNVs were substantially more favourable in areas with high population baseline risks, as determined by cases per population (recognising that this indicator often was an underestimate based on incomplete case identification) (table 3 and online supplemental table 3). Where the baseline risk was high, the ARR was likely to be larger and the NNV smaller than where the baseline risk was low. For instance, if the baseline risk of infection in a region was 1.12% (as shown for De Baca County, New Mexico, USA, during May 2021) and about 50% of positives became symptomatic without vaccine, the baseline risk of symptomatic disease was 0.56%. Assuming vaccine efficacy as measured by RRR was 95%, the risk of symptomatic disease after vaccination was (0.05×0.56%)=0.03%, the ARR was (0.56–0.03)=0.53%, and the NNV was (1/0.0053)=200. But if the baseline risk of infection was 0.08% (shown for Catron County, New Mexico, USA), the baseline risk of symptomatic disease was 0.04% without vaccine and was reduced to 0.00% with vaccine, for an ARR of 0.04% and NNV of 2500.

Supplemental material

Table 3

Absolute risk reductions and numbers needed to be vaccinated (NNVs) in geographical regions with different baseline risks, population-based data Symptom development

Table 3 shows calculations for these and other selected US counties and for states or territories of India with differing baseline risks. In regions with lowest baseline risks, the NNV became much larger than in regions with highest baseline risks. Thus, in regions of India with high baseline risks, the NNV fell to the range of 30. In the additional sensitivity analyses, which assumed that 25% and 80% of infected persons became symptomatic, we found, as expected, somewhat different ARRs and NNVs, but the overall rank ordering of the geographical areas remained the same as when we used the estimate of 50%. During September 2021, with vaccine effectiveness reduced to 66% due to the Delta variant,57 the ARR and NNV again varied substantially. Despite lower vaccine effectiveness, these indicators nevertheless became more favourable in some US subpopulations with high baseline risks, and the differences among the subpopulations remained sizeable. In India, the ARR and NNV became less favourable, as the pandemic improved and under-reporting of cases continued.

The results for hospitalisation revealed a similar pattern as those for symptomatic infection: calculation of ARR and NNV showed that vaccine effectiveness appeared higher in subpopulations with higher baseline risk (table 4 and online supplemental table 4). This finding held for both previously observed IHRs of 2.10% and 6.86%. The relationship appeared stronger during the period of Alpha variant predominance than Delta predominance, probably due to the assumption of lower vaccine efficacy against Delta, as well as possible declines in testing for some subpopulations. Notably, the NNV to prevent one hospitalisation varied widely, from the hundreds in some high-prevalence subpopulations to more than 10 000 in those with low prevalence.

Table 4

Absolute risk reductions and numbers needed to be vaccinated (NNVs) in geographical regions with different baseline risks, population-based data Hospitalisations

In the Discussion section, we consider the policy implications of this analysis for priorities in vaccine distribution.

Online supplemental appendix 1 depicts schematically the overall relationships among varying baseline risk and NNV for vaccines with different RRRs.

Comparisons with vaccines for other viral diseases

The ARRs and NNVs achieved by COVID-19 vaccines so far appear somewhat less favourable than those achieved by vaccines for some other viral diseases. For instance, systematic reviews of influenza vaccines have revealed NNVs to prevent symptomatic infections between 12 and 94.5 6 Studies of herpes zoster vaccine have yielded NNVs for symptomatic infections between 11 and 43.7 For human papilloma virus vaccine, calculated NNV to prevent one case of cervical intraepithelial neoplasia in one study was 129.7 58 Regarding smallpox, the NNV to prevent one death was 4 (table 1, calculated from data in59 60).

For such comparisons, limitations of NNV calculations include lack of consensus about targets for NNV across vaccines and inability to capture indirect benefits of vaccines regarding transmission and herd immunity, noted earlier, as well as questionable generalisability from studies conducted in specific geographical regions.7–9 Absolute measures call for analysis within an epidemiological context that includes variables such as prevalence, population vulnerability (age, immunocompromised status and comorbidities) and emergence of variants, among others. Such concerns limit the conclusiveness of comparisons suggesting that COVID-19 vaccines may reach somewhat lower levels of effectiveness than certain prior vaccines.

Discussion

Findings in context

From our scoping review, we found few publications that referred to ARR, NNV, ARI or NNH in research on COVID-19 vaccines. Lack of attention to these key elements of EBM is not unique to COVID-19 vaccines. A review of reporting practices in the medical literature showed that among 875 controlled trials, fewer than one-tenth reported at least one NNT/NNV or NNH, while slightly more than one-quarter reported at least one ARR.61 Inattention to EBM indicators is troubling for trials of COVID-19 vaccines, due to the worldwide importance of clarity in assessing benefits and harms.

EBM-based absolute measures create opportunities to clarify how the distribution of diseases differs and changes depending on variability in subpopulations’ contextual conditions and the emergence of new variants over time. Prior work generally has suggested that contextual variability reduces the ability to reach firm conclusions based on absolute measures. This study adds a perspective about differing effectiveness in subpopulations with varying baseline risks. High-prevalence subpopulations in specific geographical areas therefore become priorities for vaccine distribution.

Strengths and limitations of the study

The scoping review methodology determined that few publications about COVID-19 vaccines have analysed EBM indicators to assess benefits and to compare benefits and harms. A strength of the study emerges from calculations of these rarely discussed EBM indicators based on data from major RCTs. The study highlights the importance of presenting these metrics within vaccine trials, which has become an apparent gap in the literature. The methods of calculating these key EBM indicators facilitate a reproducible technique to compare benefits and harms for assessment of vaccines previously developed and those to be developed in the future.

We believe that this work can strengthen policy decisions focusing on prioritisation for vaccine distribution, to emphasise geographical areas with high baseline risks due to high prevalence of infection. Calculation of ARR and NNV, enhanced by sensitivity analyses in subpopulations with different baseline risks and with varying assumptions about vaccine effectiveness, could inform such policy decisions. This approach can supplement consideration of other contextual risk factors such as age, immunocompromised status, comorbidities and occupational exposures.

For instance, the analysis of hospitalisation clarifies that vaccine effectiveness varies widely in different geographical areas, based on differences in ARR. This perhaps obvious insight has not yet guided policy decisions so that vaccine delivery could emphasise the specific geographical areas with highest incidence and prevalence. At certain times during the pandemic, as shown in our findings, the NNV to prevent one hospitalisation in some parts of the world was less than 1000, whereas in other locations, it was more than 10 000. We believe that such findings focusing on baseline risks should be useful in most geographical areas of the world.

Regarding limitations, there are several. As noted earlier, concerns have arisen about the usefulness of NNV due to contextual variations in different geographical areas, changing contextual conditions over time, varying targets for NNV implemented by vaccination programmes, and lack of connection to effects of vaccination on transmission and herd immunity. But we also have noted several studies demonstrating that variations in NNV related to differing baseline risks of subpopulations can assist in policy decisions about distribution of vaccines targeting other viral diseases. These favourable experiences influenced our emphasis on absolute measures.

Other considerations about vaccine effectiveness may constrain conclusions drawn from this research. Unavailable contemporary data on subpopulations’ other contextual conditions impacting vaccine effectiveness—such as age, immunocompromised status and comorbidities—prevented multivariate modelling for the comparative analysis of ARRs. Also, the degree to which the effectiveness of the vaccines will change over time, especially as additional mutations arise and as public health practices such as social distancing and use of masks change, remains unclear.

We found that measures of harm and comparisons of harms versus benefits by calculation of ARI and NNH were challenging to achieve based on publications or accessible unpublished data. As a result, we were not yet able to resolve the question of benefits versus harms through these EBM indicators. Using EBM measures is not the only way to compare harms and benefits, and regulatory agencies that approved COVID-19 vaccines generally used comparisons that did not involve the direct quantification of NNV versus NNH.

We could not determine if there was a rationale for not reporting ARR and NNV in addition to RRR, or for not reporting quantitative EBM measures of harm such as NNH. Reasons for this decision by researchers, editors and governmental regulators to exclude previously expected EBM indicators warrant clarification. Partly because we could analyse only published data from the major publications but not the inaccessible raw data or other communications, understanding this apparent paradox awaits further investigation.

Despite these limitations, our analysis shows that reductions of absolute risk and measures of harms versus benefits deserve more attention from the standpoint of EBM than they have received so far.

Implications for research

We see several implications of this work for future research. First, in addition to RRR, research publications concerning the efficacy or effectiveness of vaccines should report other standard measures recommended in EBM, including ARR, NNV, ARI and NNH. Investigators should try to compare benefits versus harms of vaccines, through measures like NNV and NNH. Based on ARR and NNV, the research reports should show how vaccine effectiveness varies depending on baseline risks of disease in different population subgroups. For practitioners, patients and policymakers, the reports should provide practical advice that can guide informed consent procedures and decisions about vaccine distribution.

Population-based random sampling procedures for surveillance of incidence, prevalence and key contextual variables would improve work in this field. During the COVID-19 pandemic, incomplete case finding has persisted and has worsened over time. Surveillance through population-based sampling, including rapid serological testing, has occurred in several geographical locations with some success and manageable costs.40 41 62–70 This approach could permit ongoing assessment of absolute measures concerning risk reduction and harms, as well as multivariate modelling to include key contextual information for subpopulations. Surveillance research is receiving more serious consideration as the limitations of prevailing methods have become more apparent.

Implications for global health policy

Explanations to policymakers, patients and the public should present transparent information about ARR in addition to RRR. If a vaccine has effects on reducing transmission or enhancing herd immunity in addition to risk reduction at the individual level, these effects should be quantified and explained clearly. For instance, such an explanation could clarify that vaccines provided substantial protection against infection for all variants prior to Omicron, and that they had greater effectiveness in high-prevalence geographical areas.

Vaccine distribution should target subpopulations with higher baseline risks of disease, rather than focusing only on the goal of vaccinating entire populations. A strategy emphasising vaccines’ differential impacts on reducing absolute risk in targeted geographical areas could alleviate some economic and practical burdens of trying to provide vaccines for everyone, especially in poorer regions.71 If resources to support full EBM analyses are not available, using observed prevalence to guide geographical priorities in vaccine distribution could become a satisfactory alternative. Realistically, some decision-makers in power may not support such policies that favour disadvantaged groups. Nonetheless, this strategy is important as we face difficult barriers to distribution related to wealth, power, minority status, structural racism and other sources of inequality72—barriers sometimes depicted as ‘vaccine apartheid’.

How to explain these findings

Online supplemental appendix 2 presents brief explanations for possible use in informed consent procedures, educational efforts with the public, continuing medical and public health education, and advice to policymakers.

Conclusion

In summary, some key principles of EBM have not guided reports about COVID-19 vaccines. These gaps have arisen especially in the quantification of impacts on absolute risk in studies of efficacy and effectiveness, which have emphasised high RRR; calculations of ARR and NNV provide important additional perspectives. Systematic comparisons of vaccines’ benefits versus harms using EBM measures have not emerged clearly from published reports. Variations in vaccines’ impact on absolute risk depending on baseline risks of disease in different subpopulations should receive more attention in research and in global health policy recommendations about vaccine distribution. Such evidence-based principles gain even more importance in the context of barriers to vaccine access linked to profound socioeconomic inequality.

Data availability statement

Data are available upon reasonable request. All the data used in this study are publicly available online and are properly cited. More detailed information about obtaining access to the data will be provided on request.

Ethics statements

Patient consent for publication

Ethics approval

Ethical review and approval were waived for this study because all data were anonymous and aggregated without any personal information.

Acknowledgments

An early version of this work appeared as a preprint (medRxiv.org 2021, doi: https://doi.org/10.1101/2021.06.28.21259039). We appreciate the constructive suggestions of two peer reviewers for BMJ Open.

References

Supplementary materials

  • Supplementary Data

    This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

Footnotes

  • AL and HW are joint first authors.

  • Contributors AL conceptualised the study, with input from HW. AL coordinated the statistical analysis and literature reviews, with inputs from HW. AL wrote the first draft of the manuscript; HW did extensive revisions. EF edited the manuscript, contributed to revisions and assisted with reporting the results of the scoping review. KRN contributed to editing and data interpretation, especially concerning states and territories in India. All authors approved the final version of the manuscript. HW is the guarantor for this article.

  • Funding EF is an independent investigative journalist who received partial financial compensation from the Allende Program in Social Medicine (grant number not applicable) for work on this project. No other specific funding was received for this study. AL and HW are semiretired from clinical practice in internal medicine. HW receives access to library resources and personnel, internet service and uses of other facilities as a distinguished professor emeritus at the University of New Mexico. HW obtains no ongoing salary support but does receive retirement pensions as an emeritus professor from the University of New Mexico and the University of California. HW also receives payments as a primary care practitioner working to cover rural community health centres through the University of New Mexico’s Locum Tenens Program. AL receives a retirement pension from a private fund generated through his medical practice.

  • Competing interests From personal earnings based on past and present clinical work, AL and HW make financial contributions to the Allende Program in Social Medicine, a tax-exempt charitable foundation, which provides support for EF in this project and supports other projects unrelated to the one reported here. The Allende Program supports a variety of projects in social medicine but does not hold a specific value orientation that would shape expectations about the conclusions of a research study. The financial contributions of AL and HW have been general-purpose donations that provided support for several different projects. While a number of projects aim to improve access to health services (such as health and mental health services for active-duty military personnel as well as access to vaccines), there have been no explicit or implicit expectations that research projects should reach certain conclusions. The Allende Program receives a grant from the James R and Mary Jane Barrett Foundation, Baltimore, Maryland, USA, a bequest from a patient of HW, and donations from individual contributors for a clinical programme serving active-duty military personnel that HW directs and that is unrelated to the present report. HW, AL and KRN receive royalties and honoraria for books, other writings, and presentations for universities and non-profit organisations unrelated to this project.

  • Patient and public involvement Patients and/or the public were involved in the design, or conduct, or reporting, or dissemination plans of this research. Refer to the Methods section for further details.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.