Article Text

Original research
Use of community engagement interventions to improve child immunisation in low-income and middle-income countries: a systematic review and meta-analysis
  1. Monica Jain1,
  2. Shannon Shisler2,
  3. Charlotte Lane3,
  4. Avantika Bagai4,
  5. Elizabeth Brown5,
  6. Mark Engelbert3
  1. 1International Initiative for Impact Evaluation, New Delhi, India
  2. 2International Initiative for Impact Evaluation, London, UK
  3. 3International Initiative for Impact Evaluation, Washington, District of Columbia, USA
  4. 4Development Solutions, New Delhi, India
  5. 5Center for Effective Global Action, University of California, Berkeley, California, USA
  1. Correspondence to Dr Monica Jain; mjain{at}3ieimpact.org

Abstract

Objective To support evidence informed decision-making, we systematically examine the effectiveness and cost-effectiveness of community engagement interventions on routine childhood immunisation outcomes in low-income and middle-income countries (LMICs) and identify contextual, design and implementation features associated with effectiveness.

Design Mixed-methods systematic review and meta-analysis.

Data sources 21 databases of academic and grey literature and 12 additional websites were searched in May 2019 and May 2020.

Eligibility criteria for selecting studies We included experimental and quasi-experimental impact evaluations of community engagement interventions considering outcomes related to routine child immunisation in LMICs. No language, publication type, or date restrictions were imposed.

Data extraction and synthesis Two independent researchers extracted summary data from published reports and appraised quantitative risk of bias using adapted Cochrane tools. Random effects meta-analysis was used to examine effects on the primary outcome, full immunisation coverage.

Results Our search identified over 43 000 studies and 61 were eligible for analysis. The average pooled effect of community engagement interventions on full immunisation coverage was standardised mean difference 0.14 (95% CI 0.06 to 0.23, I2=94.46). The most common source of risk to the quality of evidence (risk of bias) was outcome reporting bias: most studies used caregiver-reported measures of vaccinations received by a child in the absence or incompleteness of immunisation cards. Reasons consistently cited for intervention success include appropriate intervention design, including building in community engagement features; addressing common contextual barriers of immunisation and leveraging facilitators; and accounting for existing implementation constraints. The median intervention cost per treated child per vaccine dose (excluding the cost of vaccines) to increase absolute immunisation coverage by one percent was US$3.68.

Conclusion Community engagement interventions are successful in improving outcomes related to routine child immunisation. The findings are robust to exclusion of studies assessed as high risk of bias.

  • Public health
  • Community child health
  • PUBLIC HEALTH

Data availability statement

Data are available on reasonable request.

https://creativecommons.org/licenses/by/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution 4.0 Unported (CC BY 4.0) license, which permits others to copy, redistribute, remix, transform and build upon this work for any purpose, provided the original work is properly cited, a link to the licence is given, and indication of whether changes were made. See: https://creativecommons.org/licenses/by/4.0/.

Statistics from Altmetric.com

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

STRENGTHS AND LIMITATIONS OF THIS STUDY

  • Thorough literature search of 21 major electronic databases and reporting as per Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines.

  • Presents a nuanced framework of community engagement with a typology that differentiates three types of interventions: interventions in which community engagement is embedded, and those interventions that engage community in their design or implementation.

  • The effects of community engagement interventions are robust to exclusion of studies assessed as high risk of bias for almost all the primary outcomes. The effects are also uniform across geographies and baseline immunisation rates.

  • For some immunisation outcomes the evidence base for drawing conclusions is adequate, for others it is limited.

  • Evidence base is skewed across the three engagement types with a relatively large evidence base for those interventions in which community engagement is embedded and limited for those interventions using engagement in implementation autonomy

Introduction

Immunisation is one of the most cost-effective ways to prevent and control life-threatening infectious diseases. From 2001 to 2020, projects that introduced or increased coverage of vaccines averted an estimated 14 million deaths, 350 million cases of illness, 8 million cases of long-term disability and 700 million disability-adjusted life-years.1 Nonetheless, rates of routine vaccination of children in low-income and middle-income countries (LMICs) are low or stagnant. In 2019, an estimated 19.7 million infants did not receive routine immunisations. Around 60% of these children live in ten LMICs, including Ethiopia, India, Nigeria and Pakistan as of 2019.2

Community engagement approaches feature prominently in global immunisation strategies.3 However, there is a dearth of rigorous and systematic evidence on effectiveness of community engagement approaches to improve routine childhood immunisations specifically in LMICs. In our search, we could find only two systematic reviews for LMICs, which analysed effectiveness of community monitoring interventions and preventive interventions delivered by community health workers, respectively.4 5 As such, previous systematic reviews do not provide adequate guidance to stakeholders interested in understanding whether and how alternative community engagement interventions work in LMICs to improve routine childhood immunisations and at what cost. There is, therefore, a need to make such evidence available to guide policymakers and public health practitioners in making informed decisions about these interventions. To address this knowledge gap, we conducted a systematic review examining the effects of community engagement interventions on outcomes related to childhood immunisation in LMICs, determining their cost-effectiveness and identifying contextual, design and implementation features that may be associated with intervention effectiveness.

Methods

Overview

The protocol of this systematic review with meta-analysis is registered with The Campbell Collaboration.6 We followed the Campbell and Cochrane Collaborations’ guidelines for systematic reviewing7–10 and drew on theory-based mixed-methods impact evaluation11 and systematic review12 13 concepts. We followed the PRISMA reporting guidelines. The amendments to the information provided in the protocol is reported in online supplemental appendix 1.

Conceptual framework

For our review, we defined ‘communities’ in reference to the lowest level of the health service delivery system (or whatever level provides routine immunisation services in the local context). A community is a group of people who serve or are served by a particular primary health facility. Thus, communities encompass a wide range of stakeholders, including caregivers, health service providers and influential community members such as religious or other traditional leaders.

WHO 2020 defines community engagement as ‘a process of developing relationships that enable stakeholders to work together to address health-related issues and promote well-being to achieve positive health impact and outcomes.’14 For this review, we developed a framework that classified community engagement interventions based on process of engagement as in the WHO definition. It also corresponds to the ‘utilitarian perspective’ of community engagement captured and articulated in Brunton et al15: ‘In utilitarian perspectives, health (and other) services reach out to engage particular communities that they have identified require assistance and the intervention is devised within existing policy, practice and resource frameworks.’ In addition, our framework goes beyond one-way communication to include some consultation or dialogue with the community or some decision-making by them. We considered three points within an intervention during which engagement could occur, as elaborated below and in online supplemental appendix 2.

Engagement in the design of interventions: Community input or feedback was sought before implementing an intervention (eg, pilot, needs assessment, formative evaluation and outreach).

Engagement in implementation autonomy of interventions: Community was used in intervention implementation as healthcare workers, facilitators or problem solvers and only if they had some opportunity to affect or influence its implementation.

Engagement as the intervention (engagement is embedded): A serious attempt was made to gain community buy-in for activities or new cadres of community-based structures were established (eg, village health committees or community health volunteers).

Research questions

The research questions for this review were:

  1. What evidence exists regarding the effectiveness of community engagement interventions in improving routine immunisation coverage of children in LMICs?

  2. Is there evidence for heterogeneous effects of community engagement strategies (ie, does effectiveness vary by geographical region, gender or programme implementation)?

  3. What factors relating to programme design, implementation and context are associated with better or worse outcomes along the causal chain? Do these vary by the kind of community engagement?

  4. What is the cost‐effectiveness of different community engagement interventions in improving children routine immunisation outcomes?

Search strategy

We implemented a systematic and comprehensive search strategy, in consultation with an information specialist. In May 2019 and May 2020, we searched 17 academic databases for experimental and quasi-experimental impact evaluations of community engagement interventions considering outcomes related to routine child immunisation in LMICs (using the World Bank country income classifications to determine LMIC status at the time the intervention began). We also searched 17 additional websites for grey literature. The list of sources searched and an example set of search strings are provided in online supplemental appendix 3. We complemented this with citation tracking and contacting experts. The grey literature search was conducted by AB with support from external consultant reviewers. Given the limitations of the search functions on websites we searched for grey literature, it was not possible to use the same complex search strings used in academic databases, and search strategies were developed on a site-by-site basis.

Inclusion/exclusion criteria (population, intervention, comparators, outcomes and study designs)

The population, intervention, comparators, outcomes and study designs eligible for inclusion in the study are provided in table 1. No language, publication type or date restrictions were imposed. Because our definition of community focused on the lowest levels of health facilities, we excluded interventions targeting higher levels of the health system (eg, state-level officials) (online supplemental appendix 4). The primary outcomes considered in this review were coverage rates for (A) full immunisation, which is typically defined as the percentage of 1 year old who have received one dose of Bacille Calmette-Guérin (In some countries, other vaccinations such as those for JE encephalitis and yellow fever are administered to children as a part of the routine immunisation schedule. In those contexts, we went by the definition of full immunisation mentioned in the impact evaluation study), (B) third dose of DPT or pentavalent, (C) first dose of measles or (D) the timeliness of any of these doses. Additional antigen-specific immunisation coverage outcomes and secondary outcomes reflecting upstream conditions (eg, attitudes about vaccination and access to immunisation services) and downstream effects (eg, morbidity and mortality) of the primary outcomes were also included. Official health records and parent recall were considered acceptable sources of measures of immunisation coverage. The former was used when both measures were reported separately.

Table 1

PICOS inclusion and exclusion criteria

This review includes experimental and quasi-experimental studies that estimate the causal impact of an intervention, as compared with usual practice, by establishing a counterfactual. Specifically, studies with the following evaluation designs are included: randomised controlled trials, regression discontinuity designs, instrumental variables’ estimation, statistical matching (eg, propensity score matching), difference-in-differences (or any mathematical equivalent), fixed effects estimation and interrupted time series. We excluded studies for which the reported quantitative data could not be meaningfully converted to an effect size. In cases of relevant missing or incomplete data, we contacted study authors to obtain the required information. If we were unable to obtain the necessary data, we reported the characteristics of the study but did not include these studies in the meta-analysis. We conducted additional searches for economic and qualitative evidence on the included impact evaluations (online supplemental appendix 5).

Screening

At both the title and abstract and full‐text screening stages, all papers were double screened by research consultants and supervised by MJ, ME and AB. Reconciliation meetings were held to resolve disagreements, and MJ and ME made final decisions on unresolved cases. The same reviewers manually searched for qualitative papers and project documents on Google Scholar and websites of implementing organisations and screened the papers for inclusion as they were identified.

Data analysis

Studies were coded for their engagement type by two reviewers (MJ and AB) who independently reviewed the intervention description and coded these against the definitions provided above. If studies allowed for engagement at several stages of the intervention, they could be coded as having more than one engagement type. We used Microsoft Excel to extract descriptive information and effect sizes from included studies using double coding. Coders reconciled their answers, and a study author made final decisions in case of disagreements. For qualitative analysis, all impact evaluations and additional documentation identified in the search were coded in NVivo. Cost data were single coded and checked by a study author (online supplemental appendix 6).

To avoid double-counting of evidence from different papers focusing on the same study, we linked these papers prior to analysis. We extracted data from the most recent publication. When data were reported over multiple time periods, we extracted data for each period. Where authors reported the same outcome using more than one analytical model, we extracted data from the authors’ preferred model specification. When the preference was not specified, we used the model with the most controls. Where studies reported outcomes related to multiple treatment arms and only one comparison group, we estimated an effect size for each of the treatment arms.

To assess quantitative risk of bias, we created an adapted version of the Cochrane guidelines for assessing randomised controlled trials and non-randomised studies.16 17 These assessments were conducted by two independent reviewers. Coders reconciled their answers, and a study author made final decisions in case of disagreements. For testing the sensitivity of the results to low-quality studies, we ran each analysis with and without studies scoring high risk of bias.

We critically appraised qualitative and mixed-methods studies using an adapted version of the nine-item framework developed by the Critical Appraisal Skills Programme.18 In addition, we carried out a sensitivity analysis in which we considered only the high-quality qualitative studies that had a risk of bias assessment score of 20 or higher, indicating low risk of bias. For cost evidence, we assessed risk of bias along six primary dimensions adapted from a combination of tools, including: Doocy and Tappis19; Campbell Collaboration Economic Methods Policy Brief20 and Methods for the Economic Evaluation of Health Care Programmes21 (online supplemental appendix 7).

We calculated the standardised mean difference, or Cohen’s d, its variance and SE for each effect, converting effects reported in other metrics as necessary, using formulae provided in Borenstein et al.22 In all cases we then adjusted Cohen’s d to Hedges’ g as defined in Ellis.23 For studies reporting regression results, we followed the approach of Keef and Roberts24 using the regression coefficient and the pooled SD of the outcome.

The amount of heterogeneity (ie, Embedded Image) was estimated using the DerSimonian-Laird estimator.25 The Q-test for heterogeneity26 and the Embedded Image statistic27 are reported. We complement this with an assessment of heterogeneity of effect sizes graphically using forest plots. We identified outliers using studentized residuals and identified overly influential studies using Cook’s distance. Where outliers were indicated, we report the resulting effect sizes when they are left out of the analysis. As an additional sensitivity test, we ran a full leave-one-out analysis for all models, and we report these results when and where they are useful. Whenever feasible, we conducted moderator analyses using meta-regression to investigate sources of heterogeneity. (All but two moderators were chosen a priori. Baseline coverage and vaccine hesitancy were added after feedback from an initial peer review from the Campbell Collaboration (copublisher of this work).) The analysis was carried out using R (V.4.0.4)28 and the metafor package (V.2.4.0).29 All analyses used a random effects model because we did not reasonably expect the included studies to be functionally identical and the goal was to generalise to the larger population.30

Qualitative analysis followed a mix of inductive and deductive coding approaches to identify themes related to barriers and facilitators, reasons for intervention success or failure, and uptake and fidelity challenges. An initial set of themes was developed based on familiarity with the literature. However, as new topics were identified, new themes were added. Themes were also disaggregated if it became clear they were too broad. Research consultants conducted coding with oversight from CL and AB.

Patient and public involvement in research

There was no patient or public involvement in this research.

Results

Our search identified over 43 000 records, which were reduced to 29 481 unique abstracts after deduplication (figure 1). After title and abstract screening, we considered 1285 studies for full-text screening and could not locate an additional 44, published mostly before 2000. We excluded articles at full-text for not satisfying the inclusion criteria by country (129), study type (evaluation study) (304), evaluation method (213), outcome (235) and community engagement type (172). We ultimately identified 61 impact evaluations (table 2)31–92 that assessed the effects of community engagement interventions on outcomes related to routine child immunisation in LMICs. We identified one publication in Spanish,45 with all others in English. Five studies did not include sufficient data to calculate an effect size, thus, 56 studies were included in the meta-analysis. Inter-rater reliabilities were calculated on a sample of studies, and ranged from 28% (mean effect of the intervention to 100% (eg, country, publication year and study design). All studies were reconciled prior to analyses.

Table 2

Describing characteristics of the included studies

Figure 1

PRISMA flow diagram. PRISMA, .Preferred Reporting Items for Systematic Reviews and Meta-Analyses.

Risk of bias

Of the 31 included studies with experimental designs, only two had a low risk of bias, six had some concerns and 23 had a high risk of bias. Of the 30 included quasi-experimental studies, only 2 were assessed as low risk of bias, 1 as some concerns and 27 as high risk of bias. Biases arising from outcome measurement and deviations from intended interventions were the most common across both the study designs.

Although only five qualitative studies scored strong on all key elements, most studies received strong scores on most key elements and had quality appraisal scores greater than 20, indicating low risk of bias. The most common elements found to be missing were sample characteristics and analytical methods. The quality of the cost evidence in the 22 studies that included such evidence was mixed (further information and visualisations for the risk of bias appraisals can be found in online supplemental appendix 8).

Community engagement interventions

Full immunisation

A total of Embedded Image studies examined the effect of community engagement interventions on full childhood immunisation and found Embedded Image (95% CI 0.06 to 0.23), z = 3.28, p = 0.01), indicating a small but significant benefit for the treated group of 0.14 SD units (figure 2). A 95% credibility/prediction interval for the true outcomes is given by –0.28 to 0.57. Hence, although the average outcome is estimated to be positive, in some studies the true outcome may in fact be negative.

Figure 2

Forest plot showing the observed outcomes and the estimate of the random effects model for the impact of community engagement interventions on full childhood immunisation. Note that # (number of) participants is specific to each effect and thus may not reflect the sample size for the full study.

The rank correlation test indicated funnel plot asymmetry (Embedded Image) but the regression test did not (Embedded Image; see online supplemental appendix 9 for additional information). The true effects appear to be heterogeneous (I2=94.5%, Embedded Image). Outlier analyses revealed that Banerjee may be a potential outlier, and sensitivity analyses removing Banerjee (2010) reduced the overall average effect (μ = 0.08 (95% CI 0.04 to 0.12)), but it was still positive and significant (z = 4.12, p < 0.001). (For all other outcomes, outlier analyses will be presented in online supplemental appendices.) Sensitivity analysis using the leave-one-out approach indicates there are no other studies whose removal results in substantial changes to the average effect or overall heterogeneity.

Sensitivity analyses were also conducted to examine the robustness of the results to the exclusion of low-quality studies. When studies assessed as high risk of bias were removed, the resulting effect was slightly larger and still statistically significant (Embedded Image (95% CI 0.08 to 0.27)), k = 4, z = 3.67, p < 0.001. We examined several potential sources of heterogeneity, including exposure to the intervention, evaluation period, study design, year, geographical region, data source, whether the intervention was implemented by a government agency (either alone or in combination with another agency), whether new cadres of health workers were established, presence of vaccine hesitancy and baseline vaccine coverage rates. There were no significant moderators in the context of this model (see online supplemental appendix 9).

DPT 3

A total of Embedded Image studies examined the effect on DPT3 vaccination coverage and found a small but significant benefit to the treated group compared with the untreated group (Embedded Image(95% CI 0.06 to Embedded Image, Embedded Image), Embedded Image; figure 3). A 95% credibility/prediction interval for the true outcomes is given by Embedded Image to 0.26. Hence, although the average outcome is estimated to be positive, in some studies the true outcome may in fact be negative.

Figure 3

Forest plot showing the observed outcomes and the estimate of the random effects model for the impact of community engagement interventions on DPT3 vaccination. Note that # participants is specific to each effect and thus may not reflect the sample size for the full study.

The true outcomes appear to be heterogeneous (I2=76.8%, Embedded Image). When low-quality studies were removed, the average effect increased slightly (Embedded Image (95%CI 0.05 to 0.17), k = 4), and was still statistically significant (z = 3.70, p < 0.001; see online supplemental appendix 9). Publication year was a significant source of heterogeneity; each additional year reduced the size of the effect by .014 SD units (see online supplemental appendix 9).

Measles

A total of Embedded Image studies examined the effect on measles vaccination coverage and found a very small but significant benefit for the treated group compared with the untreated group (Embedded Image (95% CI 0.03 to 0.11), z = 3.22, p < 0.01; see figure 4). A 95% credibility/prediction interval for the true outcomes is given by –0.08 to 0.22. Hence, although the average outcome is estimated to be positive, in some studies the true outcome may in fact be negative.

Figure 4

Forest plot showing the observed outcomes and the estimate of the random effects model for the impact of community engagement interventions on measles vaccination. Note that # participants is specific to each effect and thus may not reflect the sample size for the full study.

When low-quality studies were removed, the average effect increased (Embedded Image, k = 6, (95% CI 0.03 to 0.15 and was still statistically significant z=2.98, p=0.003). The true outcomes appear to be heterogeneous (I2=73.6%, Embedded Image). None of the moderators were significant sources of heterogeneity (see online supplemental appendix 9).

Vaccination timeliness

We found a small but significant effect on all three timeliness outcomes: full immunisation timeliness (Embedded Image (95% CI 0.07 to 0.24, z = 3.41), p<0.001, 95% prediction interval 0.04 to 0.27; DPT3 timeliness (Embedded Image (95% CI 0.03 to 0.14), z = 3.00, p<0.01, 95% prediction interval 0.03 to 0.14) and measles timeliness (Embedded Image (95% CI 0.14 to 0.32, z=5.06, p<0.001, 95% prediction interval 0.14 to 0.32. For all timeliness outcomes, tests of heterogeneity were not significant (p>0.05). For full immunisation and measles timeliness outcomes, the sensitivity analysis could not be conducted due to an inadequate number of studies. For DPT3, the average effect increased but became non-significant when low quality studies were removed (see online supplemental appendix 9).

Subgroups of community engagement interventions

Studies that used engagement as the intervention had a significant positive effect on full childhood immunisation, DPT3 vaccination and measles vaccination, but evidence was insufficient to synthesise measures of vaccination timeliness (table 3).

Table 3

Overview of the primary outcomes by engagement type

When studies used community engagement in the design, there was a significant positive effect on full childhood immunisation and measles vaccination but not on DPT3 vaccination. In addition, there was a positive significant effect on timeliness of full childhood immunisation and DPT3 vaccination. No studies using engagement in the design reported on timeliness of measles vaccinations.

For engagement in implementation autonomy, the analysis is based on a limited number of studies and we found no significant effect on either coverage or timeliness outcomes. There were no studies reporting on the timeliness of measles vaccination.

Finally, some studies combined multiple engagement types in their interventions. These interventions had a significant effect on DPT3 vaccination but not on measles vaccination or full childhood vaccination. Evidence was insufficient to synthesise measures of vaccination timeliness.

Secondary outcomes analyses can be found in online supplemental appendices 10–13.

Qualitative findings

Programme design characteristics were associated with intervention success or failure across all engagement types. Certain aspects of community engagement itself, such as conducting stakeholder consultations, holding community dialogues or involving community leaders were associated with better immunisation outcomes. Non-engagement intervention design features also affected intervention success. These design features include incentives given to caregivers and leadership and supportive supervision, which improved overall health service delivery and health worker performance. Among the studies that attributed intervention failure to programme design, inadequate duration, frequency or exposure to the intervention were the most notable themes.

The importance of accounting for contextual barriers to or facilitators of immunisation emerged consistently. Limited availability of services, especially insufficient staff and vaccine supply, were dominant barriers to immunisation, affecting outcomes in the early portion of the causal chain. Other common barriers to immunisation included practical barriers faced by caregivers such as costs, largely indirect and logistics (wait time and language barriers) or distance. There was more variation in barriers related to social norms, fear and an understanding of the importance of immunisation by type of engagement. Poor quality of services, including uninviting attitudes of health workers, posed a barrier to immunisation in communities that received engagement as the intervention or were engaged in the design of the intervention.

However, we also found that certain contextual factors could become facilitators of immunisation outcomes, provided an intervention has adequately situated itself to leverage them. Across all engagement types, studies associated caregivers’ awareness and perception of the benefits of vaccination with improved immunisation outcomes. Similarly, availability of health infrastructure and good quality of services were also associated with improved intervention uptake and its impact.

Implementation failures, such as low fidelity, were a common reason for intervention failure. Across all engagement types, interventions did not properly account for existing implementation constraints and practicalities on the ground and were forced to change their implementation plans. Many of these issues were encountered due to uncontrollable factors or invalid theory of change assumptions. For instance, programme design may not have accounted for the unavailability of intended participants due to competing priorities, thereby potentially invalidating the assumption of beneficiary exposure in the casual pathway. Administrative challenges were cited consistently, though their nature varied across engagement types, ranging from technical limitations (such as limited cellphone service) to political constraints to insufficient staffing levels.

Results were broadly consistent when only qualitative studies with a quality appraisal score greater than 20 were considered. The full qualitative synthesis and sensitivity analysis is available in online supplemental appendix 14.

Cost-effectiveness findings

Among the 14 studies for which we could calculate cost-effectiveness, we found that the median intervention cost per treated child per vaccine dose  (excluding the cost of vaccines) to increase absolute immunisation coverage by one percent was US$3.68 (all costs are reported in 2019 US dollars) and the average cost was US$44.10. There were three outlier observations that drove up this average cost and without them cost per vaccine dose to increase absolute immunisation coverage by 1% averaged US$3.97 (online supplemental appendix 15).

Discussion

Principal findings

We found that community engagement interventions had a small, but significant, positive effect on all primary immunisation outcomes related to coverage and their timeliness. We also found that certain features of interventions may contribute to their success. These include (A) appropriate intervention design, including building in community engagement features; (B) addressing common contextual barriers of immunisation and leveraging facilitators and (C) accounting for existing implementation constraints and practicalities on the ground. The median intervention cost per treated child per vaccine dose (excluding the cost of vaccines) to increase absolute immunisation coverage by 1% was US$3.68.

Among the four types of community engagement interventions, we found that engagement as the intervention (embedded community engagement), which involves creation of community buy-in or development of new community-based structures or cadres, had consistent positive effects on more primary vaccination coverage outcomes than the others. We also found engaging the community in the design of the intervention had a positive effect on most primary outcomes related to coverage. We found no ubiquitous patterns of heterogeneity among the primary outcomes.

While zero dose children were not the specific focus of this review, we can offer some insights based on our analyses of both DPT1 and BCG outcomes, which reflect access to initial dosing. Community engagement interventions did not show an effect on DPT1, but the evidence base was of low quality, with six of eight studies assessed as having a high risk of bias. There was a small but significant effect of community engagement interventions on BCG, but here again, the evidence base was of lower quality, with 9 of 12 studies assessed as having a high risk of bias. In both cases, the evidence base was smaller in size than for the primary outcomes. As we find positive effects of community engagement on children returning for DPT and measles doses, it may be that barriers to vaccination, like availability of health services, for zero dose children are different and unless those are addressed, community engagement itself may not be enough.

Strengths and weakness

Our systematic review uses a detailed framework of community engagement interventions to assess their effectiveness for improving outcomes related to routine child immunisation in LMICs. As far as we are aware, ours is the first systematic review to do this. Sensitivity analyses excluding high risk of bias studies showed that the effect was slightly larger and still statistically significant for almost all the primary outcomes for which we had sufficient data. The effects were also uniform across geographies and baseline immunisation rates.

We drew on 61 studies for meta-analysis, comprising 31 RCTs and 30 quasi experimental studies. However, only 56 studies provided sufficient information for calculating effect sizes and thus were included in meta-analytical models. For full immunisation, DPT3 and measles coverage, we could draw on 28, 22 and 20 studies, respectively, for pooled effects. However, for the timeliness of these coverage outcomes we had only 0–7 studies to assess the pooled effects. Thus, while for some outcomes the evidence base for drawing conclusions is adequate, for others it is limited. Among the four kinds of community engagement interventions, there was a relatively large evidence base for those with engagement as the intervention and those with multiple engagement types, while for interventions using engagement in implementation autonomy, the evidence was quite limited.

We identified additional documentation comprising qualitative studies, project reports, formative/process evaluations and observation studies for 39 of the 61 included impact evaluations. However, the crucial qualitative papers which help us gain a deeper understanding of overall intervention mechanisms of change were found for only 17 of the 61 IEs. Likewise, only 14 of the primary studies included in this review both estimated the intervention cost and reported it with sufficient detail for the review team to calculate the cost-effectiveness of the treatment. Low-quality cost data and the unavailability of underlying cost data contributed to the small number of primary studies included in the cost-effectiveness analysis.

The quantitative evidence was mostly low quality, though the randomised studies were generally of higher quality and less likely to have confounding bias than the quasi-experimental studies. The quality of qualitative studies was generally high. The quality of the cost evidence was mixed. Despite the quality concerns about quantitative evidence, the sensitivity analysis conducted by excluding low-quality studies corroborated the overall findings. Despite a comprehensive search strategy and the inclusion of grey literature, publication bias was detected for the three primary coverage outcomes (full immunisation, DPT3 and measles). While bias correction analyses indicated an identical effect size for full immunisation and DPT3, the effect for measles was reduced when publication bias was corrected for. Timeliness outcomes had an insufficient number of studies to test for publication bias, which limits our ability to interpret heterogeneity.

Agreement and disagreement with other reviews

The findings from this review are broadly consistent with Molina et al,5 which found positive effects of community monitoring interventions on immunisation coverage. Another review by Gilmore and McAuliffe4 examined the effectiveness of preventive interventions delivered by community health workers for maternal and child health in LMICs on essential newborn care and found some evidence in its support through narrative synthesis, but found the evidence base to be insufficient to draw firm conclusions.

Limitations

There are several potential limitations to the current review: (A) there were few analyses that were sufficiently powered to test for publication bias, thus,we cannot rule this out in many cases; (B) many of the moderator analyses were underpowered, meaning that in many cases we were unable to explore heterogeneity. This was particularly true in the context of the subgroup analyses of the four intervention types. In addition, it is likely that there is interdependency among moderator variables, but the current study did not allow for us to disentangle these confounds. Future studies may aim to better assess how moderators may work in tandem to affect the magnitude of change; (C) even in cases where the average effect was significant, forest plots demonstrate that some of the included studies reported a small negative affect, and prediction intervals often included both positive and negative values, which may have important implications when making decisions related to programme design and implementation; (D) we also observed very few studies which focused on subpopulation groups. This is particularly problematic given the focus on LMICs, where equity is important to consider when trying to increase coverage; (E) most of the community engagement interventions were in combination with other intervention components, thus we were not able to establish their unique contribution to changes in outcomes and (F) inclusion of primary studies into this review was based on the description of the community engagement aspects of the intervention. We may have excluded studies that should ideally have been included because of inadequate reporting of intervention components. Finally, funding for this project has concluded, thus, we do not have the resources to update our literature search last conducted in May of 2020.

Implications for policy and practice

COVID-19 has impacted routine child immunisation negatively in some countries, and community engagement interventions could be an effective way to counteract this decline. The positive effects of community engagement interventions can be expected across a variety of settings, although some engagement approaches appear to be more effective than others. Positive design features should be integrated into these interventions, including features such as holding community dialogues or involving community leaders, and non-community engagement features such as local supportive supervision and incentives to healthcare workers or caregivers. Wherever possible, binding contextual barriers to immunisation, such as weak health systems and social norms, should be accounted for in the design of interventions. Existing contextual facilitators for immunisation, such as good existing health systems or high maternal education, could be leveraged for increasing intervention impacts. Important implementation preconditions, such as regular internet service or sufficient staffing, should be assessed and established before the implementation or addressed through the design itself. Close monitoring of intervention implementation along with good understanding of context is important to help make necessary modifications in case of unexpected challenges, such as political instability.

Further research

For better-quality evidence and deeper mechanistic understanding, policy makers and practitioners should consider prioritising funding or commissioning research in the following areas: (A) ways of ameliorating outcome measurement bias due to self-reported immunisation coverage outcomes, as this was a principal source of bias; (B) better reporting of interventions, more rounded analysis of why the interventions worked through mixed-methods evaluations and greater focus on intermediate outcomes for improved understanding of causal mechanisms; (C) collection and reporting of high-quality cost data to enable cost-effectiveness analysis, which is important for decision-making within budget constraints and (D) focus on subgroup analysis, including for zero dose children, for ensuring immunisation services for the most marginalised children. It would also be useful to conduct an update of this review to include evidence produced since our final literature search in May of 2020.

Data availability statement

Data are available on reasonable request.

Ethics statements

Patient consent for publication

Ethics approval

Ethical consent from an ethics committee or institutional board was not required as this study does not involve human subjects.

Acknowledgments

This research has been undertaken as a part of 3ie’s immunisation evidence programme, supported by the Bill and Melinda Gates Foundation, Seattle, USA. We would like to thank Sohail Agha, Senior Program Officer, Gates Foundation, Seattle, for his continued support and engagement on the evidence programme and the review. Special thanks to Molly Abbruzzese whose guidance helped shape the scope of our evidence programme and aided the conceptualisation of this review. We thank Lisa Menning, Danielle Pedi, Jennifer (Fluder) Siler and Pedja Stocjicic for their valuable feedback on the review. We wish to thank John Eyers, Yoav Vardy, Daniela Anda, Shradha Parsekar, Sejal Luthra, Yue Zhan, Aditi Hombali, Lalitha Vadrevu, Reva Datar, Pankhuri Jha, Beáta Berkovics, Ashton Baafi, Meital Kupfer David Atika, Himani Aggarwal, Agrima Sahore, Mansi Wadhwa, and Harini Narayanan for excellent support and research assistance. We would further like to thank Marie Gaarder, Executive Director, Birte Snilstevit, Director of Synthesis and Reviews Office and Sebastian Martinez, Director of Evaluation Office, 3ie, for their guidance on the review process.

References

Supplementary materials

  • Supplementary Data

    This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

Footnotes

  • Contributors MJ, ME and AB conceived the review and wrote the initial protocol. ME and AB did the systematic search. ME, MJ and AB screened and identified studies and MJ and AB made final decisions regarding study inclusion. SS did the statistical analysis. CL and AB did the qualitative analysis. EB synthesised the cost evidence. MJ provided critical inputs on the whole analysis, checked data, coordinated the review and had full access to all materials and results. External consultants supported the authors in search, screening, data extraction and critical appraisal of quantitative and qualitative evidence base. All authors critically reviewed and revised the manuscript and approved the final document for submission. MJ is responsible for the overall content and is the guarantor.

  • Funding This research was funded through a grant from the Bill & Melinda Gates Foundation (OPP1115129).

  • Disclaimer The funder of the study had no role in study design, data collection, data analysis, data interpretation, or writing of the report.

  • Competing interests The International Initiative for Impact Evaluation (3ie). Through this grant, 3ie provided funding and technical assistance for seven impact evaluations of community engagement interventions for immunisation as a part of its immunisation evidence programme. This technical assistance included, but was not limited to: reviewing study designs, analysis plans and data collection instruments; advising research teams on how to improve study components and address challenges that arise during the course of the evaluation; and supporting grantees in engaging with stakeholders to promote uptake and use of evidence generated by the evaluations. As members of 3ie staff, authors MJ, AB and ME have all had varying levels of involvement in reviewing deliverables for these evaluations and providing research teams with technical assistance. Several procedural safeguards and transparency measures were put in place to mitigate the risk this conflict of interest imposed. First, all candidate studies, including those funded by 3ie, underwent a rigorous multi‐step screening process, including review at the title, abstract, and full‐text levels. To qualify for inclusion in the SR, a study was judged to meet the inclusion criteria related to study design, outcomes and population by two independent screeners who have reviewed the full text of the study. The 3ie study authors were responsible for assessing whether the studies met the inclusion criteria for community engagement because of the complexity of the framework. However, these authors have no financial interest in this area and have not published any prior reviews on the topic. The remaining study authors have no conflicts of interest to declare.

  • Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.