Objective To evaluate the impact of the community-based newborn care package (CBNCP) on six essential practices to improve neonatal health.
Methods CBNCP pilot districts were matched to comparison districts using propensity scores. Impact on birth preparedness, antenatal care seeking, antenatal care quality, delivery by skilled birth attendant, immediate newborn care and postnatal care within 48 hours were assessed using Demographic and Health Survey (DHS) and Health Management Information System (HMIS) data through difference-in-differences and multivariate logistic regression analyses.
Findings Changes over time in intervention and comparison areas were similar in difference-in-differences analysis of DHS and HMIS data. Logistic regression of DHS data also did not reveal any significant improvement in combined outcomes: birth preparedness, adjusted OR (aOR)=0.8 (95% CI 0.4 to 1.7); antenatal care seeking, aOR=1.0 (0.6 to 1.5); antenatal care quality, aOR=1.4 (0.9 to 2.1); delivery by skilled birth attendant, aOR=1.5 (1.0 to 2.3); immediate newborn care, aOR=1.1 (0.7 to 1.9); postnatal care, aOR=1.3 (0.9 to 1.9). Health providers’ knowledge and skills in intervention districts were fair but showed much variation between different providers and districts.
Conclusions This study, while representing an early assessment of impact, did not identify significant improvements in newborn care practices and raises concerns regarding CBNCP implementation. It has contributed to revisions of the package and it being merged with the Integrated Management of Neonatal and Childhood Illness programme. This is now being implemented in 35 districts and carefully monitored for quality and impact. The study also highlights general challenges in evaluating the impacts of a complex health intervention under ‘real life’ conditions.
- neonatal health
- community health worker
- complex health intervention
- propensity score
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
- neonatal health
- community health worker
- complex health intervention
- propensity score
Strengths and limitations of this study
Adopting a ‘natural experiment’ approach, we used multiple data sources and multiple statistical methods as an important strategy to validate findings.
The two datasets employed, the nationally representative cross-sectional Demographic and Health Survey and the public sector healthcare reporting system Health Management Information System, each has its own strengths and limitations but does not provide representative measures of coverage at population level.
An a priori conceptual framework defined the outcomes of the intervention and guided the analysis; along with other careful measures, such as excluding births taking place during training, this was intended to minimise bias.
Neonatal mortality as the ultimate outcome of interest could not be examined, as the datasets employed were insufficient for examining rare events at district level.
While infant and child mortality in developing countries have declined rapidly in the past decades, newborn mortality has decreased much more slowly.1 Nepal has demonstrated impressive reductions in child mortality of 76% since 1990, but over the same time period, neonatal mortality has decreased by only 50%.2 3 With 21 deaths per 1000 live births in year 2016, neonatal mortality now constitutes 54% of under-five deaths.4
Over two-thirds of newborn deaths could be prevented with relatively low-cost, low-tech interventions.5 6 A systematic review based on five randomised controlled trials (RCTs) from South Asia concluded that visits during the antenatal and neonatal periods and home-based treatment for illness reduce the risk of neonatal deaths and improve neonatal care practices, with greater survival benefit when home visits are integrated with preventive and curative interventions.7 Similarly, other South Asian studies employing different programme components and delivery approaches demonstrate improvements in uptake of antenatal care, institutional delivery and newborn care.8–10 Consequently, WHO and the Unicef recommend home visits during the first week of life by appropriately trained and supervised community health workers to promote healthy behaviours and timely recognition of newborn illness, and to provide home treatment for infections and feeding problems.11
Based on global, regional and national evidence, the Ministry of Health (MOH) combined seven community-based and home-based interventions in the community-based newborn care package (CBNCP) to tackle major causes of neonatal mortality.12 This programme comprises (1) behaviour change communication for birth preparedness and newborn care, (2) institutional delivery or clean home delivery through skilled birth attendants, (3) postnatal care, (4) care for low birthweight newborns, (5) management of newborn infections, (6) prevention of hypothermia and (7) recognition of asphyxia, initial stimulation and resuscitation. The programme is delivered through facility-based and community-based health workers as well as the Nepal-specific cadre of female community health volunteers (FCHVs), and comprises training and supervision of the health workforce and provision of essential commodities. The package included 7 days training for facility-based health workers, 5 days training for community-based health workers and 7 days training for FCHVs. Supervision and monitoring mostly uses existing approaches, supplemented with pilot-phase intensive supervision including, for example, monthly review meetings with FCHVs at the health facility level (see online supplementary file, box 1 CBNCP programme components).12 The CBNCP was piloted in 10 out of 75 districts of Nepal in 2009 and 2010 with funding from MOH, the United States Agency for International Development (USAID), Unicef and Saving Newborn Lives (SNL).
The objective of this study was to evaluate the impact of CBNCP on six essential practices to improve neonatal health in pilot districts compared with propensity score-matched comparison districts.
Study setting and population
Nepal is characterised by three distinct geographies, that is, terai or flatland, hill and mountain areas. The CBNCP was piloted in four hill and six terai districts, constituting the ‘intervention area’, to which we assigned a ‘comparison area’ (figure 1). In both areas, one site was purposively selected for an additional qualitative component of the study; methods and findings of the latter are reported elsewhere.13
The CBNCP targets all women of reproductive age, aiming to increase their interaction with the health system during pregnancy, delivery and the postnatal period. Our study was undertaken among women aged 15–49 years who had a live birth during 30 months preintervention compared with those with a live birth taking place during 7–14 months postintervention in view of Demographic and Health Survey (DHS) data being available for this period.
This quasi-experimental study uses propensity score matching and multiple data sources to assess the impact of the CBNCP (figure 2). It includes (1) before–after analysis of essential practices in the intervention versus comparison area based on DHS data, (2) before–after analysis of those same practices in the intervention versus comparison area based on Health Management Information System (HMIS) data and (3) analysis of training coverage and knowledge and skills of healthcare providers based on Newborn Health Information System (NHIS) data, which was an integral part of the CBNCP pilot and available in the intervention area only.12 14
Drawing on the comprehensive evaluation framework for evaluating the scale-up for maternal and child survival by Bryce and colleagues,15 we developed a conceptual framework, which regards the CBNCP as a complex multicomponent intervention16 17 and graphically presents the presumed causal pathway from CBNCP implementation within the health system (process and outputs) through changed practices of pregnant or recently delivered women (outcomes) to impacts on neonatal health (figure 3). Importantly, while the CBNCP’s main impetus is on training of health workers, supplies of equipment and medicines as well as supervision and follow-up, several of the outputs (eg, taking a urine sample for proteinuria test) and outcomes (eg, postnatal visits) could also be considered as components of the intervention. This conceptual framework was critical in our identification of relevant outcome variables.
Implementation of the CBNCP pilot through training of facility-based and community-based health workers and FCHVs started in May 2009 and was completed in July 2010 in pilot districts (see online supplementary table S1). Training dates were obtained from the MOH to define district-specific preintervention and postintervention periods used in the analysis of DHS and HMIS data; any births taking place during training were excluded from the analysis.
Propensity score matching
Propensity score matching is widely used to estimate the effects of health and other policy interventions, where RCTs are not feasible.18 19 It uses statistical techniques to construct a comparison group that is as similar as possible to the intervention group in an effort to reduce selection bias.20 21
Ten intervention districts were selected by the MOH in consultation with donors, considering development need, donor presence, district interest and ability to implement and monitor the programme (Parashuram Shrestha, personal communication, Nepal Ministry of Health, 2013). To reflect the propensity of a district to be selected for CBNCP implementation, we constructed a propensity score based on (1) the four components of the district Human Development Index value, (2) presence of donors involved in the CBNCP (ie, USAID, Unicef, SNL), (3) percentage rural population, (4) the MOH district performance rank and (5) road density (see table 1 for details).
As CBNCP implementation was limited to hill and terai districts, mountain districts were excluded. We used the psmatch2 command in Stata Special Edition 1222 to identify suitable comparison districts based on the nearest-neighbour method without replacement. We checked for balance in the distribution of propensity score components (using t-tests) and population and health infrastructure characteristics (using χ2 tests) between intervention (10 districts pooled) and comparison areas (10 districts pooled).
Data sources and variables
Multiple data sources were used to enable as complete an analysis of impact as possible and to triangulate information between sources with different strengths and weaknesses. The DHS provides nationally representative data on fertility, health-relevant behaviours and childhood mortality based on a multistage cluster random sampling strategy.23 The data for the Nepal DHS for 2011 are in the public domain (www.dhsprogram.com). The HMIS, owned by the MOH and primarily based on health facility records, provides information about health service use, morbidity and mortality, treatment outcomes and the availability of commodities. We used data on regular service delivery for 2009–2011, publicly available online (http://www.dohs.gov.np). We also obtained CBNCP-specific NHIS data from the CBNCP secretariat based at the Child Health Division at the MOH.24 These NHIS data were collected by the programme team as part of CBNCP delivery and monitoring, and provided insights about the knowledge and skills of programme-trained health workers and FCHVs.
Neonatal mortality as the ultimate outcome of interest was not feasible to assess given available data sources and sample sizes. Instead, with reference to our conceptual framework (figure 3), we examined changes in six essential practices to improve neonatal health by incorporating relevant contributing practices in combined binary outcomes (coded as ‘better practices’ or ‘poorer practices’). Relevant covariates were identified a priori as family characteristics (ie, wealth quintile, rural vs urban location, caste/ethnicity), maternal characteristics (ie, age at delivery, education and access to media) and child characteristics (ie, sex, parity) (see table 2 for details).
Difference-in-differences analysis estimates the change in outcome for the intervention area over a given time period by subtracting any change in outcome for the comparison area over the same time period. All outcomes were assessed as combined outcomes, that is, as the percentage of pregnant or recently delivered women adhering to ‘better practices’.25 Analyses for individual outcomes are provided as background information in online supplementary table S2.
For DHS data, difference-in-differences analysis using ordinary least squares regression was conducted for births occurring preintervention and postintervention. Where a woman had given birth more than once during the preintervention or postintervention period, only the most recent birth was included in the analysis to avoid non-independence of observations and to minimise recall bias. For HMIS data, a similar approach was adopted; however, tests of significance were not possible as the data were available only in aggregate at the district level. We also conducted logistic regression analysis of DHS data to examine if any differences between intervention and comparison areas persist after adjustment for all a priori identified covariates; here, the outcome was assessed at the individual level as either adhering or not adhering to ‘better practices’. All analyses were undertaken in Stata Special Edition 12.22
Table 1 shows that intervention and comparison areas are balanced for propensity score components as well as relevant population and health infrastructure characteristics.
Using preintervention DHS data, 533 and 347 births took place in the intervention and comparison area, respectively. Table 2 compares outcome variables and covariates for the most recent births in the 5 years preceding the DHS survey. In both areas, a majority of children are from rural locations, disadvantaged families and born to a mother with at least primary education. While respondents from intervention and comparison areas are largely comparable, there are statistically significant baseline differences in relation to family wealth status, maternal age at delivery and delivery by a skilled birth attendant even after matching.
In the 10 pilot districts, a majority of health providers were trained, that is, 1615 facility-based health workers, 902 community-based health workers and 7072 FCHVs. Overall, knowledge and skills as reported or demonstrated were fair with some variation by type of provider; availability of drugs and commodities was also good (table 3). All of these, however, showed much variation between districts, pointing to concerns with respect to quality of training, supervision and logistics (see online supplementary table S3).13
Table 4 presents findings from the difference-in-differences analysis of DHS data. With the exception of birth preparedness (no change) and postnatal care within 48 hours (increase in intervention area, decrease in comparison area), improvements were observed but to a similar extent in both areas with no statistically significant differences. For all six essential practices, the percentage of pregnant or recently delivered women adhering to better practices was lower in the comparison area at both points in time.
Similarly, difference-in-differences analysis of HMIS data showed improvements in both intervention and comparison areas for most of the practices assessed13; HMIS does not provide information on birth preparedness or immediate newborn care practices. Table 5 compares findings based on DHS and HMIS data, showing congruent trends for all essential practices despite differences in the specification of some indicators. The contradictory finding that iron supplementation decreased postintervention in the HMIS (which collects data from public service providers) but not in the DHS analysis (which reflects households seeking care from both public and private providers) is explained by government health facilities having run out-of-stock in October and November 2011.
Logistic regression analysis
The unadjusted ORs suggest statistically significant improvements in antenatal care quality (OR 1.8, 95% CI 1.1 to 2.9), delivery by a skilled birth attendant (OR 2.0, 95% CI 1.2 to 3.3) and postnatal care within 48 hours (OR 2.7, 95% CI 1.1 to 2.6) but not in the other three essential practices (figure 4). However, when adjusted for a priori identified covariates none of the changes in essential practices remained statistically significant.
Key findings and their explanation
Nepal’s CBNCP was developed based on existing studies, mostly from Nepal and South Asia, to ensure relevance to the country-specific or region-specific epidemiology, demonstrating effectiveness for a majority of the intervention components.14 The choice of interventions for integration within the package was driven by both effectiveness and feasibility considerations. However, there was no evidence for the effectiveness of the package as a whole,12 and the additional feasibility challenges of implementation at scale were probably not given sufficient attention.
The analysis of DHS and HMIS data suggests that the CBNCP did not have a significant impact on essential practices to improve neonatal health above a generally increasing trend in these practices. These findings must be interpreted with caution, given the relatively short time period between training health workers and FCHVs, which ranged from 7 to 14 months depending on the district, and assessment of relevant outcomes among programme beneficiaries. In light of the complex nature of the programme, where multiple components are intended to improve a whole range of health provider and population behaviours throughout pregnancy, delivery and the postpartum period, the present evaluation represents a very early assessment of potential impact.
Several factors are likely to interplay in explaining this current lack of impact.
Packaging of multiple interventions
The CBNCP bundled a range of specific measures in a complex package and implemented this across a large geographical area with an implementation modality largely dependent on the existing health system. In Nepal, the health system suffers from a number of problems and there is strong reliance on FCHVs. In contrast, prior studies, concerned with efficacy or effectiveness under real-world conditions, usually examined a single and relatively simple component (eg, chlorhexidine for cord care26) in a limited geographic area (eg, Mother and Infant Research Activity (MIRA)),27 implemented through a dedicated cadre of higher-level service providers (eg, Society For Education, Action and Research in Community Health (SEARCH28)) or undertaken as a distinct research project (eg, resuscitation29). It is therefore not surprising that the effectiveness of these interventions is diluted when merged in a package that is delivered by a lower-level service provider under ‘real life’ conditions. Indeed, a similar reduction of effectiveness when moving from research studies to large-scale implementation has been observed elsewhere.16 30 31 When going to scale, programme management, effective high coverage and a good match between community-based and facility-based service improvements are seen as critical.32–34
Healthcare providers and their training
The CBNCP was implemented through training of the existing cadre of facility-based (7 days) and community-based (5 days) health workers in the government system as well as FCHVs (7 days) with limited subsequent supervision and follow-up. Supervision is one of the most important elements of successful programmes, but also one of the most challenging programme elements to implement and assess. As a general indication, the Nepal Health Facility Survey35 reported that nearly 7 in 10 health facility-based workers received any kind of supervision visits during the previous 6 months. Comprehensive information on the extent and content of supervision in the context of the CBNCP is lacking, but anecdotal reports indicate concerns with respect to the frequency and effectiveness of supervision visits. While evidence from Nepal suggests that community health workers and FCHVs can identify and manage maternal and newborn health problems, this requires frequent training and mentoring.36 This study suggests much variation in programme performance across districts (see online supplementary table S3), generally indicating better results in areas where the CBNCP is implemented with more intensity. In addition, the qualitative component showed that service providers perceived the training as insufficient for them to be able to apply their skills confidently and to retain them over prolonged periods of time.13 Therefore, following the argument made by Kumar et al 37 that the effectiveness of an intervention is constrained by the weakest link in the causal-intervention pathway, the amount of training and subsequent supervision for this complex intervention package are likely to have been insufficient to promote meaningful behaviour change. Moreover, in a setting where medical shops are perceived to be more convenient than government health facilities,35 38 a programme that does not involve private providers is likely to show limited impact. In relation to antenatal services, private providers often provide specific components of those services (eg, iron folic acid supplement) and on-call services.
Other relevant health initiatives
In the last decade, Nepal has witnessed a host of programmes to improve maternal and child health, with many of these directly or indirectly impacting neonatal health.2 As adjustment for other relevant ongoing initiatives was not feasible in design or analysis of this impact study, the observed trends in essential practices to improve neonatal health and the lack of CBNCP impact in intervention relative to comparison areas are in part likely to be due to the high level of background activity.
Strengths and limitations
The CBNCP is a complex intervention, where multiple components are intended to improve a whole range of health provider and population behaviours throughout pregnancy, delivery and the postpartum period. As its implementation was outside of the control of the researchers, randomisation was not feasible and we had to adopt a ‘natural experiment’ approach. While matching largely achieved balance between intervention and comparison areas, some baseline differences persisted. Moreover, we did not match individual intervention and comparison districts but intervention and comparison areas. A major strength in addition to propensity score matching is this study’s use of multiple data sources to assess impact.
The DHS is a cross-sectional survey with retrospective recording of all pregnancies and births as well as relevant behaviours; it is thus subject to recall bias. DHS data are designed to be representative at the national level—for rare events, they are not necessarily representative at the district level and, consequently, assessment of impact on neonatal mortality was not feasible. The number of births covered is also limited, especially postintervention, as exposure time to the intervention was short (ranging from 7 to 14 months) to reflect true changes between areas. It is possible that changes in the behaviour of pregnant and recently delivered women will only become manifest after longer periods of time once health providers have internalised recommendations and implement them on a regular basis. The HMIS provides valuable information about healthcare use, knowledge and skills of service providers, and availability of key commodities and supplies in the health system. However, HMIS data are only available for the public sector and thus do not provide representative measures of coverage at population level, as many people rely on healthcare from informal and private providers.
Use of multiple data sources and multiple statistical methods was an important strategy to validate findings or lack thereof. Difference-in-differences calculations are subject to limitations, as adjustment for confounders was not possible with the information available at district level. Filtering of births for analysis (ie, before, during and after implementation) was customised by district, and the analysis excluded births taking place during training as a conservative strategy. We used an a priori conceptual framework to define the outcomes of the intervention and to guide the analysis.
Implications for research and practice
Overall, this study highlights that the design, piloting and implementation of a complex intervention such as the CBNCP must be carefully planned and evaluated. In fact, the assumption that combining a large number of intervention components, even where their individual effectiveness has been proven, will yield an effective intervention package that can be successfully implemented at scale does not hold. Importantly, evaluating under ‘real life’ conditions is not necessarily straightforward and may require the use of limited-quality routine data in combination with innovative study designs. Even though the CBNCP, as assessed through our study, was conceived as a pilot, rigorous assessment through the MOH and donors was lacking; despite increasing concerns about the quality of CBNCP implementation and a potential lack of impact, implementation continued and was rapidly extended beyond pilot districts.
The findings presented here, supported by those of the qualitative component of the study,13 suggest that the programme may need a repackaging and tightening of content as well as a revision of its implementation modality. Components with high burden and greater effectiveness (eg, infections and care for low birthweight babies) should be strengthened, whereas components with lower burden and less effectiveness (eg, asphyxia) should be removed especially for FCHVs. With respect to implementation modality, more emphasis must be placed on focused, high-quality training of all involved healthcare providers and ongoing supervision and support.
The CBNCP has been scaled up to 39 districts of Nepal. The findings presented here, which were previously shared with CBNCP stakeholders, and a move towards more integrated approaches to improve child survival prompted a removal of selected components and integration of CBNCP interventions with the Integrated Management of Neonatal and Childhood Illness (IMNCI) programme. The IMNCI programme is currently being implemented in 35 districts and monitored in terms of programme coverage, quality and impacts on behaviours, health and equity.
We would like to acknowledge the USAID-funded MEASURE DHS for providing us with the Nepal DHS dataset and the MOH for sharing HMIS data. DP undertook this analysis as part of his research under the PhD programme at the Munich Center for International Health and was funded through a scholarship offered by the German Academic Exchange Service. At the time, DP was an employee of USAID and was offered flexible working hours and time off to undertake this study as part of his PhD dissertation. We would also like to thank Jamie Bartram, Steve Hodgins and Ulrich Mansmann for their helpful comments on a previous version of this manuscript as well as Mary Adam, Jennifer Callaghan-Koru, Matthew Ellis and Zelee Hill for their thorough review and constructive feedback on the originally submitted manuscript.
Contributors DP, IBS and ER had the original idea for this paper. DP carried out data analysis and prepared the first draft. IBS, ER and MS advised on methods and interpretation of findings. IBS, ER and MS reviewed and revised the draft manuscript. All authors, except IBS because of his untimely demise during finalisation of this manuscript, read and approved the final manuscript.
Funding This study was undertaken without dedicated research funding but made possible through a PhD scholarship offered to DP by the German Academic Exchange Service. The study used DHS data in the public domain and HMIS data made available to the authors upon request. All data were processed and analysed by the authors with DP having full access to the data and all authors sharing the final responsibility for the decision to submit for publication. Neither those providing us with the data nor the German Academic Exchange Service had any involvement in data analysis, interpretation or writing of this manuscript.
Disclaimer The opinions expressed in this paper are those of the authors and do not necessarily reflect the views of USAID or the other organisations the authors are affiliated with.
Competing interests At the time of study, DP was an employee of USAID and involved in monitoring the CBNCP programme.
Patient consent Not applicable
Ethics approval Nepal Health Research Council (NHRC) and LMU PhD Ethical Review Board waived the review process based on NHRC approval.
Provenance and peer review Not commissioned; externally peer reviewed.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.