Article Text
PDF
Abstract
Introduction Mobile Vaani was implemented as a pilot programme across six blocks of Nalanda district in Bihar state, India to increase knowledge of rural women who were members of self-help groups on proper nutrition for pregnant or lactating mothers and infants, family planning and diarrhoea management. Conveners of self-help group meetings, community mobilisers, introduced women to the intervention by giving them access to interactive voice response informational and motivational content. A mixed methods outcome and embedded process evaluation was commissioned to assess the reach and impact of Mobile Vaani.
Methods The outcome evaluation, conducted from January 2017 to November 2018, used a quasi-experimental pre–post design with a sample of 4800 married women aged 15–49 from self-help group households, who had a live birth in the past 24 months. Surveys with community mobilisers followed by meeting observations (n=116), in-depth interviews (n=180) with self-help group members and secondary analyses of system generated data were conducted to assess exposure and perceptions of the intervention.
Results From the outcome evaluation, 23% of women interviewed had heard about Mobile Vaani. Women in the intervention arm had significantly higher knowledge than women in the comparison arm for two of seven focus outcomes: knowledge of how to make child’s food nutrient and energy dense (treatment-on-treated: 18.8% (95% CI 0.4% to 37.2%, p<0.045)) and awareness of at least two modern spacing family planning methods (treatment-on-treated: 17.6% (95% CI 4.7% to 30.5%, p<0.008)). Women with any awareness of Mobile Vaani were happy with the programme and appreciated the ability to call in and listen to the content.
Conclusion Low population awareness and programme exposure are underpinned by broader population level barriers to mobile phone access and use among women and missed opportunities by the programme to improve targeting and programme promotion. Further research is needed to assess programmatic linkages with changes in health practices.
- Nutrition
- Human resource management
- EDUCATION & TRAINING (see Medical Education & Training)
Data availability statement
Data are available upon reasonable request.
This is an open access article distributed in accordance with the Creative Commons Attribution 4.0 Unported (CC BY 4.0) license, which permits others to copy, redistribute, remix, transform and build upon this work for any purpose, provided the original work is properly cited, a link to the licence is given, and indication of whether changes were made. See: https://creativecommons.org/licenses/by/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
The evaluation used advanced analytical approaches to address low exposure in the outcome evaluation.
The embedded process evaluation captured the design of how the programme was integrated among self-help group members within a complex sociocultural environment.
Since the measurement of exposure was based on reported estimates, future programmes and evaluations should ensure that interactive voice recording call data records are accessible and linkable to survey data.
Introduction
When the National Family and Health Survey-4 results came out in 2016–2017, Bihar, India’s third largest state (population 99 million), had lower than national averages for most maternal and child health outcomes.1 In rural Bihar, half the children under 5 and nearly a third (32%) of mothers were underweight as compared with the national figures of 38% and 27%, respectively.1 2 One in five (21%) married women reported having an unmet need for family planning—an estimate twice that of the national average. The incidence of diarrhoea among children under 5 within the 2 weeks preceding the survey was 11% and less than half of those cases were managed with oral rehydration solution.2 Against this backdrop, the government of Bihar, with the support of the Bill and Melinda Gates Foundation, established a range of public health programmes to improve health systems and bolster health outcomes.
A maternal and child health and nutrition programme was established in 2017, with implementation support provided by Project Concern International.3 This programme was designed to share health information with women who were members of microfinance-based self-help groups. (A self-help group comprises 8–12 women who live nearby, meet weekly, save small amounts of money to take out loans internally or with banks at small interest rates, and discuss matters related to finance, livelihoods and health.)3 4 The target audience were mothers of children under 2 years of age who were either self-help group members themselves or were living in a home where at least one other woman residing in the same household was a member of a group. Prior studies had shown that sharing messages on correct maternal and child health and nutrition practices in weekly self-help group meetings coupled with home visits by group leaders and convenors of group meetings, called community mobilisers to women with young children, among others, may improve some maternal and child health outcomes, such as antenatal care visits, early initiation of breast feeding and clean cord care.4–13 (Community mobilisers are women who support the functioning of ~10 self-help groups within their village. They facilitate weekly meetings for each self-help group.) In an effort to bolster these and other outcomes, including family planning and complementary feeding, the government of Bihar sought to implement and evaluate new and innovative approaches including mobile health solutions which had experienced some success in reinforcing information to target audiences, within selected geographies.4–8 14–49 Further, as half the state’s rural population owned a mobile phone, this new innovative programme held potential.30 50–52
Mobile Vaani, developed by Gram Vaani and launched in January 2017, was a direct to beneficiary mobile health communication programme comprising a two-way messaging system which included outbound interactive voice response calls and a mechanism for beneficiaries to call in and record their own content. Trained community mobilisers, with support from programme field officers, promoted awareness of the Mobile Vaani programme in self-help group meetings. At regular intervals, group members were reminded about the programme, and taught how to dial a toll-free number in order to generate a missed call. Within 30 s to 2 min of receiving the missed call, the system would call the number back and if answered, the listener received a series of prerecorded WHO-approved health information content on nutrition,53 family planning and management of diarrhoea. Call content spanned across four priority health areas (child nutrition, maternal nutrition, family planning and management of diarrhoea) and was comprised of two types of messages: those prerecorded in a studio with actors reading off a script, or messages spontaneously recorded by self-help group women in the same district. The latter were reviewed and edited before being broadcast. Content was shared through story format which centred around the knowledge and benefits to women, children and families of appropriate maternal and child health and nutrition practices. Listeners were further encouraged to share information about Mobile Vaani with their family members, friends and neighbours, and to foster a dialogue and conversation with others on the topics listened to.
This manuscript presents findings from a mixed methods evaluation conducted with the support of JEEViKA, the government-led entity that establishes and runs self-help groups in Bihar. The evaluation was commissioned as very few interactive voice recording-based programmes had been carried out targeting hard-to-reach marginalised populations and most of these programmes had not been evaluated.16 22 23 Further, even though Mobile Vaani had been implemented in the general population before, little was known about its reach, the perceptions of the content by its intended audience, or its impact on knowledge for key maternal and child health and nutrition practices among women within self-help group households.54 55
Findings on the programme’s reach and impact on women’s knowledge of key maternal and child health and nutrition indicators captured through the outcome evaluation are shared in this manuscript. It further highlights key results from an embedded process evaluation that helps to explain how the intervention was integrated into the existing system, how the intended beneficiaries engaged with the programme and benefited from it, and how beneficiaries engaged with the technology. Study findings are anticipated to provide important insights into the impact direct to beneficiary solutions have in bolstering knowledge among women.
Methods
Evaluation design and conceptual framework
The mixed methods evaluation design consisted of an outcome evaluation and an embedded process evaluation.
The outcome evaluation used a quasi-experimental before and after study design to identify changes in knowledge on maternal and child nutrition, family planning and management of diarrhoea among self-help group members with a child under 2 years of age. Six intervention blocks were selected by the implementers for programming; six comparison blocks were matched by the evaluators based on the sociodemographic characteristics of female literacy, scheduled caste/schedule tribe and concentration of self-help groups by population.
The process evaluation sought to garner in-depth knowledge on the functioning of Mobile Vaani, to assist in interpreting outcome results, and inform potential replication.27 56–65 Using a realist evaluation approach, the process evaluation helped in building an overall understanding of (1) the system’s readiness to accept the intervention, that is, the ability of the community mobilisers to understand Mobile Vaani and share its key tenets with their members, (2) a behavioural response to the programme, as determined by the self-help group members’ receptivity to accept, use and disseminate Mobile Vaani with others; and (3) the technological context within which the intervention was being implemented, that acted as an enabler or barrier to the uptake and utilisation of the programme (figure 1).66 67
Conceptual framework outlining key components of process and outcome evaluation.
Data collection and sampling
Outcome evaluation: cross-sectional surveys
For the outcome evaluation, cross-sectional surveys were carried out at two time points: (1) baseline survey (n=2436) from October 2017 to January 2018; and (2) endline survey (n=2498) 12 months following implementation from November 2018 to January 2019 (table 1). Cross-sectional surveys were powered to detect at least 7% net change in knowledge on core themes among the target population after 12 months of intervention. Assuming 80% power, and a design effect of 1.5, a sample size of 1200 self-help group households in each arm was sufficient to detect this change. Study respondents were eligible women, aged 15–49 years, currently married, having had a live birth in the last 24 months preceding the survey and were from self-help group households. At endline, women were cross sectionally selected from the same self-help groups. Less than 1% of the data collected across the variables of interest were missing and they were missing completely at random.
Details of data collection points
In the surveys, the women were asked about their engagement with Mobile Vaani and the intervention’s impact on their knowledge of: maternal and child nutrition, family planning and management of diarrhoea. Data were collected by female enumerators (n=40) with prior experience in conducting similar surveys. Enumerators were grouped in teams of five, including one designated mentor per team, and a supervisor who accompanied them to households, supported logistics, and checked completed study questionnaires. All data were captured on mini-laptops using CS-Pro V.7.1 software. Quality control occurred at three levels: (1) supervisory oversight of individual data collectors; (2) 10% spot checks of interviews by a team mentor; and (3) outlier analyses of entered data by the core research team.
The process evaluation was achieved through a triangulation of three primary data sources that were collected between November 2017 and November 2018 and were used to measure the system’s readiness and the behavioural response to the intervention (table 1). Secondary data were analysed to better understand the technological context around the intervention.
Process evaluation: system readiness
System readiness was measured through surveys with community mobilisers and observing self-help group meetings where community mobilisers talked about Mobile Vaani with their members. Surveys with 116 community mobilisers were carried out among those community mobilisers who had attended at least two trainings with field officers. After obtaining written consent from the respondents, data were captured in Hindi using computer-assisted personal interview software on laptops by trained female enumerators. Direct observations of self-help group meetings were conducted with the same community mobilisers who had been interviewed previously. These observations helped document what and how community mobilisers shared information about Mobile Vaani with their members. Trained investigators filled in structured questionnaires. Verbal consent was sought from all participants.
Process evaluation: behavioural response
Behavioural response was captured through 180 qualitative interviews carried out among selected self-help group women who knew of Mobile Vaani. They were interviewed in Hindi by trained investigators. Data were captured through audio recordings after obtaining written consent from the respondents. The individual interviews lasted an average of 40 min. Audio files were transcribed and then translated to English. Transcripts were imported into a qualitative software programme, Atlas.ti V.6.2, for coding and in-depth analysis.
Process evaluation: technological context
The technological context was captured through secondary analyses of system-generated interactive voice recording data, which included 2.6 million call records from 44 664 unique phone numbers. Call records, take took place from February 2017 to July 2018, were obtained through the implementors and were analysed to study calling and listening patterns and to identify any potential technical hindrances or barriers users may have faced.
Variables (Table 1)
For the outcome evaluation, seven key dependent variables were measured. These included women’s knowledge on three indicators related to child nutrition, namely: that complementary feeding for an infant was to be initiated between 6 and 12 months, what the correct consistency of food for a child 6–12 months old was, and how a child’s food could be made nutrient and energy dense. The indicator capturing knowledge around maternal nutrition included the elements of adequate nutrition for a pregnant or lactating woman. Two indicators on family planning captured the respondents’ knowledge of at least two modern spacing methods of family planning, and knowledge that frontline workers could provide short-term family planning methods, for example, condoms/pills to women in need. The seventh indicator captured the interviewees’ knowledge of the symptoms of diarrhea and its management. There were two key dichotomous independent variables of interest; one was the study arm, that is, intervention or comparison, and the other was the survey round, that is, baseline and endline. Control variables were included as these indicators were known to influence findings around the key dependent variables in previous studies.1 68 69 These indicators were: women’s parity (captured as an average measure across all the women in the category), number of completed years of formal education (categories: no formal education, completed 1–5 years of schooling, 6–9 years of schooling and 10 or more years), any exposure to mass media over the past year (yes/no), belonging to the Hindu religion/not, belonging to a scheduled caste/tribe or not, household wealth (measured in quintiles) (The household wealth index was developed through principal component analysis using information on 26 household assets measured for six categories, five housing characteristics and one on asset ownership, taken from a nationally representative tool. Using these data, wealth quintiles were developed based on equal proportions of the population into five categories, from the first quintile being the poorest to the fifth quintile being the richest.), respondent is a self-help group member/not, average duration of membership (in months), living in a nuclear family structure/not and if the respondent had any contact with a frontline worker during pregnancy and 7 days postpartum.1 68 69
Process evaluation: system readiness
In order to determine whether the community mobilisers were sharing adequate information about Mobile Vaani in their groups, community mobilisers were assessed on their knowledge around four key topics Mobile Vaani intended to share with self-help group women. This was captured through 31 knowledge-related questions asked to each of the community mobiliser with a correct answer to each question garnering a point; the scores for each of the responses were summed up to form a continuous variable. Through observations of self-help group meetings, community mobilisers were also assessed on whether they could demonstrate to their self-help group members how to dial a number on a mobile phone, how to access Mobile Vaani and the key features of the programme. Other variables of interest were the number of messages community mobilisers reported being trained on (maximum being four: child nutrition, maternal nutrition, family planning and management of diarrhoea); and other characteristics of the community mobilisers, that is, age (captured as an average), caste (belonging to a scheduled caste/tribe or not), education (completed 1–10 years of schooling, 11–12 years of schooling and college/more), whether she is engaged in another occupation or not, number of years she has been a community mobilisers (0–2 years, 3–4 years, 5+ years), and exposure to none/partial access to a mobile phone or complete access to a mobile phone. (Full exposure to mobile phone meant owning a phone or having access to one 24 hours a day, and knowing how to dial a number. Partial access meant one of the elements of full exposure were missing.)
Process evaluation: behavioural response
From the qualitative interviews that were held with women, four broad themes were identified and are presented in this article as they shed light on the outcome evaluation and help explain some of the enablers and barriers to the uptake of the programme. These themes include the women’s perspectives on how Mobile Vaani was integrated into self-help group meetings, the content they preferred listening to, the stories they resonated with and shared with others, and some of the barriers they faced in accessing the programme. Key indicators such as the age and educational level of the women who were interviewed were also captured.
Process evaluation: technological context
The purpose for analysing the system-generated interactive voice recording data was to capture the lifecycle of a phone number within the system. The key variables included: a missed call, that is, the user called the programme generating a missed call, but the call was never answered back; a call back where a missed call from the user generated a call back from the system within 30 min of the user calling the system; and, an outbound dial, where the system called a registered user at periodic intervals on its own accord. In addition, the calls sent by the system were further analysed to determine whether they were answered or not answered by the users. If the calls were answered, the duration for which the users listened to the content were categorised in seconds and minutes. Further, reasons for why calls were not answered were categorised into network issues, problems generating from the programme itself or from the user, such as, the user not picking up the call.
Statistical procedures
Outcome evaluation: cross-sectional surveys
Differences between women’s individual characteristics, that is, covariates, across the intervention arms between baseline and endline were tested using ordinary least-squares (for continuous variables) and through logistic regression models (for categorical variables) (table 2). For instance, women’s parity and self-help group membership duration were measured as continuous variables, and hence tested using ordinary least-squares regression. The remaining variables were categorical, and hence tested using logistic regression model. In addition, the findings were adjusted for random clustering effect at the block and village levels by using multilevel mixed-effect regression analyses.
Sociodemographic characteristics of the women interviewed in the cross-sectional surveys of the outcome evaluation
To assess the net change in the seven outcome variables of interest from baseline to endline, two types of multivariate analyses were carried out: one, a difference-in-difference analysis and two, a treatment-on-the-treated approach was applied using instrumental variable analysis (table 3). The instrument variable was those who were exposed to Mobile Vaani—as captured by those women who recalled hearing about Mobile Vaani or who recalled listening to content on it—thus controlling for endogeneity to programme exposure.70–72 These analyses were adjusted for covariates and for random clustering effect at the block and village levels. Samples with missing data were dropped from the multivariate analyses as the sample lost was negligible. All analyses were done using Stata V.16.0 software programme.
Results from treatment-on-the-treated effect of Mobile Vaani intervention on knowledge of selected maternal and newborn health practices for outcome evaluation, 2017–2019
Process evaluation: system readiness
Bivariate and descriptive analyses were conducted on key indicators from the quantitative interviews conducted with community mobilisers and observations from self-help group meetings (table 4). The aim of the analyses was to capture the association of three key indicators that assessed the capabilities of the community mobilisers (knowledge on Mobile Vaani themes, ability to dial a number and explain all the features of the programme) with other characteristics of interest, for example, their level of training, their sociodemographic profile, etc.
Findings showing association of characteristics of community mobilisers to sharing Mobile Vaani in self-help group meetings (N=116)
Process evaluation: behavioural response
The interview guides that the investigators used in the field covered several topics that were of interest to the evaluation. Once the interviews had been transcribed and imported into the qualitative software programme, the transcripts were tagged with 70 codes that were generated from coding the transcripts. The codes were then brought together under nine broad themes of which four relevant themes are shared here.
Process evaluation: technological context
To analyse system-generated data, large volumes of data on phone calls to and from the system were merged. Cascade analyses of unique mobile numbers were conducted to determine user’s engagement with the Mobile Vaani platform over time. For this, a lifecycle of each phone number was created. While crafting out the journey of each unique number’s lifecycle, all the calls made by or to Mobile Vaani by the number were considered as exposures to Mobile Vaani. Further, durations of each call and reasons for a call not going through were also studied.
Results
Outcome evaluation
Table 2 present summary characteristics for the sample population used in the cross-sectional survey. Findings suggest that more than half the women had no formal education, >90% belonged to Hindu religion, and >30% were members of the most disadvantaged schedule caste/tribe. While the majority of the sample population reported having had contact with a frontline worker (FLW) during their last pregnancy, less than half had a postnatal visit with a FLW 7 days post delivery. Self-help group membership was generally high with about 61%–78% of the respondents in each of the study arms being a member themselves; the remaining eligible women interviewed came from a household where another adult female family member was a self-help group member. On average, self-help group members had been associated with their group for almost 3 years. For most sociodemographic characteristics assessed, the differences between study arms were not statistically significant suggesting that randomisation was successful for the study design.
At endline, 17% (n=223) of the women in the intervention arm could recall listening to Mobile Vaani content at least once. Among them, most recalled listening to content on the themes of maternal nutrition (38%), child nutrition (35%), family planning (13%) and diarrhoea management (12%). Health and nutrition messages were reported by 77% of women listeners to be the most useful.
Table 3 depicts results from the difference-in-difference and treatment-on-treated effect of the Mobile Vaani intervention on knowledge for selected maternal and child health practices. Women’s knowledge of the correct consistency of food for a child 6–12 months of age was significantly higher over time across study arms (difference-in-difference: 6.6% (1.1%–12.2%) p=0.019) but not based on awareness of Mobile Vaani (treatment-on-treated: 14.4% (–6.1%–35%), p=0.169). Women’s awareness of the need to make a child’s food nutrient and energy dense increased significantly across study arms (difference-in-difference: 9.3% (4.5%–14.1%) p≤0.001) and among those with any awareness of Mobile Vaani (treatment-on-treated: 18.8% (0.4%–37.2%) p=0.045). Women’s knowledge of proper nutrition for pregnant or lactating women increased in both groups over time but more substantially in the comparison arm from baseline to endline (intervention: 73.6%–77.0%; control: 69.5%–80.9%; difference-in-difference: −8.0% (−12.7% to −3.3%) p=0.001). Significant differences were not observed among those with any awareness of Mobile Vaani as compared with those with no awareness (treatment-on-treated: −0.3% (−16.8%–16.2%) p=0.971). Women’s awareness of at least two modern spacing methods of family planning was significantly higher in the comparison arm across study arms (difference-in-difference: −4.7 (−8.2%–1.2%), p=0.0009) but when exposure to Mobile Vaani was considered, the opposite trend was observed and, a 17.6% (4.7%–30.5%; p=0.008) increase in knowledge was observed among those with any awareness of Mobile Vaani. Any awareness of the symptoms of diarrhoea in children under 2 years of age and its management was low overall but increased significantly across study arms (difference-in-difference: 6.0% (2.0%–9.9%) p=0.003). However, these findings did not stay significant when knowledge of symptoms and management of diarrhoea was considered among those with awareness of Mobile Vaani.
Process evaluation: system readiness
Table 4 depicts the characteristics of community mobilisers who were responsible for leading self-help groups and sharing information with self-help group members on Mobile Vaani. Community mobilisers were an average of 34 years of age, with 56% having more than 10 years of formal schooling, and 13% coming from the disadvantaged scheduled caste/tribe groups. Community mobilisers had similar knowledge scores irrespective of background characteristics. However, those who had received two or more trainings could recall more messages correctly. Community mobilisers who had worked for a greater duration of time (three/more years) in their role and those with higher education (more than 10 years of schooling) had greater competency in dialling a number and explaining Mobile Vaani to their members than their counterparts.
Process evaluation: behavioural response
Findings from in-depth interviews on the integration of Mobile Vaani into self-help groups provided insights into the barriers and facilitators to Mobile Vaani use (table 5). Community mobilisers were hailed as the key peer educators to integrate Mobile Vaani into self-help groups, by sharing various aspects of the intervention with members in self-help group meetings (theme 1). There was a general affinity towards the content shared through the intervention and reasons cited for this included it being their only source of news, and a useful medium for information on maternal health, nutrition and sanitation, which they could access at any time free of cost (theme 2).
Quotes from in-depth interviews with 180 self-help group women
When it came to preferences, while most women expressed that they enjoyed listening to stories of local women from neighbouring communities, their perceptions on the prerecorded stories and messages generated in the studio were mixed with some younger and more educated women enjoying the messages that were woven into hypothetical scenarios with older and less educated women finding the content, language and dialect hard to understand and the pace of the storyline too fast, in general (theme 3). Despite this, most women who were interviewed shared either the Mobile Vaani number or the information heard on Mobile Vaani at least once with their family members, neighbours and relatives with a few recounting stories of using the information they had learnt to help others in dire need.
Qualitative findings suggest that there were economic, gendered and comprehension-related barriers, which hampered the uptake of the intervention (theme 4). The most common reasons women shared that they did not listen to Mobile Vaani were a lack of time and no apparent benefit. Some women said that they had insufficient balance in their phones to give a missed call and a few women expressed doubts about Mobile Vaani being a free service. Beneficiaries also noted that calls came without warning at timings that were not always convenient or when women had access to mobile phones. Some respondents with migrant husbands found it problematic to be on a Mobile Vaani call as their spouses complained that their (women’s) phones were busy when the husbands called. Women also expressed apprehension in using mobile phones for long durations in front of male members of the household.
Technological context: barriers to use
Beneficiaries could access Mobile Vaani either by answering an outbound call sent by Mobile Vaani or by directly giving a missed call (inbound call) and then being called back by the system. A review of the call records of the 44 664 users suggests that over a few exposures, the proportion of users receiving call-backs decreased, while the proportion of users who received random outbound calls being generated by the system increased (figure 2).
Cascade of unique numbers engaging with Mobile Vaani over the course of the programme.
An analysis of 2.6 million call records showed that around 19% of the calls were missed calls generated by the beneficiaries and 81% of all calls were made by Mobile Vaani to the users (figure 3). Of these 81%, the majority were outbound calls (62%) made randomly by the system while the remaining 38% were call-backs generated as a result of a beneficiary sending a missed call to the system. Users’ response to outbound dialled calls and call-backs were explored further and found that even though the system made more outbound calls than call-backs, the proportion of users answering the calls was much higher for call-backs as compared with outbound dialled calls, suggesting that call-backs were more cost effective for Mobile Vaani. If calls were initiated by users, 50%–60% of the calls were answered as compared with calls that were pushed by the system (20%–25%). A quarter of outbound dialled calls were disconnected within 30 s of answering, and users spent longer listening to messages on call-backs. Majority of the calls, irrespective of how they were received, were disconnected because of issues at the users’ end which included ‘normal clearing/no answer/call rejected/user busy’. Additionally, the reasons for call disconnection at Mobile Vaani end were network congestion, incapacity to handle a large volume of calls by primary rate interface lines, and other technical problems within the Mobile Vaani system (figure 3).
User’s engagement with Mobile Vaani system.
Discussion
This study presents results from a mixed methods evaluation of the Mobile Vaani programme in two districts of Bihar. From the outcome evaluation, 23% of women from households with a self-help group member reported having heard about Mobile Vaani and 17% reported listening to content within the 12 months preceding the endline survey. Among women exposed to Mobile Vaani, qualitative findings suggest that they reportedly liked the programme, found it convenient to access, and enjoyed recording and sharing content with friends and neighbours. However, the programme was met with several existing contextual barriers such as lack of mobile literacy, poor phone infrastructure, inability to use mobile phones, gender and cultural barriers, lack of time, among others.73 The programme’s impact on women’s knowledge was mixed and limited to child nutrition and modern contraceptive methods. Overall findings underscore the challenges of trying to reach poor and marginalised groups of women with limited access to and skills to use mobile phones with digital interventions.
Study findings contribute to an emerging body of evidence on direct-to-consumer mobile health interventions in India.74 75 The mobile messaging service Kilkari which aims to provide new and expectant mothers and their families with health information messages across 13 states in India similarly had a significant impact on a few but not all health behaviours assessed, namely reversible modern contraceptive methods and child immunisations at 10 weeks.76 The limited impact observed there and in our evaluation of Mobile Vaani suggests that a more targeted approach to content delivery and one which considers differentials in digital access and skills may deepen impact.
Changing behaviours within communities takes time, concerted effort and reiteration of information on correct health practices, which can be achieved through repeated messaging and voice reminders using mobile phone technology through self-help groups.19 77 78 However, it is important to note that health interventions using mobile phone technology may not function effectively in isolation but instead could be incorporated as a tool for improving indicators within a holistic health systems approach. In addition, community mobilisers also serve as additional conduits to reaching women with key health and nutrition messaging, besides frontline workers who often bear the burden of being the primary interlocutors on health issues.79–81 However, care should be taken not to overburden the community mobilisers whose primary role is to support self-help groups.
Mobile Vaani’s impact on knowledge was driven by programme coverage and exposure which were assessed based on reported awareness of and engagement with the programme. Future evaluations might consider linking system generated data, including call data records, with survey data assessing outcomes. Population level coverage of Mobile Vaani among pregnant women was 23% based on reported survey estimates. An effort to validate this measure using call data records yielded a similar estimate. While the Kilkari programme enrolled targeted women using government tracking registries, their overall population level coverage was 21% highlighting the challenge in reaching key audiences through mobile phone numbers within the Indian context.82 83
Reported exposure to Mobile Vaani was based on self-reported estimates and not call data records because of challenges linking varied data sources. Exposure was impeded by the limited number of self-help group members with mobile phones and children under 2 years of age (target population). Broader systematic challenges in women’s access to and use of mobile phones persist in Bihar and elsewhere throughout India.80 84 85 Women in the study sample reported having limited access to mobile phones, restricted by work and domestic commitments, coupled with limited digital skills. Further, patriarchal gender norms were omnipresent and restricted their use of mobile phones.85 86 Similar contextual factors have been highlighted in explaining low outreach of mass media as well, including newspapers, radio and television, among women in rural Bihar.1 2 26
The evaluation highlighted the possibility to scale up the intervention with several caveats in mind. First, a space would need to be identified where a wide community could be reached that promoted ownership and sustenance of the intended intervention. Second, evidence-based and systematic planning would be needed to design a robust programme to achieve its desired take-up by reaching intended audiences through capacity building and extensive promotion of the programme. Adequate coverage of the programme would be essential for success. Third, the content being played through the interactive voice recording system would need to focus on topics of interest and be relatable to the audience to sustain its programming. As the audience would expand to new populations, this would require additional formative research and continued monitoring to tweak the programme and allow for mid-course corrections. Fourth, for optimal results, the programme ought to coordinate its content development and dissemination with any parallel programme efforts disseminating health and nutrition information to the intended audience. Fifth, the programme would need to identify and account for various sociodemographic and economic differences with the intent of reaching different audiences equally and not exacerbating disparities of access across socioeconomic characteristics. Other scholars have also captured this phenomenon of intersecting inequalities, especially in terms of gendered phone access gaps and financial constraints.46 73 80 87 Sixth, a robust registration and follow-up system of registered phone numbers would be pivotal in ensuring that messages would reach their intended audience at the appropriate time.
Though intervention and comparison arms were matched successfully, with the treatment and control groups generally comparable for their basic characteristics, this evaluation did have some limitations. Baseline and endline samples were not always balanced across the intervention and comparison arms, but key covariates were controlled for and accounted in the analysis. The study used fresh cross-sections of respondents at both time points of survey; however, respondents at endline belonged to the same self-help groups that were included at baseline, and therefore were representative of similar geographical and socioeconomic conditions. As the study only engaged self-help group members, it may not have considerable external validity outside of this population. The exposure among the intended audience was low, but this was adjusted for in the analyses using an instrumental variable of recalled exposure to the programme. The recall bias among respondents was also accounted for through rigorously trained investigators and pilot testing the survey tools, which were formulated based on established modules from similar surveys.
Conclusion
When introduced to Mobile Vaani through self-help groups, women’s knowledge of key health and nutrition outcomes improved for two out of seven key indicators. The study provides evidence that mobile-based interventions could evolve into a modality to reach marginalised populations through systematic planning when coupled with other interventions on the ground. It would further be advisable for programmes to remain conscious of the context, and enablers or barriers driven by socioeconomic and gendered realities of the target audience, before operationalising capital intensive interventions.
Data availability statement
Data are available upon reasonable request.
Ethics statements
Patient consent for publication
Ethics approval
This evaluation was approved by the evaluation agency, Population Council’s institutional review board. The public was not involved in the design, conduct, reporting, or dissemination plans of this research. Prior to an interview, participants received written information on the survey as well the assurance of confidentiality. Written consent was taken from all interviewees, and interviews were conducted in a private space.
References
Footnotes
Contributors LI served as the PI of the study designing the evaluation, overseeing the data collection, leading the analysis and interpretation of results. LI was the primary author of the paper, had full access to the data, controlled the decision to publish and accepts full responsibility for the conduct of this study, as the guarantor. S, RM, RKV, AN and SS led the data collection, contributed to data analysis and visualisation, and wrote early drafts of the methods and results sections of the manuscript. DM contributed to data analysis, interpretation of results and critical revision of the manuscript. DD and AELF contributed to the plan and design of the study, contributed to data analysis, interpretation of results and critical revision of the manuscript for important intellectual content. AS, IC, MRC, AP, AG contributed to designing the study, assisted the evaluators to collect primary data and access secondary data, and provided critical insight into the interpretation of the results and revisions of the manuscript. All authors have reviewed and approved the final version of the manuscript.
Funding This study was supported by the Bill and Melinda Gates Foundation [grant number OPP1141832].
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.