Measuring open defecation in India using survey questions: evidence from a randomised survey experiment

Objectives To investigate differences in reported open defecation between a question about latrine use or open defecation for every household member and a household-level question. Setting Rural India is home to most of the world’s open defecation. India’s Demographic and Health Survey (DHS) 2015–2016 estimates that 54% of households in rural India defecate in the open. This measure is based on a question asking about the behaviour of all household members in one question. Yet, studies in rural India find substantial open defecation among individuals living in households with latrines, suggesting that household-level questions underestimate true open defecation. Participants In 2018, we randomly assigned latrine-owning households in rural parts of four Indian states to receive one of two survey modules measuring sanitation behaviour. 1215 households were asked about latrine use or open defecation individually for every household member. 1216 households were asked the household-level question used in India’s DHS: what type of facility do members of the household usually use? Results We compare reported open defecation between households asked the individual-level questions and those asked the household-level question. Using two methods for comparing open defecation by question type, the individual-level question found 20–21 (95% CI 16 to 25 for both estimates) percentage points more open defecation than the household-level question, among all households, and 28–29 (95% CI 22 to 35 for both estimates) percentage points more open defecation among households that received assistance to construct their latrines. Conclusions We provide the first evidence that individual-level questions find more open defecation than household-level questions. Because reducing open defecation in India is essential to meeting the Sustainable Development Goals, and exposure to open defecation has consequences for child mortality and development, it is essential to accurately monitor its progress. Trial registration number Registry for International Development Impact Evaluations (5b55458ca54d1).

Because reducing open defecation in India is essential to meeting the Sustainable Development Goals, and exposure to open defecation has consequences for child mortality and development, it is essential to accurately monitor its progress. trial registration number Registry for International Development Impact Evaluations (5b55458ca54d1).

IntrOduCtIOn
Rural India is home to more than half of the world's open defecation. 1 Because the persistence of open defecation threatens gains in child health, the Sustainable Development Goals (SDGs) call for its elimination by 2030. Progress towards eliminating open defecation in rural India will be essential to meeting this goal. India's most recent Demographic and Health Survey (DHS), conducted between January 2015 and December 2016, estimates that 54% of households in rural India defecated in the open, down from 75% in the 2005 to 2006 DHS. 2 3 This measure is based on a household-level question that asks about the behaviour of everyone in the household in the same question.
Recent evidence from studies carried out in India suggests, however, that it is common for individuals living in households with latrines to nevertheless defecate in the open. In rural parts of five north Indian states, Coffey et al found that 21% of individuals defecated in the open, despite owning a latrine. 4 In rural Tamil Nadu, Yogananth

Open access
In Odisha, Barnard et al found that less than half of members of households with latrines reported using their latrines at all times. 6 Experimental studies of sanitation interventions have found similar results. Clasen et al report on a sanitation intervention in Odisha and note open defecation among individuals living in households with latrines as a reason for not observing impacts on child health outcomes. 7 Patil et al conducted a sanitation intervention in Madhya Pradesh and experienced a similar problem: modest increases in latrine coverage, and even more modest reductions in open defecation. 8 These findings suggest that open defecation among latrine-owning households is substantial. Since it is probable that latrine use is the socially desirable response to questions on sanitation behaviour, measures based on household-level questions, such as those from the DHS, will likely underestimate true open defecation in rural India, particularly among households with latrines.
Because open defecation is an individual behaviour, an individual-level survey question may be able to more accurately measure it compared with a household-level question, particularly among households with latrines. We designed this study to experimentally test this hypothesis in rural India. We aimed to investigate whether a balanced question about latrine use or open defecation for every member of a household finds different levels of open defecation compared with a household-level question.
This is the first study to experimentally vary survey methodology to improve on the measurement of open defecation currently being used. Jenkins et al study sanitation survey methods, and report on an index they develop for quantifying household excreta disposal. 9 Their study focuses on developing and piloting a new tool rather than comparing different measures. Sinha et al compare answers to survey questions on latrine use behaviour to measures of actual behaviour generated from passive latrine use monitors that were set up in the latrines of respondents, and find poor to moderate agreement between the two measures. 10

MethOds
The study is registered in the Registry for International Development Impact Evaluations.
sample: mostly latrine-owning households in rural parts of four states This study uses as its sampling frame the study areas of 3ie's Promoting Latrine Use in Rural India Thematic Window. This window has funded four independent research teams to conduct randomised control trials of distinct behavioural campaigns to promote the use of pit latrines in rural parts of Bihar, Gujarat, Karnata and Odisha. The study areas are spread across India, representing different contexts and varying levels of rural open defecation.
Because these trials focus on behavioural strategies rather than latrine construction, they are being carried out in villages that had high levels of coverage of pit latrines at baseline, relative to other rural parts of the same states. The households that comprise the sampling frame for this study are those that were identified as having a functional latrine in a census conducted by the research teams in the villages in which they were working. In all states except for Odisha, only households that had been excluded from the research teams' samples could be selected for this study. In Odisha, the sample selected for this study overlaps with the research teams' sample. We aimed to survey households that own latrines because we expect that an important source of misreporting of open defecation comes from individuals who do not use the functional latrines that their households own, and that other members of their households use. Figure 1 describes the sample selection. The villages visited in each state were randomly selected from the full set of villages included in the 3ie research teams' studies. The full set of villages were selected by the research teams in collaboration with the implementation agencies they were working with. The research team led by Oxford Policy Management worked with World Vision in Bihar, the team led by London School of Hygiene and Tropical Medicine worked with Coastal Salinity Prevention Cell in Gujarat, Eawag worked with Wateraid in Karnataka, and Emory University worked with the Rural Welfare Institute in Odisha. Data for our study were collected in 22-25 villages in each of the four study areas. In most areas, we sampled more villages than we actually visited in order to facilitate coordination with the research teams. Ninety-five villages were visited in total.
Up to 40 households in each village were randomly assigned to receive the household or individual questions. In some villages, fewer than 40 households were assigned because fewer than 40 households met the eligibility criteria. The survey team visited as many assigned households as it could in these villages, given time constraints, and availability of household members. On average, the survey team interviewed 25 households per village. Data collection took place between March and July 2018. Sample selection. Households that were started but not completed refers to households that refused at the beginning or part of the way through and households in which a suitable respondent was not available.
In each state, data collection took place after the 3ie research teams had conducted their censuses and baselines, but before they had started their interventions. Since in all states, the households visited in this study were also visited at the time of the census, response bias may be a concern. This would not, however, impact the internal validity of this study since randomisation generates equal response bias, in expectation, across treatment arms. randomisation and masking: random variation in latrine use questions at the household level We randomly assigned the type of latrine use question administered in the survey at the household level. Roughly half of the households were assigned individual-level questions on latrine use. The other half were assigned a household-level question. One of the authors who was not involved in data collection carried out the randomisation using a random number generator in Stata. Because of the nature of the study, it was not possible to blind the respondents or surveyors to the type of survey question administered in the survey. However, in the interest of data quality, respondents were not explicitly told that the primary purpose of the survey, which took ~25 min to complete, was to measure open defecation. Additionally, surveyors did not know which survey question had been assigned to a household until starting the survey with the household. This was facilitated through SurveyCTO, the mobile data collection platform used in the study, which was programmed to store the randomisation assignment for each household ID prior to the commencement of data collection. In the field, surveyors were only given a list of households to interview. When a surveyor had correctly identified a household and was ready to start the survey, she would enter the household ID into SurveyCTO, and SurveyCTO would automatically start the questionnaire type assigned to the household.
The individual-level questions asked for every household member age 5 or older whether the individual defecated in the open or used the latrine. The preface to this series of questions was: 'I have seen that some people defecate in the open, and some people use the latrine. Now I want to ask about where you and your family members defecate'. Then, the surveyor asked the following question for each individual in the household, and coded the answer in a household roster: 'The last time [name of household member] defecated, did [name of household member] defecate in the open or use the latrine?'. The answer options included latrine, open and somewhere else. Surveyors used the last option, which meant that the household member defecated in a bedpan, cloth or other place, in <0.5% of cases. Because the priming statement and the behaviour question include both open defecation and latrine use, they are balanced between the two different behaviours and could reduce social desirability bias. The surveyor asked household members who were participating in the interview directly about their behaviour, and asked the main respondent, in most cases an adult female member of the household, to report on the behaviour of their family members who were not participating in the interview.
The rest of the households were assigned the household-level question used in India's DHS: 'What kind of toilet facility do members of your household usually use?'. 11 The answer codes were also the same as those Open access used in India's DHS: flush to piped sewer system, flush to septic tank, flush to pit latrine, flush to somewhere else, flush to don't know where, ventilated improved pit or biogas latrine, pit latrine with slab, pit latrine without slab or open pit, twin pit or composting toilet, dry toilet and no facility or uses open space or field. We also included an individual-level question on mobile ownership or preferring vegetarian food versus non-vegetarian food in the surveys that asked the household-level question so that both types of surveys would take approximately the same amount of time to complete.
There are three main factors that differ between the two types of latrine use survey modules: the level of aggregation, the reference period and the presence of a priming statement. Therefore, the differences in reported open defecation that we observe reflect the fact that the two sets of questions vary on all of these factors combined. We also conduct subgroup analyses. These analyses investigate differences in the same primary outcome measure, reported open defecation, but look at differences by question type among different subgroups. First, we investigate whether the difference in reported open defecation by question type depends on whether the latrine was constructed privately, or with assistance from the government or a non-governmental organization (NGO). In practice, assistance to construct latrines often comes from the government, but sometimes NGOs get involved in facilitating the implementation of the government programme. As part of the SBM, the Government of India assists rural households to construct latrines either by providing financial assistance directly to households so they can construct their own latrines or by local government officials constructing latrines for households. In the discussion that follows, we will describe a household as having 'received help' if it received financial assistance or a partially or completely constructed latrine from the government or an NGO.
The Indian government promotes and constructs latrines with pits that are ~60 ft 3 . 12 However, many rural Indians aspire to construct latrines with pits that are much larger, so that they can avoid emptying the pit, a task that is associated with ritual pollution. [13][14][15][16] Compared with latrines constructed privately, those constructed with government help are less likely to be used due to concerns over purity and pit emptying. Since a large fraction of rural households are likely to receive latrines with help from the government as a result of the SBM, it is important to explore how much open defecation different types of latrine use questions measure, based on having received help to construct the latrine.
The second subgroup analysis investigates whether the difference in measured open defecation between the two question types is statistically different for males compared with females. Sex differences are an important aspect to explore because observational studies have found consistently higher open defecation among latrine owners for males compared with females. 4 This observation could reflect greater demand for latrine use among females due to, for instance, greater psychosocial stress experienced when defecating in the open, 17 or it could be because of cultural norms that keep females in their reproductive years inside the home.
Means and differences in reported open defecation, by question type, are calculated using ordinary least squares regression with cluster robust SEs, clustered by village. Statistical analyses were conducted using Stata V.11.

Public involvement
The individual-level questions used in this study are the product of a deliberative process between the authors and research teams from Oxford Policy Management, London School of Hygiene and Tropical Medicine, Indian Institute of Public Health Gandhinagar, Eawag and Emory University. The findings of this study contribute to our understanding of the scale of an important public health problem.
results Table 1 shows summary statistics for households assigned the two types of latrine use questions. The total sample consisted of 2431 households, which were approximately equally divided across question type in each of the study areas. There were no significant differences on measures relevant for latrine use between households assigned different types of latrine use questions. Households in both groups had approximately the same number of household members, fraction female, fraction Hindu, educational attainment of the household head, and asset ownership of 13 assets, including mobile phone, Cluster robust SEs, clustered by village, next to means in columns 1 and 2. 95% CI next to differences in column 3. *P<0.05, **p<0.01. None of the differences are significant at the 0.05 or 0.01 level, which is why there are no * or ** in the table.

Open access
electricity, radio, television, fan, mosquito net, bicycle, motorcycle, car, chair, gas stove, pressure cooker and shoes for everyone in the family. It is important for the validity of the results that the sample is balanced on religious composition, since studies have documented an association between household religion and latrine use. 18 19 As the study design intended, most households in both groups had a latrine. Conditional on having a latrine, 64% of households had gotten help from the government or an NGO to build the latrine, and the average pit size was 180 ft 3 . It is important that the sample is balanced on these two characteristics because, compared with latrines constructed privately, those constructed with government help are less likely to be used because of concerns over purity and pit emptying. [13][14][15][16] Finally, among households with latrines, ~80% of them appeared to the surveyor to be in use on observation.  received help to construct their latrines. Comparing these estimates to the sixth set of estimates in the figure, which include all households with latrines, suggests that the household-level question underestimates open defecation by more among those that received help to construct their latrines, compared with those that did not. The seventh row in table 2 shows that, using different methods for computing differences, individual-level questions measure 28-29 (95% CI 22 to 35 for both estimates) percentage points more open defecation than the household-level question, among those that received help to construct their latrines, compared with those that did not.
The first two columns in table 3 test whether the difference in measured open defecation between the two question types is statistically different among households that received help to construct their latrines compared with households that did not receive help to construct their latrines. The first column in table 3 uses individuals as observations, and the second column uses households. The coefficients in the third row represent the difference-in-differences estimate.
The difference in measured open defecation between the two question types is 18 (95% CI 11 to 25) to 19 (95% CI 11 to 26) percentage points larger among households that received help to construct their latrines, compared with households that did not, depending on the method used to compute the difference. Among households that did not receive help to construct their latrines, the individual-level questions find 10 (95% CI 5 to 15) percentage points more open defecation than the household-level question.
The last column in table 3 investigates whether the difference in measured open defecation between the two question types is statistically different among males compared with females. Using the full sample, individual questions find 4 (95% CI 2 to 7) percentage points more open defecation than the household question among males compared with females. Breaking the sample up by sex, the individual questions find 23 (95% CI 18 to 28) percentage points more open defecation than the household question among males, and 19 (95% CI 14 to 24) percentage points more among females. The household question underestimates open defecation by more among males compared with females.

dIsCussIOn
Our findings show that in our sample, individual-level, balanced questions find 20-21 (95% CIs 16 to 25 for both estimates) percentage points more open defecation than the household-level question. This is both a statistically significant and practically important difference. This study presents compelling evidence that India's DHS, which provides the most recent nationally representative estimates of open defecation for rural India, and other surveys that ask household-level questions, greatly underestimate open defecation among households with latrines.
We also found that the difference in reported open defecation between the two question types is significantly greater for households that received help to construct their latrines compared with households that did not. Among households that received help to construct their latrines, the individual-level questions find 28-29 (95% CIs 22 to 35 for both estimates) percentage points more open defecation than the household-level question. This suggests that as more and more households receive government assistance for a latrine through the SBM, household-level questions will become even less accurate at estimating open defecation. There are no differences that are significant at the .05 level, all are significant at the .01 level. That is why * is not used in the table. The larger difference in measured open defecation between the two question types among households that received help compared with those that did not is likely arising from higher rates of open defecation among households that received help to construct their latrines. There are several reasons that could explain why households receiving assistance may be less likely to use their latrines. First, these households are likely to have lower demand for latrine use, compared with households that built latrines on their own. Second, households that received help have latrines with pits that are on average 150 ft 3 smaller than the pits of latrines in households that did not receive help. Because of concerns over ritual purity, rural Indians are less likely to use latrines with pits that need to be emptied manually every few years, like the latrines that are promoted and constructed by the government. [13][14][15][16] Whether only one, both or other factors are leading to more open defecation among households that received help, the individual-level questions are better able to capture this open defecation than the household-level question.

Open access
We also find a statistically significant difference in reported open defecation between the two question types for males compared with females. The difference between the individual-level, balanced questions and the household-level question is 4 (95% CI 2 to 7) percentage points more for males compared with females. This supports evidence that, conditional on latrine ownership, males are more likely to defecate in the open compared with females. Individual-level questions understate the difference in open defecation between the two sexes by less than the household-level questions. Compared with the difference in reported open defecation by receiving help to construct the latrine, differences by sex are not as large.
Measuring open defecation at the individual level is feasible. Our survey team's experience suggests that adding the balanced, individual-level questions on use to a survey that already contains a household roster increases survey time by about 2 min, on average. Of course, the amount of time required to ask the individual-level questions depends on the number of individuals in the household.
The household-level question asked in the DHS also collects information on the types of latrines that households own, data that are still of great interest to researchers and practitioners. Therefore, individual questions on use, combined with a separate question on the types of latrines that households own, would satisfy both goals: evaluating latrine infrastructure, and measuring open defecation as accurately as possible.
A limitation of our study is that the samples from these project areas are not representative of the rural parts of the states they are in, nor are they collectively representative of rural India. The households in this study are much more likely to have a latrine than the average rural Indian household, and therefore, the individuals in this study are more likely to use a latrine. For this reason, the estimates presented here should not be considered as estimates of open defecation for any of the states, or the country as a whole. Rather, they show a large and significant difference in reported open defecation based on the type of question asked.
Measuring open defecation at the individual-level in a large household survey is doable and will provide a more accurate estimate of open defecation in rural India. Since reducing open defecation in India is important for meeting the SDGs, and since open defecation is an important factor contributing to poor health among children in India, it is important to measure its progress as accurately as possible. teams. We are also grateful for the research teams' cooperation on this study. In particular, we thank researchers from Oxford Policy Management, London School of Hygiene and Tropical Medicine, Indian Institute of Public Health Gandhinagar, Eawag, and Emory University, and the implementing agencies with which they are working, including World Vision, Coastal Salinity Prevention Cell, Wateraid, and Rural Welfare Institute, for sharing their census data with us, coordinating field work and supporting this endeavor. This study would not have been possible without their collaboration and assistance.
Contributors SV, NS, DS and DC contributed to the study design. SV, NS, DC and DS designed the survey instruments. NS, DM, SS and AT oversaw data collection. SV, NS and NG coordinated between the 3ie research teams and the survey team for this study. SV, RB, DC and DS contributed to analysis. All authors contributed to drafting the report.
Funding International Initiative for Impact Evaluation, Bill & Melinda Gates Foundation. RB, who is employed on the WSH programme of BMGF, contributed to the analysis of the results. NG contributed to coordinating data collection. Both contributed inputs to drafting the report.
Competing interests None declared.
Patient consent for publication Not required.