Examining non-response bias in substance use research—Are late respondents proxies for non-respondents?

Abstract

Background

Non-response is a major concern among substance use epidemiologists. When differences exist between respondents and non-respondents, survey estimates may be biased. Therefore, researchers have developed time-consuming strategies to convert non-respondents to respondents. The present study examines whether late respondents (converted former non-participants) differ from early respondents, non-consenters or silent refusers (consent givers but non-participants) in a cohort study, and whether non-response bias can be reduced by converting former non-respondents.

Methods

6099 French- and 5720 German-speaking Swiss 20-year-old males (more than 94% of the source population) completed a short questionnaire on substance use outcomes and socio-demographics, independent of any further participation in a cohort study. Early respondents were those participating in the cohort study after standard recruitment procedures. Late respondents were non-respondents that were converted through individual encouraging telephone contact. Early respondents, non-consenters and silent refusers were compared to late respondents using logistic regressions. Relative non-response biases for early respondents only, for respondents only (early and late) and for consenters (respondents and silent refusers) were also computed.

Results

Late respondents showed generally higher patterns of substance use than did early respondents, but lower patterns than did non-consenters and silent refusers. Converting initial non-respondents to respondents reduced the non-response bias, which might be further reduced if silent refusers were converted to respondents.

Conclusion

Efforts to convert refusers are effective in reducing non-response bias. However, converted late respondents cannot be seen as proxies of non-respondents, and are at best only indicative of existing response bias due to persistent non-respondents.

Introduction

Non-response is a serious problem in epidemiological and substance use studies. When response rates are low, survey validities are often questioned due to the risk of non-response bias, occurring when survey estimates based on respondent outcomes differ from those of the total sample that included non-respondents (Lahaut et al., 2003). The magnitude of non-response bias is defined as a function of the non-response rate and the difference between respondents and non-respondents (Biemer and Lyberg, 2003). Because data on non-respondents are usually not available, standard survey methodologies commonly recommend attaining higher response rates in order to prevent the risk of non-response bias (Alreck and Settle, 1995, Babbie, 2007, Singleton and Straits, 2005).

During the last few decades, participation rates in survey research have sharply declined (Bradburn, 1992, Galea and Tracy, 2007, Steeh, 1981, Tolonen et al., 2006, Tourangeau, 2004). In order to realize acceptable response rates and minimize the risk of non-response bias, researchers have developed time-consuming and expensive strategies in attempts to convert reluctant or hesitating participants to (late) respondents. These can be incentives, reminders, or encouraging telephone calls, etc. The use of these strategies presupposes that late respondents resemble non-respondents more than initial respondents do. Therefore, increasing response rates by converting reluctant or hesitating participants to respondents should reduce the magnitude of response bias, because the pool of respondents becomes more representative of the total sample.

However, the assumption above has been challenged by other researchers, who suggest that in circumstances where the cause of non-response is related to topic sensitivity or saliency, the relationship between response rate and non-response bias is not clear-cut (Groves, 2006, Groves and Peytcheva, 2008). Sensitive questions are seen as intrusive, embarrassing or threatening to disclosure because they touch on topics that are socially undesirable and are thought to negatively affect non-response rates (Tourangeau and Yan, 2007). Unlike topic sensitivity, topic saliency reflects survey characteristics that are perceived by some individuals to be interesting and thus motivate participation (Groves and Peytcheva, 2008, Groves et al., 2004, Heberlein and Baumgartner, 1978). Thus, when a low response rate is due to topic sensitivity or lack of saliency, increasing response rate by the conversion of initial non-respondents to late respondents may not reduce the non-response bias. This is because late respondents (often converted only after much effort) are mostly those of the initial non-respondents that saw the topic as sufficiently salient and not too sensitive, whereas the remaining holdouts do not participate because they perceive the topic sensitivity and saliency in the opposite. Consequently, early and late respondents are more or less similar, but both differ substantially from the remaining non-respondents. Thus, survey estimates based on both early and late respondents do not change with increasing response rates, but differences between non-respondents and early or late respondents do increase.

Substance use is often considered a sensitive topic (Bradburn and Sudman, 1979, Singer, 1978). Numerous studies have examined non-response bias in substance use surveys, but have shown no clear patterns of results. Compared to respondents in some research, non-respondents reported higher use of alcohol (Goldberg et al., 2006, Heath et al., 2001, McCoy et al., 2009, Torvik et al., 2012, Wild et al., 2001, Zhao et al., 2009), cigarettes (Boström et al., 1993, Cunradi et al., 2005, Goldberg et al., 2006, Hill et al., 1997, Korkeila et al., 2001, McCoy et al., 2009, Smith and Nutbeam, 1990, Torvik et al., 2012, Woodruff et al., 2000) or cannabis (Zhao et al., 2009). In contrast, some studies showed no significant differences between non-respondents and respondents on alcohol use (Cunradi et al., 2005, Gmel, 2000, Korkeila et al., 2001, Kypri et al., 2004, Strote et al., 2002, Trinkoff and Storr, 1997, Ullman and Newcomb, 1998), tobacco use (Strote et al., 2002, Ullman and Newcomb, 1998) or cannabis use (Ullman and Newcomb, 1998) outcomes. Regarding alcohol use, some studies have even found that non-respondents more often were abstainers (Lahaut et al., 2002, Lahaut et al., 2003, Torvik et al., 2012) and drank less than respondents (Cranford et al., 2008, Hill et al., 1997).

Inconsistencies across studies raise the issue of substance use among non-respondents. There are usually no data available to compare non-respondents with respondents, since they did not complete any questionnaires. In existing studies of non-response bias in substance use outcomes, researchers have employed three distinct methods as proxy measures for non-respondents: (a) using very short questionnaires to follow-up non-respondents and comparing their answers with that of regular respondents (Boström et al., 1993, Cranford et al., 2008, Hill et al., 1997, Smith and Nutbeam, 1990, Strote et al., 2002); (b) comparing early and late survey respondents (Korkeila et al., 2001, Kypri et al., 2004, Lahaut et al., 2002, Lahaut et al., 2003, Trinkoff and Storr, 1997, Ullman and Newcomb, 1998, Zhao et al., 2009); and (c) comparing baseline characteristics of respondents to dropouts at subsequent follow-ups in longitudinal designs (Cunradi et al., 2005, Gmel, 2000, Goldberg et al., 2006, Heath et al., 2001, McCoy et al., 2009, Torvik et al., 2012, Wild et al., 2001).

Each of these methods has limitations. The first two approaches use late respondents as proxies for non-respondents because they assume that they would have been non-respondents had the data collection stopped earlier, or had the questionnaire not been shortened. The model underlying these approaches has been called the “continuum of resistance” (Lin and Schaeffer, 1995) and pre-supposes that all non-participants are similar to each other and that if they eventually participated in the survey, their responses would mimic those who are the most difficult to engage. These methods consistently fail to obtain data on the most reluctant non-respondents who would very rarely participate in the survey. Studies comparing baseline characteristics of respondents to dropouts in follow-up may partly overcome this limitation. Nevertheless, initial non-respondents are missed at baseline, and dropping out at follow-up could be related to a condition that was not present at baseline (e.g., increased substance use). The present study aims at overcoming these handicaps. Over the course of one year, a near-census of young army conscripts was asked to participate in a cohort study. At the enrolment phase, a short questionnaire on substance use was distributed among them who were early (response without extra effort), late respondents (response after increased efforts, i.e., encouraging telephone calls), silent refusers (giving consent, but not participating), or non-consenters (not participating and not consenting) to the cohort study. The overall response rate for the short questionnaire was 94%. This approach allows an analysis of whether late respondents differ significantly from early respondents or silent refusers or non-consenters on the substance use outcomes they reported in the short questionnaire.

In this study two competing hypothesis were tested: (a) late respondents are more similar to non-respondents than to early respondents, thus efforts undertaken to increase response rate can decrease the non-response bias; (b) late respondents are more similar to early respondents than to non-respondents because only those non-respondents who perceive the topic as being least sensitive or most salient can be converted into late respondents after special efforts to increase the response rate; strategies to increase response rate would have no effect on non-response bias.