Objective To examine the effects of stamped reply envelope and the timing of newsletter distribution.
Design A randomised controlled trial in a prospective cohort study with a 2×2 factorial design of two interventions.
Setting The Japan Nurses’ Health Study (JNHS), a prospective cohort study for women's health.
Participants The present study included 6938 women who were part of the first-year entry cohort for the fifth wave of the biannual follow-up survey of the JNHS.
Intervention The participants were randomly allocated into four groups; Group-1 (business-reply, newsletter with initial mailing), Group-2 (business-reply, newsletter with reminder), Group-3 (stamped envelopes, newsletter with initial mailing) and Group-4 (stamped envelopes, newsletter with reminder). The thank-you and reminder letters were mailed out at the end of the sixth week. This study was censored at the end of 12 weeks.
Main outcome measures Main outcome measures were cumulative response at the end of 6 and 12 weeks after mailing out the questionnaire.
Results The cumulative response at 12 weeks were 58.3% for Group-1, 54.1% for Group-2, 60.5% for Group-3 and 56.7% for Group-4 (p=0.001). The odds of the response was higher for stamped envelopes than for business-reply envelopes (OR (95% CI)=1.10(1.00 to 1.21)). The odds was higher for newsletter delivery with initial mailing than for with reminder (1.18(1.07 to 1.29)). The response in first 6 weeks for stamped envelope was significantly higher than for business-reply envelope (p=0.047). Although the response in 6 weeks for women received the newsletter with initial mailing was lower than for women who did not, the proportions did not differ significantly (p=0.291).
Conclusions The style of return envelope affected response rates of mail survey. The results of this study suggest that practices of provision of the additional information, should be handled individually in advance, as a separate event from sending follow-up questionnaire or reminder letters.
- PUBLIC HEALTH
- STATISTICS & RESEARCH METHODS
- HEALTH ECONOMICS
- SERVICES ADMINISTRATION & MANAGEMENT
This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/2.0/ and http://creativecommons.org/licenses/by-nc/2.0/legalcode.
Statistics from Altmetric.com
- PUBLIC HEALTH
- STATISTICS & RESEARCH METHODS
- HEALTH ECONOMICS
- SERVICES ADMINISTRATION & MANAGEMENT
In the present study, drawing on data from the fifth wave of the Japan Nurses’ Health Study (JNHS), we examined whether and how stamped reply envelopes compared to business reply envelopes and the timing of newsletter delivery affected the odds of a response.
This study randomly allocated subjects to four different survey methods within the established study population with a 2×2 factorial design, by utilising the JNHS, the first nationwide women cohort sturdy in Japan.
Enclosing stamped return envelopes was effective to increase the response rates.
The results of this study suggested that practices such as provision of additional information should be handled individually in advance, as separate events, for maintaining the cohort.
Strength and Limitation
The cohort of the JNHS follow-up survey consists of female healthcare professionals with a nursing license who agreed to participate in the survey by signing an informed consent form. It may be problematic to apply the results of the present analysis to a broader population.
A major advantage of the present analysis is that the JNHS is a nationwide occupational cohort study in Japan, and drawing on data from the cohort study, we could randomly allocate items of research interest, that is, type of return envelopes and timing of newsletter delivery, within the survey population.
The Japan Nurses’ Health Study (JNHS) is a nationwide prospective occupational cohort study to explore women's health in Japan.1 The JNHS was designed as a prospective study, which consists of a cross-sectional baseline survey that includes a 6-year entry period from 2001 to 2007, and a 10-year follow-up study, from 2001 to 2017.1 For a prospective cohort study, maintaining the cohort, that is, maintaining high follow-up response rates is a major issue. A cohort study is used to estimate risks, rates or occurrence times of events, and thus requires that the whole cohort remain under observation for the entire follow-up period.2 Loss of subjects during the study period lowers the validity of the study, because it makes estimation more difficult due to unknown outcomes of lost subjects. Prospective cohort studies that take many years are likely to experience difficulties with locating people over the study period.2 Follow-up studies that maintain less than about 60% of subjects are considered insufficient to provide confident estimates.2
In an effort to achieve high response rates, offering incentives to respondents has become prevalent. A systematic review of 292 surveys showed that monetary and non-monetary incentives improved the odds of returning the questionnaire.3 Also, prior studies reported that the odds of receiving responses were increased when postoffice stamped reply envelopes were used compared with enclosing prepaid business-reply envelopes,3 although the results were mixed.4 Furthermore, sending advance letters have been shown to increase response rates, as well as providing follow-up contacts such reminder letters, telephone contacts and providing non-respondents with a second copy of the questionnaire.3 ,5–7 In addition, a study has reported that sending a cover letter that asks recipients to decline participation within 7 days if they do not want to participate raises response rates.8 Yet, using information leaflets upon recruitment did not affect the number of participants in the survey.9
As far as the JNHS is concerned, follow-up questionnaires are mailed to the cohort along with a newsletter. The newsletters are designed to update participants on new information about women's health and the progress of the JNHS. Women who do not respond to the first mailed questionnaire receive a second mailing within 6 months. Subsequently, women who still do not respond receive a third and fourth questionnaire. If the JNHS coordination centre cannot contact participants by mail, the JNHS Follow-up Committee confirms if the subject has moved, and a questionnaire is sent to the new address, which is obtained from the resident registry of the corresponding local district.
In the present study, drawing on data from the fifth wave of the JNHS, we examined whether and how stamped reply envelopes compared with business-reply envelopes and the timing of newsletter delivery affected the odds of a response. In Japan, studies that receive public funds are not allowed to offer incentives or stamped reply envelopes to survey participants as participants may not respond, and such a practice is regarded as a waste of research expenses for a study supported by the national government. However, for the JNHS it is expected that the use of stamped envelopes and delivery of a newsletter will have favourable effects on response rates. It is hoped that because the present study population draws from a homogeneous cohort consisting of healthcare professionals, information regarding women's health in general and results of previous JNHS surveys would encourage participant involvement in the study.
The primary objective of the study was to examine whether stamped reply envelopes and enclosed newsletter improve response rates of mail survey by a 2×2 factorial randomised controlled trial in the JNHS cohort. The secondary objective was to explore the demographic and lifestyle factors that affect response rate of mail survey in a women cohort.
The JNHS consists of a cross-sectional baseline survey that includes a 6-year entry period, from 2001 to 2007 and a 10-year follow-up study, from 2001 to 2017. The study population was designed for female registered nurse, licenced practical nurses, public health nurses and/or midwives, who were at least 30 years of age and resident in Japan. Although the participants were licenced to practice nursing, they did not necessarily function as nurses. The baseline survey includes 49 927 responses from participants in Japan. Among them, 14 932 women signed an informed consent form and participated in the follow-up survey. Institutional review boards of Gunma University and the National Institute of Public Health reviewed and approved the JNHS study protocol. The study design of the JNHS has been presented elsewhere.1The present study included 6938 women who were in the first-year entry cohort for the fifth-wave follow-up survey of the JNHS.
To estimate the effect of types of return envelope and timing of newsletters, women were randomly allocated into the four groups with a 2×2 factorial design. For Group-1 and Group-2, business-reply return envelopes for postpayment by the recipient were enclosed, and for Group-3 and Group-4, stamped return envelopes were provided. In terms of timing of newsletter delivery, for Group-1 and Group-3, the newsletter was enclosed when the questionnaires were mailed out and for Group-2 and Group-4, the newsletters were sent with the reminder letters. The questionnaires were mailed to participants on 22 December 2009, and a thank-you and reminder letter was mailed out to all respondents (regardless of whether they had already returned their self-administered questionnaires to the data centre) at the end of the sixth week (2 February 2010). The present study was censored at 16 May 2010 (12 weeks or 84 days). Sample size was determined by the size of the available cohort. With an expected number of 3469 (ie, 6938/2) per group, and a reference response rate of 60%, for 80% power and 5% significance, the detectable difference in response rate was ±3.3%.
When the participants registered at baseline survey, the sequential unique seven-digit ID numbers were assigned randomly by the JNHS data centre. According to the ID numbers, participants were allocated to the four groups. The allocated group number for each participant was the remainder when the ID number was divided by four.
Primary outcome measure was cumulative response proportion at 12 weeks after mailing out the questionnaire. Secondary outcome measure was cumulative response at the end of 6 weeks after initial mailing, just before delivering the reminder letters.
The participants were simultaneously randomised to two interventions; type of return envelope (business reply vs stamped return envelopes) and timing of newsletter delivery (newsletter with initial mailing vs newsletter with reminder). Besides these two variables, following demographic and lifestyle variables were used to explore the factors affecting response rates: age at the survey, type of nursing licence (registered nurse, licenced nurse, midwives and public health nurse), region of residence (Hokkaiko, Tohoku, Kanto, Hokuriku_Koshin, Tokai, Kinki, Cyugoku, Shikoku, Kyusyu and Okinawa) and type of residence area (urban (Tokyo metropolitan area and other 19 large cities designated by government ordinance) and not-urban area), work status (not-working and working), smoking status (smoking and not-smoking), alcohol drinking (<3 days a week and ≥3 days a week), pregnancy (pregnant and not-pregnant) and menopausal status (postmenopausal and others). All the data were obtained from the available latest wave of survey. These variables included factors previously studied10–12 and reproductive health-related issues in women.
Characteristics of participants were compared between groups using analysis of variance and χ2 test to check the relevance of randomisation process. Before examining main effect of two interventions, type of return envelope (business reply vs stamped) and timing of newsletter provision (with initial mailing vs with reminder) on cumulative response proportion at 6 and 12 weeks after initial mailing, the interaction of these two interventions was tested by logistic regression model. The main effects of the interventions were tested by χ2 test. In order to examine the factors affecting the responses in 12 weeks after initial mailing, logistic regression models were used to estimate unadjusted and age-adjusted ORs and their 95% CIs. For all statistical analysis, SAS V.9.1 (SAS Institute, North Carolina, USA) was used and p<0.05 was set as statistically significant.
Characteristics of participants
Of the 6938 women of the first-year entry JNHS cohort, 1727, 1728, 1687 and 1796 women were randomised into Group-1, Group-2, Group-3 and Group-4, respectively (figure 1). With the questionnaire, 3455 received business-reply return envelopes and 3483 women were provided with stamped envelopes. A total of 3414 women received the newsletter with initial mailing and 3524 women received the newsletter with thank-you and reminder mailing. The four groups did not differ significantly in demographic and lifestyle characteristics (table 1).
Cumulative response proportion by group
The cumulative response proportions at 12 weeks were 58.3% for women in Group-1, 54.1% for Group-2, 60.5% for Group-3 and 56.7% for Group-4 (figure 1), and these proportions significantly differed among the groups (χ2 = 15.5; df=3; p=0.001). There was not statistically significant interaction effect of two interventions, type of enclosed return envelope and timing of newsletter delivery, on the proportions (p=0.881). The response for women who received stamped reply envelopes (Group-3 and Group-4) was 58.6%, and it was significantly higher compared with the proportion of 56.2% for those who received business-reply envelopes (Group-1 and Group-2) (χ2 = 4.15; d.f.=1; p=0.042). With respect to the effect of newsletter, the cumulative response proportion at 12 week when the newsletter was delivered at initial mailing was significantly higher than the response when the newsletter was delivered with thank-you and reminder letters (χ2 = 11.1; d.f.=1; p<0.001); 59.4% for Group-1 and Group-3 and 55.4% for Group-2 and Group-4.
We compared the cumulative proportions at 6 weeks to confirm the main effects of interventions without the effect of reminder mailing (figure 2). The proportion at 6 weeks for business-reply envelopes (Groups-1 and 2) was 40.3% and the proportion for stamped reply envelopes (Group-3 and Group-4) was 42.7%, and those proportions differed significantly (χ2 = 3.93; d.f.=1; p=0.047). The proportion for women received the newsletter with initial mailing (Group-1 and Group-3) was 40.9% and the proportion for women who did not receive it with initial mailing (Group-2 and Group-4) was 42.1%, and these proportions did not differ significantly (χ2 = 1.11; d.f.=1; p=0.291).
Factors affecting response
The ORs and 95% CIs for cumulative responses at 12 weeks are shown in table 1. With respect to two interventions, unadjusted ORs showed statistically significant effects. The stamped envelopes raised the response by 10% relative to provision of business-reply envelopes (OR (95% CI)=1.10 (1.00 to 1.21)), and the newsletter delivery with initial mailing raised the response by 18% relative to the delivery with reminder letters (1.18 (1.07 to 1.29)). However, when adjusted by age at the survey, the effect of stamped return envelopes became non-significant (table 2).
Regarding other factors that showed significant effects on the response by age-adjusted analyses, nursing licence, region of residence, type of residence area, smoking, menopause and participation in previous survey were associated with the odds of response.
Timing of newsletter delivery and type of return envelope
Results of the present study showed that provision of the newsletters with the questionnaires tended to decrease the odds of returning the self-administered questionnaire. In addition, if the newsletters were provided to participants 6 weeks later with reminder letters, it would further keep participants (non-responders at that point) from returning their questionnaires. Thus, the results suggest that each practice, such as provision of information, request for collaboration and encouragement of contribution, should be managed individually, as separate events in advance. As prior studies3 ,10 documented that advanced contacts via letters, cards and phone calls increase the rates, if the update of the survey is offered via newsletter to respondents in advance, it may facilitate their understanding of the research issues and then improve response rates as well as enhance the quality of responses by reducing the number of items left blank or incomplete and decreasing inconsistent answers.
In the light of cost-performance, some studies suggest that allocating large sums of money to achieve high response rates may not always significantly improve the quality of the sample.13–15 Prior studies documented that with properly high response rates (approximately 70%), the bias due to non-response was unlikely to affect estimation of the survey.13–15 However, for a follow-up cohort survey like the JNHS, maintaining high follow-up response rates is crucial to maintaining the cohort.
Provision of a stamped return envelope had a significant effect of raising the odds of returning the questionnaire in the JNHS follow-up survey. Although there is an argument that providing stamped return envelopes is an inappropriate use of research expenses (especially for research with public funds) and is not cost-beneficial, it depends on the survey response rates that you expect to achieve. We should discuss about the cost-performance based on the results of an actual cost analysis.16 If we assume provision of stamped return envelopes, compared with business-reply envelopes for postpayment by the recipient, increases the response rate by 10%, in a case survey of 10 000 participants, when the response rate with business-reply envelope is 50%, assuming that the stamped return envelope approach can improve the odds of response by 10%, the business-reply approach is better in terms of cost-performance (table 3). However, if the response rate is 80% in a survey of 10 000 participants, mailing costs for the stamped return envelope approach and for business reply envelope approach will be 285 yen/response and 273 yen/response, respectively. Consequently, if the response rate is as high as 80%, the stamped return envelope approach will be more advantageous than the business-reply envelope approach in terms of cost-performance. In that case, providing stamped return envelopes is the best way for the JNHS to maintain the cohort with a better cost-performance.
Other factors affecting the response
In addition to the effect of the type of return envelope and timing of the newsletter, the present analysis showed interesting points with regard to factors predicting the response to the survey. Participation in a previous survey increased the odds of responding to this survey. In particular, participants who were involved in the most recent survey were 18 times more likely to return the questionnaire (table 2). In addition, there appeared to be some differences in women's responses to the survey based on their residence regions. As shown in table 2, women living in Hokkaido were more likely than those living in Kanto to respond to the survey. In contrast, women living in Kyusyu and Okinawa were less likely to respond to the survey.
Women who experienced menopause were more likely to participate in the present survey, even after the odds were adjusted by age. The questionnaire of the JNHS included several items with respect to reproductive health-related issuers, such as pregnancy and menopause. Recognising the association of the research issues with women's personal experiences would promote their involvement in the study. In contrast, smokers in the previous survey were less likely to respond to the present questionnaire. Given a recent negative image of smoking and the public trends against smoking, smokers would be reluctant to answer questions with regard to their health. However, if this tendency becomes prominent, health effects of smoking could be underestimated, especially in later surveys.
There are several limitations to the present study. A major one refers to generalisation of results. The cohort of the JNHS follow-up survey consists of female healthcare professionals with a nursing licence who agreed to participate in the survey by signing an informed consent form. It may be problematic to apply the results of the present analysis to a broader population. There is, however, a major advantage of the present analysis. The JNHS is a nationwide occupational cohort study in Japan, and drawing on data from the cohort study, we could randomly allocate items of research interest, that is, type of return envelopes and timing of newsletter delivery, within the survey population.
We are grateful to the Japanese Nursing Association (47 prefectures’ individual Nursing Associations) and Japan Society of Menopause and Women's Health for their collaboration in designing and conducting the Japan Nurses’ Health Study. We also thank the Independent Study Monitoring Committee, which periodically monitors the study process. The JNHS is supported partly by a Grant-in-Aid for Scientific Research (B:22390128) from the Japan Society for the Promotion of Science.
Contributors KH was responsible for developing of study concept and design, and acquisition of data. CW, KN and KH performed the statistical analyses. CW interpreted the results of the data analyses and drafted the manuscript. All authors have been involved in revising and elaborating it critically in the intellectual context.
Funding The present study is funded by the National Center for Child Health and Development (NCCHD), which is fully sponsored by the Japanese Ministry of Health, Labor, and Welfare.
Competing interests None.
Ethics approval Institutional review boards of Gunma University and the National Institute of Public Health.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement There are no additional data sharing to other parties.