Follow-up by mail in clinical trials: does questionnaire length matter?

https://doi.org/10.1016/j.cct.2003.08.013Get rights and content

Abstract

In large clinical trials where outcome assessment is possible using questionnaires, it may be more cost-effective to mail them to patients than to conduct interviews in-person. However, nonresponse to mailed questionnaires reduces the effective sample size and can introduce bias. We conducted a systematic review and meta-analysis of randomized controlled trials evaluating the effect of questionnaire length on response rates. We searched 14 electronic bibliographic databases, the reference lists of relevant trials, and we contacted the authors of eligible trials to ask about unpublished data. For each trial identified, we used logistic regression to estimate the odds ratio for response per one page increase in the number of pages included in the questionnaire. We pooled the regression coefficients in a random effects meta-analysis. Heterogeneity among the coefficients was assessed using a chi-square test at a 5% significance level. We specified a priori that the reduction in the odds of response per one page increase would be greatest among trials comparing relatively short questionnaires. We used meta regression to examine the relationships between the regression coefficients, the length of the questionnaires used in each trial, and other study characteristics. A total of 38 randomized controlled trials were identified where participants were allocated to questionnaires of differing lengths and where the number of pages used was known. There was significant heterogeneity between the regression coefficients estimated from each trial. In meta regression, most of the heterogeneity was explained by variation in the length of the questionnaires used in each trial. Among trials in which the shortest questionnaire was a postcard, the odds of response were more than halved for each additional page used (0.39; 95% CI 0.34 to 0.45). In the remaining trials, pooled effect sizes were much smaller. In trials of one page compared with either two or three pages, the odds of response per one page increase was 1.01 (95% CI 0.82 to 1.24). For one page compared with four or more pages, and for two or more pages compared with longer alternatives, the odds ratios per one page increase were 0.90 (95% CI 0.83 to 0.98) and 0.98 (95% CI 0.96 to 0.99), respectively. There were no statistically significant associations between trial results and other study characteristics. It appears that response can be increased by using a shorter questionnaire. Moderate changes to the length of shorter questionnaires will be more effective than moderate changes to the length of longer questionnaires. If a choice of follow-up questionnaire exists for a clinical trial, the shorter one should be used. If a new follow-up questionnaire is to be designed, it should be made as short as possible without compromising the data collection requirements of the trial.

Introduction

To reliably detect modest but nevertheless plausible and clinically important treatment effects, clinical trials must enroll many thousands of patients [1]. In large trials where outcome assessment is possible using a questionnaire, it may be more cost-effective to mail them to patients than to conduct interviews in-person. However, nonresponse to mailed questionnaires reduces the effective sample size and can introduce bias [2].

The Medical Research Council (MRC) Corticosteroid Randomization After Significant Head Injury (CRASH) Trial aims to enroll and follow-up 20,000 head-injured patients over 5 years [3]. Long-term outcome is assessed using the Glasgow Outcome Scale (GOS), the most commonly used measure in head injury trials [4]. The GOS is an ordinal outcome scale with five levels: good recovery, moderate disability, severe disability, vegetative state and dead. An extended version of the GOS has also been developed that includes eight levels [5]. During the pilot phase of the CRASH Trial, questionnaires were developed for the GOS and the extended GOS that could be self-administered or completed by a caregiver [6]. Both versions were shown to be reliable in test–retest studies and in comparison with telephone administration by trained nurses. The questionnaire for the GOS comprises seven questions on a single page, and the questionnaire for the extended GOS comprises 15 questions over three pages. Having the choice of two questionnaires raised an important question—when mailed, which questionnaire will provide the highest response rate, thereby maintaining sample size and reducing bias?

To inform the choice and design of follow-up questionnaires for large clinical trials, we conducted a systematic review and meta-analysis of randomized controlled trials evaluating the effect of questionnaire length on response rate, with the aim of describing this relationship. We hypothesized that the propensity to respond to a questionnaire is inversely related to its length. The focus of this paper is on the results of the systematic review. They may be relevant to other research in which outcomes can be assessed using mailed questionnaires.

Section snippets

Systematic review

To examine the effect of questionnaire length on response rate, we conducted a systematic review and meta-analysis of randomized controlled trials of shorter versus longer questionnaires. A systematic search was initially made for all randomized controlled trials of any method to influence response to a postal (mailed) questionnaire [7]. There was no restriction by language, questionnaire topic or study population. We searched 14 electronic bibliographic databases (Table 1), the reference lists

Results

The systematic search for randomized controlled trials of methods to influence response to mailed questionnaires yielded 26,937 records of potentially relevant reports. After screening records and obtaining copies of the reports considered to be relevant for further inspection, a total of 251 reports were found to contain one or more such randomized controlled trials [7].

Of the 251 reports, 27 were about trials evaluating the effect of questionnaire length on response rates. Eight reports

Discussion

This systematic review and meta-analysis of randomized controlled trials evaluating the effect of questionnaire length on response supports the hypothesis that response is inversely related to length. It also suggests that the effect on response is much greater for very short questionnaires. Before we consider the implications of these results on the choice and design of follow-up questionnaires in clinical trials, several methodological issues with a bearing on the validity of the results must

Conclusion

Response can be increased by using a shorter questionnaire. Moderate changes to the length of shorter questionnaires will be more effective than moderate changes to the length of longer questionnaires. If a choice of follow-up questionnaire exists for a clinical trial, the shorter one should be used. If a new follow-up questionnaire is to be designed, it should be made as short as possible without compromising the data collection requirements of the trial. Within health research, further trials

Acknowledgments

We are grateful to Paul Dorman, Eliv Lund and Matthew Murawski for providing additional information about their studies. We would also like to thank the referees and Graham Try whose comments helped to improve this paper.

References (16)

  • R. DerSimonian et al.

    Meta-analysis in clinical trials

    Control. Clin. Trials

    (1986)
  • S. Yusuf et al.

    Why do we need some large, simple randomized trials?

    Stat. Med.

    (1984)
  • B.K. Armstrong et al.

    Principles of exposure measurement in epidemiology

  • D. Yates et al.

    Corticosteroids in head injury—the CRASH trial

    J. Accid. Emerg. Med.

    (1999)
  • K. Dickinson et al.

    Size and quality of randomised controlled trials in head injury: review of published studies

    BMJ

    (2000)
  • J.T. Wilson et al.

    Structured interviews for the Glasgow Outcome Scale and the extended Glasgow Outcome Scale: guidelines for their use

    J. Neurotrauma.

    (1998)
  • J.T. Wilson et al.

    Reliability of postal questionnaires for the Glasgow Outcome Scale

    J. Neurotrauma.

    (2002 (Sep))
  • P. Edwards et al.

    Increasing response rates to postal questionnaires: systematic review

    BMJ

    (2002)
There are more references available in the full text version of this article.

Cited by (138)

View all citing articles on Scopus
View full text