Pilot and feasibility studies: Is there a difference from each other and from a randomised controlled trial?

https://doi.org/10.1016/j.cct.2014.04.001Get rights and content

Abstract

Background

A crucial part in the development of any intervention is the preliminary work carried out prior to a large-scale definitive trial. However, the definitions of these terms are not clear cut and many authors redefine them. Because of this, the terms feasibility and pilot are often misused.

Aim

To provide an introduction to the topic area of pilot and feasibility trials and draw together the work of others in the area on defining what is a pilot or feasibility study.

Methods

This study used a review of definitions and advice from the published literature and from funders' websites. Examples are used to show evidence of good practice and poor practice.

Results

We found that researchers use different terms to describe the various stages of the research process. Some define the terms feasibility and pilot as being different whereas others argue that these terms are synonymous. All reflective papers agree that feasibility/pilot studies should not test treatment comparisons nor estimate feasible effect sizes. However, this is not universally observed in practice.

Summary

We believe that the term ‘feasibility’ should be used as an overarching term for preliminary studies and the term ‘pilot’ refers to a specific type of study which resembles the intended trial in aspects such as, having a control group and randomisation. However, studies labelled ‘pilot’ should have different aims and objectives to main trials and also should include an intention for future work. Researchers should not use the title ‘pilot’ for a trial which evaluates a treatment effect.

Introduction

During recent years, there has also been an increasing emphasis on the importance of preliminary work prior to the organisation of large-scale, publicly funded randomised controlled trials. Many large public funding bodies now expect substantial work to have been done prior to the main bid. Some funding streams, such as UK National Institute for Health Research (NIHR) Research for Patient Benefit (RfPB) [1] and the US NIH R34 funding mechanism [2], recognise this through the provision of substantial sums of money to support such work. The value of preliminary work is now recognised and researchers are encouraged to publish their pilot work in advance of their main trial, and some publishers are willing to publish such results. However, there remains much confusion about the purpose of preliminary work and also of terminology used. The NIHR use the terms ‘feasibility’ and ‘pilot’ to distinguish between different stages in the research process [3]. Although these terms are frequently used in the literature, they are used inconsistently and interchangeably [4], while other authors choose to use different terms completely to define the stages of development [5].

There is also the temptation to label a trial ‘pilot’ to excuse a small sample size, or one conducted in one locality, but still with the intention of running a study with treatment comparison as the main objective.

The aim of this paper is to provide an introduction to the topic area of pilot and feasibility trials. We will draw together the work of others that has been done in this area, describing current definitions, their overlaps and points of divergence. We use examples to illustrate good and poor practice and conclude with some recommendations on the use of the terms. This paper adds to our earlier work [4] by critiquing earlier definitions, and providing examples to support our criticism.

Within the pharmaceutical sector testing, drug efficacy has long had a tradition of clearly defined stages, from the initial phase 1 first-into-man studies through the phase 4 post-marketing studies. However, for large publicly funded trials, particularly of complex interventions and modes of care, the definitions and stages of trials have been less well defined/clear-cut. There have been several attempts to provide guidance on the definitions of a pilot and feasibility study. A review of papers published in 2001 in seven major journals looked at the objectives of pilot studies in the literature [6] to clarify the definition of pilot study. This was repeated in 2010, and the work extended to distinguish between pilot and feasibility studies in the article search and looking at the components of the studies [4]. The authors of these studies found that studies labelled ‘pilot’ generally used stricter methodology than studies labelled ‘feasibility’ and that pilot studies mostly reported their results as inconclusive and suggested further work, whereas feasibility studies did not state the same intention. They argue that the distinction between the two terms is not clear cut. However, they suggest the adoption of the NETSCC (NIHR Evaluation, Trials and Studies Coordinating Centre) definition which does distinguish between the two types of study [3].

The NETSCC [3] define feasibility studies as studies used to estimate important parameters that are needed to design the main study, e.g., standard deviation of the outcome measure, willingness of patients to be randomised, willingness of clinicians to recruit participants, number of people eligible, follow-up rates, response rates and adherence/compliance rates. Feasibility studies may have no plan for further work and their aim is to assess whether it is possible to perform a full-scale study.

The NETSCC [3] define a pilot study as a version of the main study run in miniature to determine whether the components of the main study can all work together. They suggest that a pilot should focus on the processes of running the main study, i.e., to ensure the mechanisms of recruitment, randomisation, treatment and follow-up assessments. The aim of the pilot is to provide training and experience in the running of the trial and to highlight any problems so they may be corrected before the main study begins. There must also be a plan for further work. A pilot study can be either external or internal to the main study.

This latter definition is comparable to the UK NICE definition of a pilot study as ‘a small-scale “test” of a particular approach … The aim would be to highlight any problems or areas of concern and amend it before the full-scale study begins [7].’

However, in contrast, Arnold et al. [5] provided three separate definitions for different types of pre-clinical work: pilot work, pilot studies and pilot trials. They defined pilot work as ‘any background research that informs a future study’; pilot studies as ‘studies with a specific hypothesis, objective and methodology’; and a pilot trial as ‘a stand-alone pilot study with a randomisation procedure’. Indeed the authors advocated against using the term feasibility study, arguing that it ‘does not reflect the scope of many pilot studies’. These definitions differ from most others in that they distinguish between the different possible objectives of pilot studies, but do not include the term feasibility whatsoever. The movement through development stages is defined by using the words; work, study and trial instead of the terms feasibility and pilot.

Thabane et al. [8], in their tutorial on pilot studies, do not distinguish between feasibility and pilot studies and simply note that the terms are used synonymously. They do however note that the main focus of a pilot study should be to test the feasibility of conducting a full study rather than statistical significance, and that many pilot studies fail to recognise this.

Leon et al. [9] state that a pilot study can be used to evaluate the feasibility of recruitment, randomization, retention, assessment procedures, new methods and implementation of the novel intervention. A pilot study is not a hypothesis testing study. Safety, efficacy and effectiveness are not evaluated in a pilot. Contrary to tradition, a pilot study does not provide a meaningful effect size estimate for planning subsequent studies due to the imprecision inherent in data from small samples. Thus, effect sizes provided by pilot studies should not be used to power a subsequent full trial. Instead clinical experience should be used to define a clinically meaningful effect. A pilot study is a requisite initial step in exploring a novel intervention or an innovative application of an intervention. Pilot results can inform feasibility and identify modifications needed in the design of a larger, ensuing hypothesis testing study.

This is similar to the British Medical Research Council's (MRC's) complex interventions guidelines, which urge the reader to exercise caution when using the results of a pilot study to make assumptions about the required sample size, likely response rates, etc., when the evaluation is scaled up [10]. These guidelines do not give an exact definition of a pilot or feasibility study; instead, they focus on the outcomes of the feasibility and piloting stage. Investigators should be confident that the intervention can be delivered as intended and be able to make safe assumptions about the effect sizes, variability, recruitment rates and retention to aid in the designing of the main study. They do note that ‘a pilot study need not be a “scale model” of the planned main stage evaluation, but should address the main uncertainties that have been identified in the development work’.

Krarup et al. [11] describe a trial, the ExSTroke Pilot trial, to examine the benefits of exercise in patients who have had a stroke. They intended to recruit 300 subjects, but this was powered on a postulated difference in treatment groups from a surrogate outcome, the Physical Activity Scale for the Elderly (PACE). The reason for the term ‘pilot’ in the title could be inferred because the study was not powered for recurrent stroke, MI, or mortality. The results were published [12] as a randomised controlled trial. The trial was criticised because it did not follow guidelines for the developing of complex interventions such as those of the MRC [10], and ‘we might have expected modelling of active ingredients of the intervention (given that it was a pilot study) and testing the feasibility of the approach’ [13].

In contrast, the LIFE study [14] is also described as a pilot study. The study intended to recruit 400 adults and the aims were as follows: (a) estimate the sample size needed for a full scale trial, (b) examine the consistency of the effects of the physical activity intervention on several continuous measures of physical function, (c) assess the feasibility of recruitment, (d) evaluate study adherence and retention, (e) evaluate the efficacy of a stepped care approach for managing inter-current illness in this at-risk population and (f) develop a comprehensive system for monitoring and ensuring participant safety. Two points can be made. First, the objectives of the study are consistent with the objectives of a pilot study, except (e) since it was not powered to evaluate efficacy. Second, the size of the projected pilot, at 400, exceeds the size of many full studies and is not justified in relation to the objectives. The outcomes of some of these objectives were subsequently published. For example, the investigators evaluated the longitudinal distributions of four standardised outcomes to contrast how they may serve as primary outcomes of future clinical trials: ability to walk 400 m, ability to walk 4 m in ≤ 10 s, a physical performance battery and a questionnaire focused on physical function. They concluded that the ability to walk 400 m as a dichotomous outcome provided the smallest sample size projections and that a 4-year trial based on the outcome of the 400-m walk is projected to require n = 962–2234 to detect an intervention effect of 30%–20% with 90% power [15]. In fact, they are now running the main study, a trial of 1600 people followed up for 2.7 years [16]. This outcome is entirely coherent with that of the pilot study. However, in view of the size of the pilot, they could not resist also doing some treatment comparisons [17], [18]. It is also of note that the size of the pilot was 25% of the main study, which leads one to query the correct ratio in size of the pilot and main study.

Cooper et al. [19] conducted the COSMOS pilot trial to investigate the use of computerised cognitive behavioural therapy (CCBT) for the treatment of depression in patients with multiple sclerosis. They recruited 24 patients based on the precision of estimates to be used to design the main trial [20]. The patients were randomised between CCBT and the usual treatment. The objectives of the study were to estimate the recruitment, withdrawal and dropout rate, sample size estimation and preferences for service delivery (home or elsewhere) and to test the feasibility of recruitment methods, questionnaires and the proposed outcome measures.

This is a good example of a pilot trial. The aims of the trial are consistent with the definition of a pilot trial, outlined in the previous section (to assess the trials processes and procedures, i.e., the questionnaires and the recruitment strategy, and to estimate values for the future trials sample size calculation, i.e., the standard deviation and the dropout rate). Although the effect size was calculated it is not assessed for statistical significance.

The trial was evaluated on the level of recruitment achieved. They had to approach nearly 600 patients to get the 24 in the trial, which resulted in a recruitment rate of 4.1% which was much lower than expected, and only 9/12 (75%) completed at least four out of the eight sessions. The authors concluded that a further trial was not feasible without a change in the eligibility criteria.

Section snippets

Discussion

It can be seen that there is still confusion around the terms. Some use the terms feasibility and pilot interchangeably [8] whereas others define the terms separately [3], [4], [6]. It is problematic to look to the literature to find a difference between pilot and feasibility study as a trial may be labelled as a pilot or feasibility study, but this does not mean that it is a pilot or feasibility study under someone else's definition.

From the review of the literature, we found that the

Conclusion

The distinction between pilot and feasibility studies is still a very grey area, with various definitions having been suggested by clinical trial methodology researchers. We suggest it is futile to ascribe a particular meaning to the term ‘feasibility’ and that all preliminary trial work could be described as ‘feasibility’; therefore, it could be thought of as a catch-all term for preliminary work. However, the term ‘pilot’ could be reserved for a study that mimics the definitive trial design

Acknowledgements

We thank members of the CONSORT team on Pilot and Feasibility trials for helpful discussion (Sandra Eldridge, Gill Lancaster, Christine Bond, Sally Hopewell and Lehana Thabane).

References (20)

There are more references available in the full text version of this article.

Cited by (0)

View full text