Working Definitions

We understand loss to follow-up as the main reason for MOD, i.e. missing or incomplete ascertainment of the primary outcome for participants in an RCT. We did not take ambiguous terms like ‘drop-out,’ ‘withdrawal,’ and ‘not completing the trial’ (outcome ascertainment still possible) to necessarily mean MOD unless the authors of methodology articles made it clear they referred to loss to follow-up or MOD. Vice versa, we use the term “complete follow-up” to indicate no missing outcome data.

We considered as relevant methodology articles those that assessed how ITT was defined in other trial publications, discussed how ITT should be defined or reported, or examined methods that may be employed to address the issue of MOD when conducting an ITT analysis. These could include journal articles, reviews, editorials, or letters to the editor.

The concept of ITT has two main facets: (1) how to deal with participants for whom outcome data are available, and (2) how to deal with participants for whom outcome data are not available. If all participants had outcome data, all of the three ITT definitions (see below) on our data extraction form would lead to identical terminology and identical practice: analysis in the randomization arm for all participants for whom investigators have recorded the outcome of interest, regardless of protocol deviations and participant compliance. In case of MOD under ITT, we identified three mutually exclusive characterizations of ITT (i.e. the answer to the question “what is an ITT analysis”) and refer to these characterizations as definitions:

Eligibility Criteria

We included publications that devoted at least one paragraph to ITT and two other paragraphs to either ITT or MOD. Eligible articles mentioned the terms “intention to treat” or “intent to treat” in the title or abstract and were published in peer-reviewed journals either as articles, editorials, or letters to the editor. Because we were interested in capturing methodological discussions and fundamental concepts of ITT and not a description of how ITT was applied, we excluded original reports of RCTs, observational studies, and clinical systematic reviews. We excluded articles written in languages other than English.

Search Strategy and Article Selection

A research librarian (N.B.) trained in health research methodology developed an initial pilot search strategy for MEDLINE. The librarian and an investigator (M.A.) subsequently used relevant articles identified through a pilot search to refine the search strategy. Appendix S2 presents the detailed search strategy for MEDLINE using the Ovid platform to identify methodological papers from 1950 to February 2009.

Working in teams of two, investigators (E.A., M.A., C.T. and L.R.) independently screened titles and abstracts in duplicate. When one investigator deemed an abstract as potentially eligible, the full text document was retrieved. Next, pairs of investigators independently assessed eligibility of full text publications based on eligibility criteria. Disagreements were resolved by discussion and consensus or if necessary by third party arbitration (G.G.).

Data Extraction

Investigators (M.A., T.H., S.G., L.R., and C.T.) working in teams of two independently extracted the different descriptions of ITT in relation to MOD from each eligible article using standardized, pre-piloted forms (Appendix S3). Data extractors chose from the following options to describe an author’s stance for each of the three definitions of ITT in relation to MOD above: ‘sole definition,’ ‘definition desirable,’ ‘definition possible but undesirable,’ ‘definition mentioned but preference unclear,’ ‘definition specifically excluded,’ and ‘definition not mentioned’ (see Table 4 in the thesis of M.A. (Appendix S1) for details about each of these categories). Our categorization system allowed us to capture instances in which an author held multiple definitions as ‘desirable’, but did not comment on a preference.

To accommodate new insights from articles we modified the data extraction form throughout the study to best reflect the data. [9] Three investigators adjudicated discrepancies between data extractors (G.G., T.H., and M.A.).

Inter-rater Reliability

Agreement was calculated at the title and abstract screening and at the full text screening stage, as well as for categorical/dichotomous variables during data extraction. We used Kappa (κ) to determine the degree of agreement between pairs of reviewers and interpreted it according to Landis and Koch (κ values of 0 to 0.20 represent slight agreement; 0.21 to 0.40, fair agreement; 0.41 to 0.60, moderate agreement; 0.61 to 0.80, substantial agreement; and greater than 0.80 values represent almost perfect agreement). [10].

Summary of Findings

This systematic review of methodology articles found great variation in the proposed definitions of ITT in relation to MOD and in how MOD should be handled in RCTs under the ITT principle. Given the high proportion of RCTs with MOD for the primary outcome (60–75%) and the increasing use of the term ITT in RCT reports (up to 80%), this constitutes a serious problem. [5], [6], [14].

Although others have shown that the concept of ITT is inconsistently applied in clinical trials, [5], [6] the present review is the first to demonstrate that this variability in the definition of ITT extends to methodological articles. Not only did articles provide differing definitions, but 17 out of 36 articles (47%) that addressed the issue of MOD endorsed more than one definition of ITT.

Limitations and Strengths

A potential limitation of our systematic review is that we focused our search exclusively on MEDLINE, articles in English, and articles that mentioned the terms “intention to treat” or “intent to treat” in the title or abstract potentially leading to less representative findings. Searching multiple databases and broadening our eligibility criteria might have increased the generalizability of our results. However, we would likely have found even more variability in definitions of ITT, suggesting that our findings in this respect are if anything conservative. Strengths of our review include our success in capturing a variety of journal types (methods, statistics, general medicine, and clinical specialty). We modified the data extraction form throughout the study to best reflect the data and to accommodate new insights from articles as they were acquired. [9] Furthermore, all articles were screened and abstracted in duplicate and experienced research methodologists adjudicated discrepancies.

We sought a criterion for a substantive discussion of the concepts of ITT and its relation to MOD and chose a criterion of three paragraphs. The criterion we chose is arbitrary, and one could argue that a criterion based on word rather than paragraph count would have been more appropriate. We found, however, that eligible articles did provide substantive discussion, and we were able to classify these articles according to their suggestions regarding ITT and MOD. Whether a less stringent criterion could have achieved the same goal with more eligible articles remains open to question.

Suggested Strategies for Dealing with MOD in ITT

Of those who considered ITT and MOD to be related issues, the authors endorsed either imputing data and/or requiring complete follow-up. Of 16 articles that mentioned it, 13 (81%) excluded complete case analysis as a method of dealing with MOD under ITT; the others expressed the view that complete case analysis was a possibility under ITT, although it was an undesirable strategy. The position that MOD should be treated as a separate issue from ITT may implicitly accept complete case analysis as an acceptable alternative. Among clinical trialists, complete case analysis is the most popular approach to handle MOD: a recent review showed that complete case analysis was used in about half of RCTs reporting ITT. [5] These discrepancies highlight the current troubling state of the ITT principle in RCTs.

Solution to the Problem

Several institutions have acknowledged that varying meanings of the term ITT is problematic. In its updated 2010 statement, CONSORT [15], [16] no longer advocates the use of the term ITT and “replaced mention of ‘Intention to treat’ analysis, a widely misused term, by a more explicit request for information about retaining participants in their original assigned groups.” [15] Still, the ITT definition by CONSORT strictly forbids the exclusion of any randomized individuals from the analysis, although a complete case analysis is deemed reasonable in the presence of MOD. [16] The Cochrane handbook has recently expanded its presentation of ITT. The current version of the handbook lists three principles of ITT: [17].

“1. Keep participants in the intervention groups to which they were randomized, regardless of the intervention they actually received. 2. Measure outcome data on all participants. 3. Include all randomized participants in the analysis.”

It acknowledges that “there is no clear consensus on whether all criteria should be applied. [6] While the first is widely agreed, the second is often impossible and the third is contentious.” If there are MOD the Cochrane handbook mentions two options, complete case analysis and data imputation, recommending a sensitivity analysis in either case. Both options involve assumptions about missing data and may be problematic, but using imputation achieves the ‘ITT quality label’ and the – in our view equally justified – complete case analysis does not. [17].

We propose a solution to the problem: First, we must separate the issue for which there is consensus, the analysis of those for whom outcome data is available, from the issue in which no consensus exists, how to conduct an analysis according to the ITT principle in the presence of MOD. Both issues are important items for assessing the risk of bias in RCTs. Substantial MOD as well as ‘per protocol’ or ‘as treated’ analyses may introduce bias. [18], [19] If investigators adopt the fundamental strategy of analysing those with complete outcome data in the groups to which they were randomized, they can claim ITT; one can then define four sub-categories of MOD under ITT (Table 3).

To date, while authors have suggested reasonable alternatives for handling MOD, [20] no compelling empirical evidence exists to guide the optimal data analysis which most likely depends on the context. Furthermore, investigators will inevitably have to make unverifiable assumptions about the missingness process, because they can never know what would have happened had missing data actually been observed. [21] Therefore, the goal should always be to transparently and clearly report how investigators dealt with MOD in their analysis. This is true both for individual trials and for systematic reviews/meta-analyses of RCTs. Subsequent empirical investigations could address the impact of various imputation approaches on the robustness of trial results. [22].

Conclusion

We found a large variation in the definition of ITT in relation to MOD among methodology articles and how MOD should be handled in RCTs under ITT. We therefore propose to separate the ITT principle from handling of MOD in order to increase clarity and transparency of reporting trials and systematic reviews. We define four distinct sub-categories of ITT depending on the presence of patients with MOD and, if present, how investigators handle such patients in their analysis (Table 3). Use of this taxonomy would clarify the confusion in the current use of the term ‘ITT’.