Reporting of eligibility criteria of randomised trials: cohort study comparing trial protocols with subsequent articles
BMJ 2011; 342 doi: https://doi.org/10.1136/bmj.d1828 (Published 05 April 2011) Cite this as: BMJ 2011;342:d1828
- Anette Blümle, research fellow1,
- Joerg J Meerpohl, research fellow12,
- Gerta Rücker, research fellow1,
- Gerd Antes, director1,
- Martin Schumacher, director1,
- Erik von Elm, senior researcher13
- 1German Cochrane Center, Department of Medical Biometry and Statistics, Institute of Medical Biometry and Medical Informatics, University Medical Center Freiburg, Berliner Allee 29, 79110 Freiburg, Germany
- 2Division of Pediatric Hematology and Oncology, Department of Pediatrics, University Medical Center Freiburg, Germany
- 3Swiss Paraplegic Research, Nottwil, Switzerland
- Correspondence to: A Blümle bluemle{at}cochrane.de
- Accepted 9 January 2011
Abstract
Objective To determine whether and how eligibility criteria of participants prespecified in protocols of randomised trials are reported in subsequent articles.
Design Cohort study.
Setting Protocols submitted to the ethics committee of a German medical faculty.
Data sources 52 trial protocols and 78 subsequent publications published between 2000 and 2006.
Main outcome measure Proportion of matching, missing, modified, or newly added eligibility criteria between trial protocols and subsequent publications.
Results Differences were found between protocols and subsequent publications for all 52 trials. Information on eligibility criteria was missing in the publications for all 52 trials (100%, 95% confidence interval 93% to 100%), modified for 44 (85%, 72% to 93%), and newly added for 21 (41%, 27% to 55%). The mean number of eligibility criteria for each trial was 25 (range 7-43) and the mean proportion of matching eligibility criteria per trial was 50% (95% confidence interval 44% to 55%, range 13-93). Of 1248 eligibility criteria prespecified in the protocols, 606 (49%, 46% to 51%) were matching in subsequent publications, 479 (38%, 36% to 41%) were missing, and 163 (13%, 11% to 15%) were modified. 51 eligibility criteria were added to publications. Most prespecified eligibility criteria were about comorbidity (42%, 39% to 45%), treatment (20%, 18% to 22%), or type or severity of illness (17%, 15% to 19%). Most of the missing eligibility criteria (96%, 94% to 97%) and modified eligibility criteria (54%, 46% to 62%) suggested broader study populations and most of the added eligibility criteria (86%, 74% to 94%) suggested narrower study populations.
Conclusions Many users of trial information rely on published journal articles. These articles generally do not reflect the exact definition of the study population as prespecified in the protocol. Incomplete or inadequate reporting of eligibility criteria hampers a proper assessment of the applicability of trial results.
Introduction
Published information about clinical trials is often incomplete.1 If the reporting is selective the available information misrepresents the scientific evidence. It has long been recognised that only part of the results of trials are published in peer reviewed journals. For instance, “positive” results confirming the effect of new treatments are more likely to be published.2 Evidence is now accumulating that not only entire studies but also individual study outcomes are reported selectively, leading to outcome reporting bias.3 4 The reporting of trial results and related biases has been studied extensively, but less is known about how the design of trials and methods is reported. Definitions of medical interventions have been shown to be reported incompletely.5 Incomplete reporting of trials also occurs for other key information, such as the calculation of sample size and methods of statistical analysis. For example, only 18% of trials approved by a scientific ethics committee in Denmark in 1994-5 described a priori sample size calculations adequately in both the protocol and the publications.6 Differences between protocols and publications were also found for the reporting of allocation concealment and adverse events.7 8
Little is known about the reporting of the eligibility criteria used for selection of trial participants. A precise definition of a trial’s study population is important to assess whether results can be applied to other patients with the same condition.9 Trial participants often do not represent the patient population that clinicians see in their daily practice.10 11 12 If, for example, it is unclear whether patients with particular comorbidities were excluded from a trial, practitioners using the trial report for medical decision making cannot know whether the results can apply to their patients with one of these comorbidities. Unreported exclusion criteria falsely let the reader assume that the results of a trial are more widely applicable. A patient with several health problems may, however, be at increased risk of severe side effects of the intervention.
We characterised the eligibility criteria defined in trial protocols to investigate whether they are reported in subsequent articles and described trial characteristics potentially associated with differences in the reporting of eligibility criteria.
Methods
Cohort of study protocols
The research ethics committee of the University of Freiburg, Germany, granted access to all study protocols submitted in 2000 for approval, including amendments, progress reports, and correspondence. We classified the design of all submitted studies and included only randomised controlled trials. Of 141 randomised controlled trials identified, 103 were completed. The factorial design was considered a variant of a parallel group design; we classified crossover trials as randomised studies only if the treatment allocation had been randomised. We systematically searched Medline (Ovid), the Cochrane Central Register of Controlled Trials, Current Contents, full-text databases of several publishers, and the university’s publication registry up to March 2007. For each protocol we designed a new search strategy including relevant keywords, such as experimental drug, study name or acronym, studied disease or condition, and names of applicants, which also allowed for variants of keywords and additional search terms. The literature search was complemented by a survey of the investigators asking them to confirm corresponding publications already identified by us and to indicate additional ones. The methodology is described in detail elsewhere.13
Data collection
We screened the trial protocols and amendments for information on eligibility criteria. Relevant information that was labelled differently or found in other protocol sections or in a synopsis was also considered pertinent. As the protocol was unavailable for two trials we used the information on the mandatory application form submitted to the ethics committee, which required information on eligibility criteria. Data were extracted into a spreadsheet and compared with those reported in the publications. We classified eligibility criteria into seven categories according to their content (box). The classification was developed as we went along with data extraction, and reassessed for consistency after completion.
Categories of eligibility criteria according to content
Comorbidity
Previous or current health condition not studied in the trial. Also used if investigators were allowed to exclude people for any other health condition
Treatment
Previous or current drug intake, surgery, or participation in another study
Type or severity of illness
Illness or health condition studied in the trial
Pregnancy related criteria
Criteria concerning pregnancy, lactation, and contraceptive methods
Personal criteria
Examples are age, sex, ethnicity
Diagnostic procedure
Concerns the procedure, not the result, of a diagnostic measurement. Examples are measurement of blood pressure or a specific laboratory test as a prerequisite for participation in the trial
Other
Examples are specific individual criteria, such as exclusion if a partner is pregnant or inclusion depending on place of residence, or informed consent. Also used if investigators were allowed to exclude people for any other reason, such as anticipated non-compliance
We then classified eligibility criteria as matching, missing from, modified, or added in a publication. Eligibility criteria were considered matching if information specified in the protocol was identical in the publications, missing if not reported in the publications, and modified if reported in the publications but changed in minor aspects. Eligibility criteria not prespecified in the protocol but newly mentioned in publications were classified as added. When comparing information in protocols with that in publications we focused on the content rather than exact wording. If an eligibility criterion was described as an inclusion criterion in one source but an exclusion criterion in the other we regarded it as matching. Criteria mentioned twice in the same data source (for example, termed as inclusion and exclusion criterion) were considered only once. If eligibility criteria were changed in amendments to the protocol, we considered the most recent information. For protocols with several corresponding articles we regarded an eligibility criterion as matching if the information in the protocol matched with at least one article and as missing if not mentioned in any of the articles. If publications referred to study methods published elsewhere, we retrieved this additional information. From the publications we extracted the definitions of the study population only as stated in the methods sections and disregarded information about the study population actually achieved (such as the baseline characteristics of participants usually described in the results section). If an eligibility criterion was composed of two or more criteria we classified each separately. For example, in one trial, participants with a history of major surgery or trauma within the past three months were excluded, and we regarded this as two criteria, one related to treatment (surgery) and one to comorbidity (trauma). Two investigators independently extracted all data; a total of four investigators participated in data extraction. Disagreements were resolved by discussion and consensus.
For each missing, modified, or added criterion, we considered whether the difference between protocol and publication would broaden or narrow the study population assumed by a reader of the publication. For example, if the age range of participants was defined as “60-80 years” in the protocol but not in the publication, readers would assume that people aged under 60 and over 80 were also eligible and would infer a broader study population. Inversely, readers would assume a narrower study population if comorbidities were mentioned as exclusion criteria in the trial report but not in the protocol.
Statistical analysis
Our study is based on a sample of trial protocols, each including several eligibility criteria. We analysed the data in two ways. Firstly, we used the protocols as the unit of analysis with eligibility criteria stratified by trial (analyses based on trial protocols). For each trial we calculated separately the proportion of eligibility criteria matching between protocols and publications (nominator) and all eligibility criteria defined in the protocol (denominator). For proportions we calculated binomial 95% confidence intervals. The primary outcome was the proportion of matching eligibility criteria. Using an exploratory approach, we investigated whether the proportion of matching eligibility criteria was associated with key characteristics of the trials, including sample size, centre status (multicentre or single centre study), and industry funding or other. For instance, we wondered whether the interests or constraints of authors carrying out large multicentre trials funded by industry would differ from those of authors carrying out small investigator driven trials and whether this would result in discrepancies in the study populations defined in the protocols and subsequent trial reports. We used a multiple linear regression model to assess the relations between the logit transformed proportion of matching eligibility criteria and study characteristics. Secondly, we used the individual eligibility criterion as the unit of analysis (analyses based on eligibility criteria) to calculate the distribution of inclusion or exclusion criteria for each content category and the different types of eligibility criteria (matching, missing, modified, added). We used Stata/SE 10.1 for Unix (February 2009) for calculations.
Results
One or more publications were identified for 54 of the 103 completed trials (52%, 95% confidence interval 42% to 62%). One protocol was excluded because it was on diagnostic devices and another because subsequent publications referred to a cross sectional substudy. The final sample comprised 52 trial protocols. In 30 of these (58%, 43% to 71%) study methods were changed, as documented in the amendments submitted to the research ethics committee. In 13 (25%, 14% to 39%) protocols the eligibility criteria were changed. Table 1⇓ lists the characteristics of the trials.
Characteristics of included trials
Overall, 78 trial reports published between 2000 and 2006 were identified in 50 journals: 36 trials resulted in one publication each and 16 in two or more. One article referred to an online data supplement and two others to documents specifying eligibility criteria, but none included further definitions of the study population.
Analyses based on trial protocols
In all 52 trials differences were found between the protocol and subsequent publications. Eligibility criteria were missing in the publications for all 52 trials (100%, 93% to 100%), modified for 44 (85%, 72% to 93%), or newly added for 21 (41%, 27% to 55%). Across all 52 trials the proportion of matching eligibility criteria ranged between 13% and 93% (mean 50%, 95% confidence interval 44% to 55%).
Variables included in the exploratory regression model were sample size, centre status, and industry and other funding. These study characteristics were not associated with the proportion of matching eligibility criteria, although the statistical power to detect associations may have been low (table 2⇓).
Results of multiple regression analysis of factors potentially associated with proportion of matching eligibility criteria
For each of the 52 trials the proportion of missing or modified criteria that would lead a reader to assume a broader or narrower study population was calculated. Six missing and 26 modified criteria were excluded because the direction of influence was unclear. The mean proportion of criteria suggesting a broader study population was 85% (95% confidence interval 80% to 91%, range 20-100). The mean proportion of criteria suggesting a narrower study population was 9% (5% to 13%, range 0-50).
Analyses based on eligibility criteria
Overall, 1248 eligibility criteria were identified in the 52 protocols; of those, 606 (49%, 95% confidence interval 46% to 51%) were matching in the publications, 479 (38%, 36% to 41%) were missing, and 163 (13%, 11% to 15%) were modified. Fifty one eligibility criteria relating to 21 trials were not mentioned in the protocols but were added to the publications. Table 3⇓ gives examples of these discrepancies. Trial protocols comprised a mean of 25 eligibility criteria (range 7-43). Of all 820 eligibility criteria reported in the publications, 6% were added. Most of the missing eligibility criteria (96%, 94% to 97%) and modified eligibility criteria (54%, 46% to 62%) suggested broader study populations, and most of the added eligibility criteria (86%, 74% to 94%) suggested narrower study populations. Of all 1299 eligibility criteria mentioned in either protocols or publications or both, 422 (32%, 30% to 35%) were inclusion criteria and 856 (66%, 63% to 69%) were exclusion criteria. In one trial, 21 eligibility criteria (2%, 1% to 3%) were labelled neutrally as patient selection criteria. A mean of eight criteria were related to inclusion and 16 to exclusion. The most common content category was comorbidity, followed by treatment and type or severity of illness (table 4⇓). Criteria related to comorbidity, treatment, and pregnancy were mainly expressed as exclusion criteria, whereas criteria on type or severity of illness, personal characteristics, and diagnostic procedures were mostly inclusion criteria. The seven content categories had similar proportions of matching and not matching criteria (figure⇓).
Number of matching, missing, modified, and added eligibility criteria (n=1299) for each content category
Examples of eligibility criteria that differed between trial protocols and subsequent publications
Eligibility criteria classified by content category and type of discrepancy of protocol and subsequent publications
Discussion
Considerable differences exist between the eligibility criteria specified in protocols of randomised trials and what is reported in subsequent publications. For all trials a proportion of prespecified eligibility criteria were either missing or modified in the publications. To a much lesser extent, eligibility criteria were not prespecified in the protocols but were added to the publications. Our findings are supported by an earlier analysis of reports of trials in hip and knee osteoarthritis showing that key information needed to assess the external validity of trials was poorly reported.14 In fact essential information such as the method and duration of recruitment was described in less than half of the examined publications. Furthermore, in an analysis of 283 reports of trials published between 1994 and 2006 in high profile general medical journals the reporting of exclusion criteria was often poor and incomplete.15 In this analysis, 84% of trial reports contained at least one poorly justified exclusion criterion, and in 61% more than one quarter of the trial’s exclusion criteria were poorly justified. Our study did not allow for the effect of the discrepancies on study size or results to be quantified. This would have required access to the individual data of all people eligible for and participating in the 52 trials. Our findings suggest that discrepancies in the reporting of eligibility criteria are common. This has several implications. Firstly, most discrepancies would let readers assume a falsely broader study population, consequently inferring wider applicability of the trial’s results. For example, in one trial, participants with “primary refractory disease” were excluded according to the protocol but included according to the publication (table 3). Clinicians using the trial results in practice could assume that the study intervention had been studied in patients with primary refractory disease and might consider treating such patients accordingly. Secondly, differences in study populations are a source of heterogeneity in the synthesis of trial results, such as in meta-analyses.16 A proper analysis of sources of study heterogeneity is hampered if researchers are unable to determine a difference between study populations in trials investigating the same or similar interventions. Thirdly, the published information often did not allow determination of the exact study population but was suggestive of broader study populations in most cases. Trialists and sponsors may have vested interests in a wider applicability of their results—for example, in different age groups or subgroups of patients with other types or stages of disease.17 Consequently, any ambiguous information may influence policy decisions, such as the patient groups for which a new drug will be approved.
On average, the trial protocols in our study had 16 exclusion criteria, and many of these were about comorbidity or concomitant medical treatments, thus confirming results of an earlier study.15 Consequently, individuals eligible for trial inclusion are more selected and homogenous than the populations of patients in which study results will be applied later. In most of the studied trials clinical criteria such as comorbidity were used to decide on eligibility. Accordingly, an earlier analysis of randomised controlled trials on HIV showed that by using such criteria a large part of a representative cohort of women infected with HIV in the United States would have been excluded from participation in the trial.18
Selective enrolment of participants was also found in cardiovascular trials. Relative to their disease prevalence women and adults aged 75 or older were underrepresented in trials of acute coronary syndromes, although formal eligibility criteria did not preclude their participation.19 It has been recommended that each eligibility criterion should be justified explicitly and reassessed when new trials are planned.20 Almost all study protocols used the distinction between inclusion and exclusion criteria. Only one used the term “patient selection criteria”. It has been argued that eligibility criteria can be formulated in both ways and that listing the same condition as an inclusion criterion and again as an exclusion criterion in the protocol is redundant.15
Limitations and strengths of the study
We accessed trial protocols approved during one year by the research ethics committee of a German medical faculty and used rigorous and comprehensive methods to ascertain subsequent full publications. Nevertheless, we identified publications for only about half of the included trials.13 This low proportion of fully published trials is consistent with findings from other cohorts following trials from the protocol stage to full publication.2 Given that ethical approval is mandatory and inclusion of protocols in our study did not depend on the trialists’ consent, the included trials represented an unbiased sample of interventional clinical research projects. As most trials were international multicentre studies our findings may well be applicable to trial reports from other settings or countries. However, our data sources also have some limitations. We could not ascertain whether the accessed files comprised all the documents of the submitted trials. For two trials the detailed study protocol was not available (multicentre studies already approved elsewhere), and data extractions were solely based on the study information requested in the mandatory application forms. The study protocols were written about 10 years ago and the corresponding publications in the following years. If the reporting of eligibility criteria has improved in the meantime our findings would probably overestimate the current magnitude of the problem. Some of our study methods were developed for the purpose of this study. For example, in the absence of a standard we developed our own classification for content categories. We used some common sense approaches, such as defining what represents a difference or not, and in these instances alternatives would have been conceivable. Furthermore, the sample of 52 trials might have been too small to identify factors associated with a higher proportion of discrepancy in eligibility criteria.
Possible reasons for discrepancies
Trial protocols often comprise detailed descriptions of study methods largely exceeding the length of methods sections in journal articles. Restrictions on print space have been a possible reason for discrepancies between protocols and subsequent publications in the past and may have explained why eligibility criteria deemed less important were omitted from journal manuscripts. With the rise in electronic publishing, journals increasingly offer publication of comprehensive versions of manuscripts or appendices on their websites. Space restrictions should no longer be a reason for omission of important information from publications. We found many modified or additional criteria in the publications. Clearly, changes to eligibility criteria may become necessary, for instance because of problems during the study course such as slow recruitment of trial participants or unexpected side effects in a particular group of participants. However, in these cases, the changes to the study protocol should be documented in amendments of the original study protocol. Furthermore, prespecified criteria, such as the exclusion of people with relatively rare comorbidities, possibly were never applied during recruitment of participants and consequently not mentioned in later publications. Lastly, the observed inaccuracy in reporting of trial information may have other reasons such as different authors of protocols and publications, long time lags between the writing of protocols and publications, or mere carelessness in drafting manuscripts.
With our approach, a protocol listing detailed eligibility criteria probably yields more discrepancies in subsequent publications on the level of individual eligibility criteria than a protocol with only broad terms, such as for target diseases or comorbidities. Importantly, our study could not determine any causes of discordance and to what extent the observed differences reflected real changes during study conduct.
What could be done?
The consolidated standards of reporting trials (CONSORT) statement aims at helping authors include all the essential information in reports about medical research.21 A complete description of eligibility criteria is one of its elements. Although the CONSORT statement was first published in 1996, the reporting of trials is still poor at present.22 23 If in trial reports some eligibility criteria are reported as prespecified but others are modified or not reported at all, it is impossible for readers to obtain a clear view on who actually was eligible to participate in a given trial. Of the 50 journals publishing the 78 articles included in our study, 23 (46%) endorsed the CONSORT statement in their author instructions (as of May 2009). However, an assessment of the completeness and correctness of the information on study populations in submitted manuscripts would require access to the trial protocols. Consequently, some journals ask for the protocol to be submitted along with the manuscript. To improve the content of protocols of randomised controlled trials, the standard protocol items for randomized trials (SPIRIT) initiative is currently developing guidelines for the core information to be included in trial protocols.24
Carefully drafted protocols may also lead to improved information in publicly accessible trial registries. These registries are a valuable source of study information, allowing users to retrace and compare the definitions used at the different stages of the research.4 Twenty items are considered the minimal set of trial information that the World Health Organization’s international clinical trials registry platform requests for a trial to be regarded as prospectively registered, of which one is about inclusion and exclusion criteria for participant selection.25 Mandatory prospective trial registration has become a requirement for publication of trial reports in many journals.26 Better access to the detailed study information would facilitate the assessment of the validity of clinical trials.27 Consequently, trial protocols should be made publicly accessible to make the conduct and reporting of clinical research more transparent.
Conclusion
We showed that the eligibility criteria published in trial reports do not adequately reflect those prespecified in the study protocols. This may have consequences for clinical practice, research, and policy. Even if a reader has precise information about a trial’s study population, the interpretation of published results of therapeutic research is fraught with problems. In contrast with a study’s internal validity, its applicability cannot be assessed without substantial information that goes beyond the study itself.28 Often it is unclear whether an experimental intervention is also effective in a different clinical scenario.9 29 30 The difficulties in using trial results in clinical practice, research, and policy augment if the published information on the study populations is incomplete or reported selectively.
What is already known on this topic
Deficiencies in the reporting of results from randomised trials have been described
Trial reports often lack important information about study methods, such as randomisation
Detailed information on trial participants is important for readers who want to know to whom the results can be applied
What this study adds
Published eligibility criteria did not adequately reflect the study populations as prespecified in trial protocols; about half of the eligibility criteria were either modified or not reported in the publications
Discrepancies between prespecified and published eligibility criteria make a proper assessment of the applicability of trial results in clinical practice, research, and policy more difficult
Trial protocols should be made publicly accessible to make the conduct and reporting of clinical research more transparent
Notes
Cite this as: BMJ 2011;342:d1828
Footnotes
We thank the University of Freiburg’s research ethics committee for access to its files and cooperation and Julia Lindenberg for help with data extraction.
Contributors: All authors contributed to the study concept and design, data analysis, and interpretation. AB is guarantor; she had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. AB and EvE drafted the manuscript.
Funding: This study was funded by the German Research Foundation (DFG) (EL 544/1-1). The funder had no role in the design and conduct of the study, data collection and analysis, interpretation of the data, writing of the manuscript, and decision to submit the manuscript for publication.
Competing interests: All authors have completed the Unified Competing Interest form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: AB was supported by the German Research Foundation (DFG) (EL 544/1-1); no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.
Ethical approval: The University of Freiburg’s research ethics committee approved access to all study protocols submitted in 2000, including amendments, progress reports, and correspondence. Ethical approval for the research was not required.
Data sharing: No additional data available.
This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/2.0/ and http://creativecommons.org/licenses/by-nc/2.0/legalcode.