Original ArticleA simple and valid tool distinguished efficacy from effectiveness studies
Introduction
Randomized controlled trials (RCTs) are the gold standard in evaluating the effects of treatments. To be clinically meaningful, results must be relevant to specific patient populations in specific settings [1]. Multiple factors determine the external validity (i.e., generalizability or applicability) of RCTs: patient characteristics, condition under investigation, drug regimens, costs, compliance, comorbidities, and concomitant treatments. For practical reasons, trials cannot always take these factors fully into consideration (e.g., costs, poor compliance). Also, certain aspects of study design—eligibility criteria, study duration, mode of intervention, outcomes, adverse events assessment, or type of statistical analysis—greatly influence the degree of generalizability, given an appropriate population of interest.
Clinicians and policymakers often distinguish between the efficacy and the effectiveness of an intervention. Efficacy trials (explanatory trials) determine whether an intervention produces the expected result under ideal circumstances. Effectiveness trials (pragmatic trials) measure the degree of beneficial effect under “real world” clinical settings [2]. Hence, hypotheses and study designs of an effectiveness trial are formulated based on conditions of routine clinical practice and on outcomes essential for clinical decisions.
Efficacy and effectiveness exist on a continuum. Generalizability depends largely on the viewpoint of the observer and the condition under investigation. Baseline patient characteristics (e.g., gender, age, severity of the disease, racial groups) are primary factors in generalizability; thus, depending on the population of interest, generalizability of the same study can range from low to high. Geographic settings (urban vs. rural) and health care systems can also be significant factors [1], although geography may have less influence on generalizability of drug trials than trials of other interventions (e.g., screening programs, behavioral therapy).
Ensuring generalizability may compromise internal validity. Under everyday clinical settings, factors such as patient or doctor preferences [3], [4], or patient–doctor relationships [5], [6] can influence response and compliance. Random allocation, allocation concealment, and blinding negate these factors, thereby increasing internal validity on the one hand and decreasing external validity on the other. Therefore, to some extent, the operational definition of “effectiveness trial” delineates the necessary trade-offs with internal validity. An ideal definition would balance this equilibrium at a point at which satisfactory internal validity accompanies a high degree of generalizability.
Systematic reviews, including meta-analyses, have become an increasingly important source of information for clinical practice. If well conducted, they synthesize large amounts of information and provide estimated effect sizes that have greater precision and generalizability than individual studies [7]. Distinguishing between efficacy and effectiveness contributes an important aspect to analyzing any body of clinical evidence. Furthermore, greater emphasis on effectiveness studies may lead to changes in presentation in systematic reviews and policy initiatives.
In this article, we propose and test seven hallmarks of study design to create a tool that can help researchers and those producing systematic reviews, as well as clinicians who are interested in the generalizability of study results, to distinguish more readily and more consistently between efficacy and effectiveness studies.
Section snippets
Methods
Based on clinical and methodological considerations and the published literature, we selected seven domains of study design that, in our view, demonstrably influence the generalizability of trial results (Table 1). We searched MEDLINE® to identify published literature on instruments to distinguish effectiveness from efficacy studies. We found various definitions of effectiveness studies [8], [9], [10], [11], [12] but no validated rating instruments. Additional searches on Web sites of the US
Results
The EPC directors identified 26 studies. Of these, 6 were intended to illustrate efficacy trials and 20, effectiveness studies. We excluded two studies from the latter group because they did not meet eligibility criteria [24], [25]: one was an observational follow-up of three RCTs [24] and the other a pooled analysis of clinical trials [25]. Of the remaining 24 studies (Table 2, alphabetical by author), 22 were RCTs, 3 with an open-label design [26], [27], [28]; the other 2 were a
Discussion
Our objective was to identify drug effectiveness studies reliably based on seven proposed criteria. We focused on studies of medications because they are common, but many of the same principles can be applied to other types of interventions. Because we attribute greater value to effectiveness studies than to efficacy studies, the specificity of this process had to be high. That is, we wanted to ensure that efficacy studies are not falsely rated as effectiveness studies. Erring on the side of
Acknowledgments
We would like to thankfully acknowledge the Directors of the United States and Canadian EPCs who selected the sample studies and provided insightful comments to the study protocol. We would also like to thank Jean Slutsky and Kenny Fink from AHRQ for their support of this methods project.
Disclaimer: The authors of this report are responsible for its content. Statements in the report should not be construed as endorsement by the AHRQ or the US Department of Health and Human Services of a
References (57)
External validity of randomised controlled trials: “to whom do the results of this trial apply?”
Lancet
(2005)- et al.
Explanatory and pragmatic attitudes in therapeutical trials
J Chronic Dis
(1967) - et al.
Large-scale randomized evidence: large, simple trials and overviews of trials
J Clin Epidemiol
(1995) - et al.
Current methods of the US Preventive Services Task Force: a review of the process
Am J Prev Med
(2001) Clinical trial efficacy: what does it really tell you?
J Allergy Clin Immunol
(2003)- et al.
Safety, efficacy and effectiveness of the influenza virus vaccine, trivalent, types A and B, live, cold-adapted (CAIV-T) in healthy children and healthy adults
Vaccine
(2001) - et al.
Randomised trial of efficacy and safety of oral ganciclovir in the prevention of cytomegalovirus disease in liver-transplant recipients. The Oral Ganciclovir International Transplantation Study Group
Lancet
(1997) - et al.
Rationale and design for the Antihypertensive and Lipid Lowering Treatment to Prevent Heart Attack Trial (ALLHAT). ALLHAT Research Group
Am J Hypertens
(1996) - et al.
Efficacy and safety of intravenous levosimendan compared with dobutamine in severe low-output heart failure (the LIDO study): a randomised double-blind trial
Lancet
(2002) - et al.
Pragmatic controlled clinical trials in primary care: the struggle between external and internal validity
BMC Med Res Methodol
(2003)
Patients' decisions about whether or not to take antihypertensive drugs: qualitative study
BMJ
Understanding patients' decisions. Cognitive and emotional perspectives
JAMA
General practice consultations: is there any point in being positive?
Br Med J (Clin Res Ed)
Influence of context effects on health outcomes: a systematic review
Lancet
Rationale for systematic reviews
BMJ
Efficacy, effectiveness, variations, and quality. Boundary-crossing research
Med Care
Introduction to the special section: efficacy and effectiveness in studies of child and adolescent psychotherapy
J Consult Clin Psychol
Principles from clinical trials relevant to clinical practice: Part I
Circulation
Undertaking systematic reviews of research on effectiveness: CRD's guidance for those carrying out or commissioning reviews. CRD Report Number 4 (2nd edition)
Reporting the recruitment process in clinical trials: who are these patients and how did they get there?
Ann Intern Med
Surrogate end points in clinical trials: are we being misled?
Ann Intern Med
Measuring compliance with inhaled medication in asthma
Arch Dis Child
Compliance with inhaled asthma medication in preschool children
Thorax
Comparison of patients' compliance with prescribed oral and inhaled asthma medications
Arch Intern Med
Interpretation of changes in health-related quality of life: the remarkable universality of half a standard deviation
Med Care
What is meant by intention to treat analysis? Survey of published randomised controlled trials
BMJ
Systems to rate the strength of scientific evidence
Evid Rep Technol Assess (Summ)
Donepezil is associated with delayed nursing home placement in patients with Alzheimer's disease
J Am Geriatr Soc
Cited by (166)
In clinical trials, efficacy vs. effectiveness language is confusing
2023, Journal of Clinical EpidemiologyPopular interventions to enhance sustained attention in children and adolescents: A critical systematic review
2022, Neuroscience and Biobehavioral ReviewsExploring the pragmatic-explanatory spectrum across cardiovascular clinical trials
2022, Contemporary Clinical TrialsGeneralizability of randomized controlled trials in primary health care: Applying the PRECIS-2 tool on published protocols
2023, Journal of Evaluation in Clinical Practice
Funding for this research was provided to the RTI-UNC EPC through a contract from the AHRQ to RTI International (contract number 290-02-0016). The funding source had no involvement in the design and conduct of the study, and in the analysis and interpretation of data.