Original ArticleMeta-regression detected associations between heterogeneous treatment effects and study-level, but not patient-level, factors
Introduction
Series of randomized controlled trials (RCTs) collected for meta-analysis often exhibit substantial heterogeneity of treatment effects. For example, Engels et al. [1] detected statistically significant heterogeneity in about half of the 125 meta-analyses they reviewed. Interpreting heterogeneous clinical study results has become a key issue when using evidence for clinical decision making. Increasingly, researchers have recognized that study results differ, not because they are inherently measuring different processes, but because variation in study protocols, study cohorts, or study quality can affect the magnitude of observed treatment efficacy [2], [3]. In these cases a single estimate of treatment effect may be insufficient. Regression offers a particularly parsimonious description of the heterogeneity [4].
Increasing evidence indicates that much variation in treatment efficacy can only be explained by relating outcomes to risk factors measured on individual patients with patient-level data [5], [6]. Nevertheless, the usual unavailability and expense of collecting such data and the availability of summary data from published studies has led to the application of meta-regression for predicting summary treatment effects by summary patient statistics across studies [7], [8], [9]. In meta-regression, the study is the unit of analysis and the outcomes are treatment effects (e.g., odds ratios). These treatment effect outcomes are correlated with risk factors measured at the study level. A significant correlation suggests treatment interaction; the treatment effect varies with the factor. These factors are inherently either true study-level factors applying equally to all patients in a study- or patient-level factors that take different values for each patient, but are then aggregated into a summary study statistic. Study-level variables typically relate to the design of the study and involve the nature of the intervention (e.g., medication dose, concomitant treatments, type of controls), the location of the participants or the planned length of follow-up. Some study-level variables, such as use of a proper randomization technique, a placebo control, or application of double blinding, relate to the quality of the study design [10]. Information about patient-level factors typically derives from descriptive measures such as the average age and blood pressure or the gender and ethnic composition of the trials.
Although meta-regression accommodates study-level factors directly, the technique has some serious limitations when factors vary at the patient level. First, group-level variables are subject to ecological bias [11], [12] and may not represent the individual-level effects for which they are supposedly surrogates. For example, the effect of moving to an area with a higher mean socioeconomic status may be quite unlike the effect of increasing one's own status without moving. Second, summary statistics will not vary as much as the values upon which they are based. Thus, meta-regression may fail to capture within-study treatment variation across a covariate because the between-study averages vary little [5]. Third, few studies report all relevant covariates. A bias may be introduced because only information pertinent to a significant interaction is reported or because the relevance of some factors may have been unrecognized in earlier studies. Finally, the effective sample size of the meta-regression is the number of studies, usually a very small number. Meta-regression, then, would seem to be most useful for studying factors that vary at the study, rather than the patient level.
Nevertheless, meta-regression has been successfully applied with both study-level and patient-level variables. Berkey et al. [13] showed that the efficacy of the BCG vaccine for tuberculosis increased with distance of the study site from the equator. Thompson demonstrated that cholesterol-lowering drugs were more effective in reducing ischemic heart disease in studies in which the treatment groups achieved greater average reductions in serum cholesterol levels relative to their respective control groups [9]. Schmid demonstrated that mortality was inversely proportional to the time between onset of chest pain and treatment with a thrombolytic agent among patients suffering a myocardial infarction [7]. Berlin et al. [6] showed that failure of kidney transplants was reduced significantly in patients with elevated panel reactive antibodies (>20%). In the latter two cases, analysis of the interactions at the patient level with individual patient data validated the meta-regressions.
When meta-regression fails, it may still be possible to use the baseline risk (the average outcome in the control group) as an aggregate proxy for individual level risk in meta-regression [14], [15], [16], [17]. This global measure potentially reflects multiple factors, including different study populations, underlying risk, severity of illness, length of follow-up, and the method of treatment delivery. To its advantage, the baseline risk is always available, and is a convenient data reduction tool that succinctly summarizes diverse potential risks in a single variable. But it cannot differentiate multiple effects as a study-level variable, and is not directly interpretable for an individual.
In this article, after describing the statistical models used we examine the utility of meta-regression empirically through two investigations. The first compares meta-regression using aggregate data with analysis of individual patient data from a collection of 11 trials investigating the efficacy of angiotensin converting enzyme (ACE) inhibitors for treating patients with renal disease but no diabetes [18], [19]. The expense of collecting extensive data on many variables from each patient in each study has limited the number of comparisons of the aggregate and individual level approaches. Several comparisons have focused on the average treatment effect, ignoring covariates [20], [21]; others have directly examined treatment interaction with covariates [6], [7], [22]. The second investigation reports the results of meta-regression applied to study-level covariates (including baseline risk) available in the reports of 232 meta-analyses of RCTs collected from overviews in the Cochrane Collaboration [23], [24] and from 24 medical journals searched between 1990 and 1998. We sought to determine how often and with which study features meta-regression can find treatment interactions. Finally, we discuss our findings in a concluding section.
Section snippets
Multilevel model for patient-level and study-level factors
We use a multilevel modeling structure to appropriately incorporate all sources of uncertainty in the generation of the observed outcomes [25], [26], [27]. This structure consists of an observational level that describes the generation of the observed data conditional on the true unknown study random effects and a structural level that relates the random effects across studies. This parsimonious description permits the simultaneous estimation of individual study effects and the population
Case study: ACE inhibitors for nondiabetic renal disease
As an example of the application of these methods, we consider a database of individual patient data from 11 clinical trials [19], [40]. By selecting summary statistics and measures from each study, we can use this database to compare meta-regression of summary data with analysis of individual patient data.
ACE inhibitors are drugs that help reduce blood pressure and protein in the urine. Although best known to fight cardiovascular disease, ACE inhibitors have been shown in a number of clinical
Case study: study-level covariates across many meta-analyses
Although individual patient data offer the best evidence for exploring treatment interactions, most meta-analysts only have access to summary data. We therefore sought to discover how often treatment effects were correlated with aggregate variables and whether the frequency of significant correlations had any discernible patterns in a large collection of meta-analyses of randomized controlled trials.
We selected meta-analyses of randomized controlled trials from two main sources: major medical
Discussion
Under ideal conditions of availability and cost, meta-analyses with individual patient data are superior to those using summary statistics reported in the literature. Even when interest focuses only on a common treatment effect available with either method, comparison of the two approaches has shown that the individual patient data add important information such as event times that improve pooled treatment effect estimates [20], [21], [43]. The collaboration of the study teams needed for
Acknowledgements
This work was supported by grants RO1-HS10064 from the Agency for Healthcare Research and Quality, and RO1-DK53869A from the National Institute of Diabetes and Digestive and Kidney Diseases. The authors would like to thank the members of the Angiotensin-Converting Enzyme Inhibition in Progressive Renal Disease Study Group which include Drs. P.C. Zucchelli (Malpigi-Bologna, Italy); A. Kamper and S. Strandgaard (Copenhagen, Denmark); R.D. Toto (Dallas, TX); B.M. Brenner, N.E. Madias, T. Karim, M.
References (58)
- et al.
A comparison of summary patient-level covariates in meta-regression with individual patient data meta-analysis
J Clin Epidemiol
(2002) - et al.
Summing up evidence: one answer is not always enough
Lancet
(1998) - et al.
Constructing a database of individual clinical trials for longitudinal analysis
Control Clin Trials
(2003) - et al.
Individual patient- versus literature-based meta-analysis of survival data: time to event and event rate at a particular time can make a difference, an example based on head and neck cancer
Control Clin Trials
(2001) - et al.
Meta-analysis of the literature or of individual patient data: is there a difference?
Lancet
(1993) - et al.
Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses?
Lancet
(1998) - et al.
Assessing the quality of reports of randomized clinical trials: is blinding necessary?
Control Clin Trials
(1996) - et al.
Heterogeneity and statistical significance in meta-analysis: an empirical study of 125 meta-analyses
Stat Med
(2000) - et al.
Investigating causes of heterogeneity in systematic reviews
Stat Med
(2002) Why sources of heterogeneity in meta-analysis should be investigated
BMJ
(1994)
Explaining heterogeneity in meta-analysis: a comparison of methods
Stat Med
Individual patient versus group-level data meta-regressions for the investigation of treatment effect modifiers: ecological bias rears its ugly head
Stat Med
Exploring heterogeneity in randomized trials via meta-analysis
Drug Inf J
B blockade after myocardial infarction: systematic review and meta-regression analysis
BMJ
Controversies in meta-analysis: the case of the trials of serum cholesterol reduction
Stat Methods Med Res
The hazards of scoring the quality of clinical trials for meta-analysis
JAMA
Uses of ecologic analysis in ecological research
Am J Public Health
Ecological inference
A random-effects regression model for meta-analysis
Stat Med
Importance of trends in the interpretation of an overall odds ratio in the meta-analysis of clinical trials
Stat Med
The population risk as an explanatory variable in research synthesis of clinical trials
Stat Med
Investigating underlying risk as a source of heterogeneity in meta-analysis
Stat Med
Effect of angiotensin-converting-enzyme inhibitors on the progression of nondiabetic renal disease: a meta-analysis of randomized trials
Ann Intern Med
A comparison of meta-analytic results using literature vs. individual patient data
JAMA
A case study comparing a meta-analysis and a pooled analysis of studies of sinonasal cancer among wood workers
Epidemiology
Cochrane database of systematic reviews. Issue 1
A multilevel model framework for meta-analysis of clinical trials with binary outcomes
Stat Med
Meta-analysis of continuous outcome data from individual patients
Stat Med
Cited by (211)
Secular Trend in GFR Decline in Non-Dialysis CKD Based on Observational Data From Standard of Care Arms of Trials
2024, American Journal of Kidney DiseasesModernizing evidence synthesis for evidence-based medicine
2023, Clinical Decision Support and beyond: Progress and Opportunities in Knowledge-Enhanced Health and HealthcareHeart rate and skin conductance associations with physical aggression, psychopathy, antisocial personality disorder and conduct disorder: An updated meta-analysis
2022, Neuroscience and Biobehavioral ReviewsAcceptance and Commitment Therapy for chronic pain on functioning: A systematic review of randomized controlled trials
2021, Neuroscience and Biobehavioral Reviews