There is no consensus on whether studies with no observed events in the treatment and control arms, the so-called both-armed zero-event studies, should be included in a meta-analysis of randomised controlled trials (RCTs). Current analytic approaches handled them differently depending on the choice of effect measures and authors' discretion. Our objective is to evaluate the impact of including or excluding both-armed zero-event (BA0E) studies in meta-analysis of RCTs with rare outcome events through a simulation study.

We simulated 2500 data sets for different scenarios varying the parameters of baseline event rate, treatment effect and number of patients in each trial, and between-study variance. We evaluated the performance of commonly used pooling methods in classical meta-analysis—namely, Peto, Mantel-Haenszel with fixed-effects and random-effects models, and inverse variance method with fixed-effects and random-effects models—using bias, root mean square error, length of 95% CI and coverage.

The overall performance of the approaches of including or excluding BA0E studies in meta-analysis varied according to the magnitude of true treatment effect. Including BA0E studies introduced very little bias, decreased mean square error, narrowed the 95% CI and increased the coverage when no true treatment effect existed. However, when a true treatment effect existed, the estimates from the approach of excluding BA0E studies led to smaller bias than including them. Among all evaluated methods, the Peto method excluding BA0E studies gave the least biased results when a true treatment effect existed.

We recommend including BA0E studies when treatment effects are unlikely, but excluding them when there is a decisive treatment effect. Providing results of including and excluding BA0E studies to assess the robustness of the pooled estimated effect is a sensible way to communicate the results of a meta-analysis when the treatment effects are unclear.

A simulation study thoroughly investigated the impacts of including or excluding both-armed zero-event studies in meta-analyses by comparing all commonly used pooling methods.

The simulation parameters were chosen according to the characteristics of meta-analyses in the Cochrane Database of Systematic Reviews to closely reflect the reality.

Our results not only confirmed the findings from the previous empirical studies but also added more details on how including or excluding both-armed zero-event may affect the estimates of meta-analyses differently depending on the magnitude of true treatment effects.

Only OR was investigated through simulations; thus, the findings from this study may not be able to be fully extended to other effect measures such as relative risk or absolute risk difference.

Systematic review with meta-analysis has become an important research tool for the health research literature which synthesises evidence from individually conducted studies that assess the same outcomes on the same topic. The PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) Statement

This research focuses on the RCTs with dichotomous outcomes, that is, the participants do or do not experience the defined event. The total number of observed events in such a RCT is likely influenced by the event rate, sample size and study period. When the event rate is low, the sample size is small and the study period is short, it is possible that no outcome event is observed in the RCT although the probability of the event happening is not zero. A study with no outcome event observed in either treatment or control arms is called a zero-event study. An extreme case of the zero-event study is both-armed zero-event (BA0E) study, which is defined as a study in which no event is observed in treatment and control arms, and is also known as a double-zero-event or zero-total-event study.

When rare adverse events or rare diseases are used as the study outcomes, it is not an uncommon phenomenon that no outcome events are observed at the end of the study. In the USA, a rare adverse event is defined as 1 per 1000 patients.

For single-armed zero-event studies, there is consensus on their inclusion in meta-analyses. Bradburn

There are two major reasons why BA0E are handled variably in meta-analyses. First, the statistical methods and software such as RevMan (The Cochrane Collaboration. RevMan 5.1 User Guide. 2011) Stata's metan module

Since number of events observed in studies using dichotomous outcomes is determined by event rates and number of subjects, zero-events are more likely to occur with the conditions of extremely low event rates or small sample sizes even though the event rates are different between two study groups. In the intuitive way, the arithmetical difference between two study groups with no observed events is null. Therefore, we hypothesise that the inclusion of BA0E studies in meta-analysis affects the pooled estimates of treatment effects in different ways, depending on the presence or absence of a true treatment effect. In the absence of a true treatment effect, that is, similar event rates in both arms, the inclusion of BA0E studies narrows the CI of the pooled estimate of treatment effect. When a true treatment effect exists, the inclusion of BA0E studies adds bias to the magnitude of the pooled estimate, leading to the underestimation of the treatment effect.

To test the hypotheses, we conducted a simulation study to evaluate the impact of excluding and including BA0E studies in meta-analysis under two circumstances—the presence and absence of a true treatment effect. Although it is not difficult to statistically deduce that the bias brought by including BA0E studies is affected by the factors such as (1) low event rate, (2) large treatment effect and (3) small sample size, stimulation is still needed to quantifying the magnitude of the bias. Our investigation focused on comparing the standard statistical pooling methods adopted by the commonly used software such as RevMan and Stata V.13.1 for meta-analysing aggregated data. We hope our study can provide some practical guidance to the researchers in this area.

OR and RR are the most commonly used effect measures for assessing the treatment effect for dichotomous outcomes in meta-analyses. The results of these two effect measures are similar when the event probability is <20%.

The simulation scenarios in our study were chosen based on a combination of several simulation parameters. Three types of parameters were used in this simulation study: fixed (a single value was assigned to a certain parameter), varied (multiple values were assigned to a certain parameter) and derived (the value of a certain parameter was calculated according to a statistical formula). We believed that some parameters had more impact on the simulation results than others. We chose fixed values for the low impact parameters (unlikely to influence the simulation results) across all simulation scenarios, and let the values of those high impact parameters (more likely to influence the simulation results) vary in certain ranges. The parameter values were drawn from the published literature (

Simulation parameter setup

Parameter | Assigned values | Rationale | Reference |
---|---|---|---|

OR | 0.2, 0.5, 0.8, 1, 1.25, 2, 5 | No treatment effect, small to medium, large and extremely large treatment effects | |

Control group event probability (p) | 0.001, 0.005, 0.01 | 1 in 2000 rare diseases in EU; 1 in 1000 rare adverse events | |

Number of studies in each meta-analysis (m) | 5 | Median=3; IQR: 2–6; <1% >29 | |

Number of patients in each individual study (n) | 50, 100, 250 | Median=102; IQR 50–243 | |

Between-study SD | 0.1, 0.5, 1 | Small, moderate, large | |

Ratio of group size (r) | 1:1 | 78% trials had equal group ratio |

The derived parameters were calculated by the input parameters according to a statistical formula. For the fixed parameters, we tested the following values. The numbers of studies (m) in each meta-analysis was set at 5. The review published in 2011 reported that the median (IQR) of the numbers of studies included in the meta-analysis in the Cochrane Database was 3 with IQR from 2 to 6.

For the following parameters, we chose to input multiple values instead of constants. The control arm event probabilities (p) investigated in this simulation were 0.001, 0.005 and 0.01. They are chosen according to the varying definitions of rare events.

In this simulation study, the treatment arm event probabilities were calculated through the control arm event probabilities and treatment effects (OR)._{T}=treatment arm probability, p_{c}=control arm probability, Ω=OR, i=1, 2, …, study.

We simulated 2500 data sets for each scenario to ensure the accuracy of our simulation results.

Five pooling procedures were used to meta-analyse each simulated data set. They were Peto, Mantel-Haenszel with fixed-effects and random-effects models, and inverse variance (IV) method with fixed-effects and random-effects models.

To implement the above five pooled methods to incorporate studies with BA0E in meta-analysis, a continuity correction factor was added to each of the four cells of the 2×2 table for a BA0E study, that is, event in the treatment arm, non-event in the treatment arm, event in the control arm and non-event the in control arm. Continuity corrects were also used to incorporate single-armed zero-event studies for all methods except Peto's. We chose to use the constant continuity factor 0.5. It is common and plausible choice when the group ratio is balanced between treatment and control arms.

Four measures were used to assess the performance of this simulation study

Measures for evaluating simulation performance

Criteria | Formula |
---|---|

Percentage bias ((δ/β)%) | |

RMSE | |

Average width of 95% CI | |

Coverage of 95% CI | Percentage of times the 95% CI of |

β, true value of estimate of interest—log OR;

δ, absolute bias—the difference between the mean of the estimates of log OR and log OR;

We also reported the inclusiveness of the approach of excluding BA0E studies in meta-analysis, which reported the percentage of number of studies included in the pooling process.

Since the focus of our investigation was whether and when including BA0E studies would introduce bias to the pooled estimates of the treatment effect in meta-analyses, we evaluated the simulation performance regarding the bias in different scenarios by varying the values of the treatment effect, control arm probability, number of patient and between-study variance. Other simulation performance measures (RMSE, average width of 95% CI and coverage of 95% CI) were evaluated on a common simulation scenario to minimise the required amount of work.

The data sets for each simulation scenario are generated using R V.2.15.2 (The R Foundation for Statistical Computing; simulation code of data generating is attached as online

In this study, we reported 13 simulation scenarios based on the input values of the simulation parameters (treatment effect, control arm probability, number of patients and between-study variance). Among all simulated meta-analysis data sets, 31.5% (minimum=21%; maximum=40%) of them were BA0E studies.

Our simulation results supported our hypothesis that when there is no true treatment effect (OR=1), the approach of including BA0E studies in meta-analyses had the best overall performance regardless of the choice of pooling methods, which gave the smallest bias (<0.1%;

Impact of the treatment effect changes on bias

Number of studies=5 | Number of patients=100 | Group ratio=1 | Control arm probability=0.001 | Number of simulated data sets=2500 | Between-study SD=0.5 | |||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|

Excluding BA0E studies | Including BA0E studies | |||||||||||||||

Positive treatment effect | No treatment effect | Positive treatment effect | ||||||||||||||

OR=1.25 | OR=2 | OR=5 | OR=1 | OR=1 | OR=1.25 | OR=2 | OR=5 | |||||||||

Methods | %bias | %bias | %bias | %bias | % bias | %bias | %bias | %bias | ||||||||

IV random effects | 1.11 | −12.6 | 1.45 | −37.9 | 2.28 | −119.3 | 1.01 | 0.8 | 1.00 | <0.1 | 1.03 | −21.4 | 1.13 | −77.0 | 1.56 | −220.5 |

IV fixed effects | 1.11 | −12.6 | 1.46 | −37.0 | 2.28 | −119.3 | 1.01 | 0.7 | 1.00 | <0.1 | 1.03 | −21.4 | 1.15 | −73.9 | 1.56 | −220.5 |

M-H random effects | 1.11 | −12.5 | 1.45 | −37.9 | 2.28 | −119.3 | 1.01 | 0.8 | 1.00 | <0.1 | 1.03 | −21.4 | 1.13 | −77.0 | 1.56 | −220.5 |

M-H fixed effects | 1.11 | −12.6 | 1.46 | −37.0 | 2.30 | −117.4 | 1.01 | 0.8 | 1.00 | <0.1 | 1.03 | −21.4 | 1.15 | −73.9 | 1.62 | −208.6 |

Peto | 1.19 | −5.0 | 1.87 | −7.0 | 3.68 | −35.9 | 1.01 | 1.4 | 1.00 | <0.1 | 1.04 | −20.2 | 1.19 | −68.1 | 1.92 | −160.4 |

IV random effects | 0.88 | −9.9 | 0.70 | −40.6 | 0.47 | −133.1 | 0.99 | −23.2 | 0.97 | −93.0 | 0.94 | −370.7 | ||||

IV fixed effects | 0.88 | −9.9 | 0.70 | −40.6 | 0.47 | −133.1 | 0.98 | −23.0 | 0.96 | −92.3 | 0.93 | −367.4 | ||||

M-H random effects | 0.88 | −9.9 | 0.70 | −40.6 | 0.47 | −133.1 | 0.99 | −23.2 | 0.97 | −93.0 | 0.94 | −370.7 | ||||

M-H fixed effects | 0.88 | −9.9 | 0.70 | −40.6 | 0.47 | −133.1 | 0.98 | −23.0 | 0.96 | −92.3 | 0.93 | −367.4 | ||||

Peto | 0.80 | 0.2 | 0.54 | −7.8 | 0.26 | −30.6 | 0.95 | −22.6 | 0.90 | −90.6 | 0.92 | −360.9 |

BA0E, both-armed zero-event; IV, inverse variance; M-H, Mantel-Haenszel.

Impact of the control arm probability changes on bias

Number of studies=5 | Number of patients = 100 | Group ratio=1 | OR=0.5 | Number of simulated data sets=2500 | Between-study SD=0.5 | |||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|

Excluding BA0E studies | Including BA0E studies | |||||||||||

p_{c}=0.001 | p_{c}=0.005 | p_{c}=0.01 | p_{c}=0.001 | p_{c}=0.005 | p_{c}=0.01 | |||||||

Methods | %bias | %bias | %bias | %bias | %bias | %bias | ||||||

IV random effects | 0.70 | −40.6 | 0.68 | −35.1 | 0.64 | −28.5 | 0.97 | −93.0 | 0.85 | −70.8 | 0.76 | −51.3 |

IV fixed effects | 0.70 | −40.6 | 0.67 | −34.9 | 0.64 | −27.3 | 0.96 | −92.3 | 0.84 | −68.5 | 0.74 | −48.0 |

M-H random effects | 0.70 | −40.6 | 0.68 | −35.1 | 0.64 | −28.5 | 0.97 | −93.0 | 0.85 | −70.7 | 0.76 | −51.3 |

M-H fixed effects | 0.70 | −40.6 | 0.67 | −34.9 | 0.64 | −27.3 | 0.96 | −92.3 | 0.84 | −68.5 | 0.74 | −48.0 |

Peto | 0.54 | −7.8 | 0.52 | −4.6 | 0.51 | −1.1 | 0.90 | −90.6 | 0.80 | −59.5 | 0.67 | −33.2 |

BA0E, both-armed zero-event; IV, inverse variance; p_{c}, control arm probability; M-H, Mantel-Haenszel.

Impact of the number of patient changes in each individual study on bias

Number of studies=5 | Control group probability=0.001 | Group ratio=1 | OR=0.5 | Number of simulated data sets=2500 | Between-study SD=0.5 | |||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|

Excluding BA0E studies | Including BA0E studies | |||||||||||

n=50 | n=100 | n=250 | n=50 | n=100 | n=250 | |||||||

Methods | %bias | %bias | %bias | %bias | %bias | %bias | ||||||

IV random effects | 0.73 | −45.7 | 0.70 | −40.6 | 0.68 | −36.5 | 0.98 | −96.8 | 0.97 | −70.7 | 0.93 | −86.0 |

IV fixed effects | 0.73 | −45.8 | 0.70 | −40.6 | 0.68 | −36.3 | 0.98 | −96.5 | 0.96 | −68.5 | 0.92 | −84.5 |

M-H random effects | 0.73 | −45.7 | 0.70 | −40.6 | 0.68 | −36.5 | 0.98 | −96.8 | 0.97 | −70.7 | 0.93 | −86.0 |

M-H fixed effects | 0.73 | −45.8 | 0.70 | −40.6 | 0.68 | −36.3 | 0.98 | −96.5 | 0.96 | −68.5 | 0.92 | −84.5 |

Peto | 0.58 | −15.2 | 0.54 | −7.8 | 0.51 | −2.4 | 0.98 | −95.8 | 0.95 | −59.5 | 0.90 | −80.7 |

BA0E, both-armed zero-event; IV, inverse variance; M-H, Mantel-Haenszel.

Impact of the between-study variance changes on bias

Number of studies=5 | Control group probability=0.001 | Group ratio=1 | OR=0.5 | Number of simulated data sets=2500 | Number of patients per arm=100 | |||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|

Excluding BA0E studies | Including BA0E studies | |||||||||||

SD=0.1 | SD=0.5 | SD=1 | SD=0.1 | SD=0.5 | SD=1 | |||||||

Methods | %bias | %bias | %bias | %bias | %bias | %bias | ||||||

IV random effects | 0.68 | −35.3 | 0.70 | −40.6 | 0.88 | −76.7 | 0.96 | −92.5 | 0.97 | −93.0 | 0.99 | −97.3 |

IV fixed effects | 0.68 | −35.3 | 0.70 | −40.6 | 0.88 | −76.7 | 0.96 | −91.6 | 0.96 | −92.3 | 0.99 | −97.0 |

M-H random effects | 0.68 | −35.3 | 0.70 | −40.6 | 0.88 | −76.7 | 0.96 | −92.5 | 0.97 | −93.0 | 0.99 | −97.3 |

M-H fixed effects | 0.68 | −35.3 | 0.70 | −40.6 | 0.88 | −76.7 | 0.96 | −91.6 | 0.96 | −92.3 | 0.99 | −97.0 |

Peto | 0.50 | −0.9 | 0.54 | −7.8 | 0.80 | −60.5 | 0.95 | −89.9 | 0.90 | −90.6 | 0.98 | −96.4 |

BA0E, both-armed zero-event; IV, inverse variance; M-H, Mantel-Haenszel.

Comparing root mean square error (RMSE). BA0E, both-armed zero-event; IV, inverse variance; M-H, Mantel-Haenszel; RMSE, root mean square error.

Comparing width of 95% confidence interval (CI). BA0E, both-armed zero-event; IV, inverse variance; M-H, Mantel-Haenszel.

Similarly, excluding BA0E studies for meta-analyses introduced little bias on the pooled estimates (0.7–1.4%) when there was no true treatment effect (

Among all five pooling methods, the Peto method excluding BA0E studies produced lowest bias across all simulation scenarios. When the treatment effect and between-study variance were from none (OR=1; SD=0.1) to moderately large (OR=2, OR=0.5; SD=0.5), with the reasonable number of patients in each study (>100), Peto method excluding BA0E studies generated the relatively reliable estimates of the pooled treatment effect (percentage bias<−8%). However, when the treatment effect and between-study variance were extremely large (OR=5, OR=0.2; SD=1) and number of patients in each study was small (<50), the bias of the estimates from this approach increased dramatically towards underestimating the treatment effects (

Our simulation study verified that when there was no true treatment effect (OR=1), the approach of including BA0E studies consistently outperformed the approach of excluding BA0E studies across all five pooling methods by providing nearly unbiased results. However, whenever a true treatment effect was present, the results from the approach of including BA0E studies introduced larger bias comparing to the approach of excluding them in the direction of underestimating the true treatment effect. Among all evaluated pooling methods for these two approaches, Peto methods excluding BA0E studies produced the least biased estimates when the true treatment effect existed.

This simulation study investigated the impact of including or excluding BA0E studies in meta-analyses for rare event outcomes when OR is used as the effect measure for pooled estimates of dichotomous outcomes. We found that including BA0E studies provided more accurate overall pooled estimates than excluding them when there was no true treatment effect. However, when there was a true treatment effect, the results from both approaches underestimated the true treatment effect, and including BA0E studies increased bias further in the direction of underestimating treatment effects. Among the pooling methods, Peto's method with exclusion of BA0E studies provided the pooled OR considerably closer to the true treatment effect for small to moderate treatment effects under the condition of small to moderate between-study variance and relatively large sample size.

Our simulation study confirmed the empirical findings obtained by Friedrich

This simulation study confirmed that among all five commonly used pooling methods, only the Peto method without inclusion of BA0E studies produces a pooled OR approaching the true treatment effect when sample size is relatively large. This finding is consistent with the simulation study conducted by Bradburn

This simulation study clearly showed that including both-armed (and even single armed) zero-event studies in meta-analysis could severely underestimate the treatment effects for beneficial and harmful events. However, when the treatment effect is evaluating harmful outcomes, underestimating treatment effect may lead to more serious consequence such as compromising patients' safety in seeking new treatment. In reality, it is not easy or sometimes even impossible to know whether a true treatment effect exists or not. Therefore, a comprehensive approach of a series of sensitivity analyses needs to be conducted when performing systematic reviews that include zero-event studies. An example could be used is Dahabreh and Economopoulos

Although we chose the values of simulation parameters from literature review, we realise that the results of our simulation study cannot be generalised to all situations in meta-analysis. To reduce the simulation scenarios to a manageable level, we used fixed values for some parameters. We only considered the balanced group ratio between treatment and control arms, but only 22% of RCTs used unbalanced design among previous in a recent review.

The commonly used meta-analysis pooling methods we discussed in this simulation are based on parameter estimation, which requires the use of continuity correction to include zero-events. Some likelihood maximisation-based statistical models such as Poisson regression and β-binomial regression can incorporate both-armed or single-armed zero-events without continuity correction and supposedly generates an unbiased estimate of RR. The simulation from Spittal

Strategies in dealing with BA0E studies

Approaches | Scenarios |
---|---|

Including BA0E studies |
No evidence of the presence of treatment effects Strong rationale on seeking the most conservative estimates of the treatment effect for beneficial outcomes when evaluating new drugs or interventions The magnitude of the treatment effects is unclear when evaluating beneficial outcomes |

Excluding BA0E studies with Peto method |
Evidence of the presence of treatment effects Evaluating harmful outcomes such as mortality, mobility or adverse events |

BA0E, both-armed zero-event.

To conclude, we recommend including BA0E studies in meta-analysis using OR as effect measure in the following three scenarios: (1) when treatment effects are unlikely to present, (2) having strong rational for seeking the most conservative estimates on treatment effect when evaluating beneficial outcomes and (3) magnitudes of the treatment effects unclear when evaluating beneficial outcomes for new treatments. We recommend excluding BA0E studies in conjunction of Peto method in the following two scenarios: (1) when treatment effects are likely to present and (2) when evaluating harmful outcomes such as mortality, mobility or adverse events. When the above recommendations cannot apply, it is important to present the results of meta-analyses using approaches of including and excluding BA0E studies so that the readers can make their informed interpretation within the clinical content.

JC, EP, JKM and LT designed the study. JC wrote the simulation programme, conducted the statistical analysis and drafted the manuscript. JC, EP, AI and LT provided input on statistical concept. JKM and AI provided the clinical expertise. All authors revised the manuscript for important clinical and statistical contents and approved the final manuscript.

This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

None declared.

Not commissioned; externally peer reviewed.