Non-inferiority trials: are they inferior? A systematic review of reporting in major medical journals

Objective To assess the adequacy of reporting of non-inferiority trials alongside the consistency and utility of current recommended analyses and guidelines. Design Review of randomised clinical trials that used a non-inferiority design published between January 2010 and May 2015 in medical journals that had an impact factor >10 (JAMA Internal Medicine, Archives Internal Medicine, PLOS Medicine, Annals of Internal Medicine, BMJ, JAMA, Lancet and New England Journal of Medicine). Data sources Ovid (MEDLINE). Methods We searched for non-inferiority trials and assessed the following: choice of non-inferiority margin and justification of margin; power and significance level for sample size; patient population used and how this was defined; any missing data methods used and assumptions declared and any sensitivity analyses used. Results A total of 168 trial publications were included. Most trials concluded non-inferiority (132; 79%). The non-inferiority margin was reported for 98% (164), but less than half reported any justification for the margin (77; 46%). While most chose two different analyses (91; 54%) the most common being intention-to-treat (ITT) or modified ITT and per-protocol, a large number of articles only chose to conduct and report one analysis (65; 39%), most commonly the ITT analysis. There was lack of clarity or inconsistency between the type I error rate and corresponding CIs for 73 (43%) articles. Missing data were rarely considered with (99; 59%) not declaring whether imputation techniques were used. Conclusions Reporting and conduct of non-inferiority trials is inconsistent and does not follow the recommendations in available statistical guidelines, which are not wholly consistent themselves. Authors should clearly describe the methods used and provide clear descriptions of and justifications for their design and primary analysis. Failure to do this risks misleading conclusions being drawn, with consequent effects on clinical practice.


For all articles
There were 12(7%) justifications classed as "other":  Based on previous trial. No evidence for consultation with external expert group, and no reference to previous trial of the control arm  Based on unpublished data. No evidence for consultation with external expert group, and no reference to previous trials of the control arm  Clinical basis and based on previous trials and guidelines. No evidence for consultation with external expert group, and no reference to previous trials of the control arm  Clinical basis. Attempted to justify based on preservation of treatment effect, but were unable to do so due to paucity of previous trials.  Expert group external to the authors and previous trial. No reference to previous trial of the control arm  Justified based on treatment effect of control, but margin actually bigger than control arm treatment effect  Placebo controlled study. Clinical basis, previous trials and literature review  Preservation of treatment effect. Reference to separate paper justifying margin  Regulatory guidelines (WHO), but recommendation is for superiority. No evidence for consultation with external expert group, and no reference to previous trials of the control arm  Synthesis approach  Unclear

NEJM protocols
There were 6 (10%) justifications classed as "other":  Based on previous trial. No evidence for consultation with external expert group, and no reference to previous trial of the control arm  General comment that margin was decided according to FDA request  Justified based on treatment effect of control, but margin actually bigger than control arm treatment effect  Preservation of treatment effect based on estimates of control arm effect from previous trials and clinical basis  Preservation of treatment effect based on estimates of control arm effect from previous trials, clinical basis and according to FDA guidelines  Preservation of treatment effect. Reference to separate paper justifying margin

Figure 1a: Chosen analysis by primary or secondary analysis
NB: One study performed ITT and PP analyses but it was unclear which of the two was taken as primary and secondary all patients randomised who received at least one dose of treatment/intervention 21 (16%) All patients randomised excluding missing data 7 (5%) All patients randomised excluding errors in randomisation 3 (2%) All patients randomised who received at least one dose of treatment/intervention, excluding missing data 1 (1%) All patients randomised with exclusions from one centre which was removed due to misconduct 1 (1%) Other 17 (13%) Unclear 1 (1%) Not defined 10 (8%)

PP 90
Patients who received allocated treatment/intervention 8 (9%) Excluding patients with major protocol violations 5 (6%) Patients who completed allocated treatment/intervention as intended 4 (4%) Patients who adhered to treatment 2 (2%) Excluding patients with protocol deviations 2 (2%) Patients with no exclusion criteria and who received specific amount of treatment/intervention 2 (2%) Patients who received allocated treatment/intervention, no major protocol violations with outcome 2 (2%) Excluding patients who switched treatment 1 (1%) Patients who received at least one dose of treatment/intervention 1 (1%) Patients who adhered to the protocol 1 (1%) Patients who completed the assigned study regimen or adhered to treatment before an event 1 (1%) Patients who received correctly allocated treatment/intervention excluding withdrawals 1 (1%) Patients who received specific amount of treatment/intervention and adhered to protocol 1 (1%) Patients who received allocated treatment/intervention, excluding non-adherence 1 (1%) Patients who adhered to protocol excluding withdrawals 1 (1%) Excluded patients with protocol deviations in addition to mITT definition 1 (1%) excluded patients that received rescue medication and protocol violations 1 (1%) Patients who received at least one dose of drug/intervention and received allocated treatment/intervention excluding missing outcome data 1 (1%) All patients who received at least one dose of treatment/intervention and did not have major protocol violations and were followed for event while receiving drug 1 (1%) All patients who received at least one dose of treatment/intervention and did not have major protocol violations 1 (1%) Excluding patients who were ineligibile, excluding patients who were administered the incorrect dose of medication and excluding patients who were allocated the incorrect treatment 1 (1%) All patients randomised who received at least one dose of treatment/intervention with an outcome, completed the study and complied with protocol 1 (1%) Non-adherence, patients who declined follow up, errors in randomisation, recurrent atrial fibrillation before randomisation were excluded 1 (1%) The per-protocol population (which consisted of the modified intention-to-treat population with the exclusion of patients with major protocol deviations and a compliance rate of <80%) was of primary interest, since a noninferiority analysis that is based on the modified intention-to-treat population is deemed to be not conservative Patients were not eligible for per-protocol analysis for the following reasons: no follow-up visit; systemic treatment with other antimicrobial drugs up to day 28 (visit three); or missing more than one dose of the study drug during the first week of treatment or more than two doses during the whole treatment period 1 (1%) Excluded missing inclusion criteria; incorrect dosing; received prohibited medication; missing assessments 1 (1%) Per-protocol analyses excluded participants who had missing data at 1 month or who had major protocol violations (e.g., death, pregnancy, withdrawal from the study, loss to follow-up, or noncompliance). NB: Two results were presented for PP where compliance was included and excluded.

(1%)
Per-protocol prespecified analyses included children with complete follow-up or a confirmed treatment failure, and excluded those treated for malaria without confirmatory microscopy, those for whom the alternative Plasmodium species was detected, and those who defaulted from follow-up despite repeated attempts at contact Flow chart includes: "and followed protocol" Patients who, during the intended treatment period, had a venogram adjudicated as assessable, who developed confirmed deep vein thrombosis or pulmonary embolism, or who died from any cause); patients who had important protocol violations were excluded from the per-protocol analysis.

(1%)
The per-protocol population was defined as all patients included in the ITT analysis, excluding those who did not receive the regimen as prescribed. These were patients who received less than 6 weeks of treatment (42 days of daily treatment or 36 days of 6-days-a-week treatment) or more than 9 weeks of treatment (63 days of daily treatment or 54 days of 6-days-a-week treatment) in the intensive phase and those who received less than 42 doses (ie, 4 weeks of missed treatment) or more than 60 doses (ie, 2 weeks of extra treatment) in the continuation phase (the protocol requirement is that patients receive 18 weeks of 3-times-weekly treatment, ie, 54 doses). Also excluded were patients whose treatment was modified for reasons other than bacteriological failure or relapse (including patients changing treatment for adverse drug reactions, following return after default, or attributable to concomitant HIV infection).

(1%)
Per-protocol snapshot analysis, which included all participants who were enrolled, received at least one dose of study drug, and did not meet any of the following prespecifi ed criteria: discontinuation of study drug before week 48 or HIV RNA data missing in week 48 analysis window (accounting for 80% of excluded patients), and adherence in the bottom 2.5th percentile (accounting for 20% of the excluded patients) The perprotocol group consisted of all patients who were enrolled, had no major protocol deviation, received the full treatment, and were assessed at day 15 or 31, 1 (1%) day 45, and 6 months (-2 to +6 weeks).
Criteria to exclude patients from this set were violation of major in-or exclusion criteria, change of treatment arm, early treatment discontinuation or relevant dose deviations of chemo-or radiotherapy unless caused by death or progression, radiotherapy without PET panel recommendation or omission of radiotherapy against recommendation, PET panel decision to take the patient off protocol treatment, or missing documentation of treatment 1 (1%) The per-protocol analysis set additionally excludes patients with change of treatment arm, early treatment discontinuation or relevant dose deviations of chemo-or radiotherapy unless caused by death or progression, or missing documentation of treatment 1 (1%) The perprotocol analysis was based on all participants who received 3 doses of vaccine according to 1 of the study's vaccine dosing schedules, were seronegative to the relevant HPV type at baseline, and had a valid serology result after the third dose of the HPV vaccine 1 (1%) Not defined. Taken from flow chart: Patients not meeting the definition of having received adequate treatment provided they have not already had an unfavourable response to treatment. Other exclusions done as well, but are not defined in flow chart 1 (1%) All patients who underwent randomization, completed a full treatment course or had early treatment failure before treatment was completed, had outcome data for the primary efficacy end point on day 28, and complied with the protocol to the extent that would allow efficacy evaluation 1 (1%) We also conducted a perprotocol analysis, which included those who completed the 2-month visit while receiving treatment (108 oral, 113 intratympanic) because intention-to-treat analyses may bias toward noninferiority. Flow chart also shows patients who withdrew before the 2m follow up, those who discontinued treatment but completed follow up and those who completed treatment but missed 2m follow up were excluded.

(1%)
Which consisted of participants who received all three doses of vaccine within 1 year, did not have the HPV type being analyzed (i.e., were seronegative on day 1 and PCR-negative from day 1 through month 7), and had no protocol violations 1 (1%) A total of 12 (10%) patients in each group did not undergo PEG for anatomical reasons. Between the PEG procedure and the follow-up visit, five patients died, one patient pulled out the PEG catheter without ensuing complications, three patients were lost to follow-up, and one patient who was randomised to cefuroxime received co-trimoxazole instead.

(1%)
Will include all subjects in the MITT population grouped by randomized treatment assignment regardless of treatment received with the exception of the following additional exclusions 1. Subjects not meeting the definition of having received an adequate amount of their allocated study regimen (see below for definition), provided they have not already been classified as having an unfavourable outcome 2. Subjects lost to follow-up or withdrawn before the Month 6 visit, unless they have already been classified as having an unfavourable outcome. 3. Subjects whose treatment was modified or extended for reasons (e.g. an adverse drug reaction or pregnancy) other than an unfavourable therapeutic response to treatment, unless they have already been classified as having an unfavourable outcome 4. Subjects who are classified as "major protocol violations"• (see section 6.5), unless they have already been classified as having an unfavourable outcome on the basis of data obtained prior to the protocol violation 1 (1%) The per-protocol analysis excluding the 6 patients who were lost to follow-up and the 3 patients who received postoperative corticosteroids (including the 4 patients who experienced primary bleeding events)

(1%)
Excluded patients who received a platelet transfusion for reasons not recommended in the protocol 1 (1%) We also did a per-protocol analysis of the medical outcomes, excluding outpatients discharged more than 24 h after randomisation and inpatients discharged 24 h or less after randomisation.

(1%)
Not defined. Flow chart shows the following exclusions: had another histology or malignancy; withdrew informed consent; had an allergic reaction on first rituximab infusion and consecutively other treatment; only had radiotherapy; received incorrectly allocated treatment; did not meet inclusion or exclusion criteria; no therapy; death before therapy 1 (1%) Not defined. Flow chart suggests patients were excluded if they did not receive the protocol and withdrawals 1 (1%) Censoring of events if any component of the initial randomised trial treatment was stopped 1 (1%) Not defined. Flow chart shows inclusion/exclusion criteria violated, nonadherence, prohibited medication and missing results were excluded 1 (1%) Participants who did not follow protocol and/or were seropositive or polymerase chain reaction-positive for HPV-16, HPV-18, HPV-6, or HPV-11 at enrolment were excluded from the per-protocol population analysis but retained for the intentionto-treat population analysis. Participants were eligible to continue with the 18-and 36-month follow-up if they had all of their doses of vaccine and a 7-month blood sample collected. If participants were excluded from the per-protocol population analysis at 7 months, they remained excluded for the remainder of the study but were retained for intention-to-treat analysis.

(1%)
The per-protocol population included all patients who completed the study (1 year), and for whom the second reading of a CT-scan confirmed the diagnosis of uncomplicated appendicitis.

(1%)
For analyses based on the per-protocol population, patients were analysed according to their randomly assigned treatment group. To be included in the perprotocol population, a patient was required to meet the following criteria: Had a mean baseline hemoglobin ≥ 8.0 and < 11.0 g/dl; Completed the study through at least week 36, and at least 5 hemoglobin values were obtained during the evaluation period; Had no missing administrations of study medication between weeks 21 and 35, inclusive; Had not received any RBC or whole blood transfusions within the 12 weeks prior to randomization; Had not received any RBC or whole blood transfusions for reasons other than lack of effect of study medication (lack of effect of study medication was documented as "Anemia of CRF"• on the case report form) between weeks 21 and 35, inclusive; Had not received any ESA other than the assigned study treatment between weeks 21 and 35, inclusive; Had adequate iron status at baseline and during the evaluation period (defined as serum ferritin ≥100 ng/ml and TSAT ≥ 20% during weeks 24, 28, and 32) 1 (1%) Not defined. Flow chart shows exclusions: caesarean section or forceps; short umbilical cord or nuchal cord; need for resuscitation; team became unavailable; weight scale malfunctioned; parent withdrew consent 1 (1%) Completers (observed cases; included patients in the full analysis set who did not have important protocol violations, completed at least 684 days of treatment, and had HbA1c measured at week 104)

(1%)
For analyses based on the per-protocol population, patients were analyzed according to their randomly assigned treatment group. To be included in the perprotocol population, a patient was required to meet the following criteria: Had a mean baseline hemoglobin ≥ 10.0 and ≤ 12.0 g/dl; Completed the study through at least week 36, and at least six haemoglobin values were obtained during the 1 (1%) evaluation period.; Received ≥ 75% of total prescribed (i.e., expected) doses of study medication between weeks 25 and 35, inclusive (detailed algorithms for this determination were specified in the Statistical Analysis Plan).; Had not received any RBC transfusions within the 12 weeks prior to randomization.; Had not received any RBC transfusions for reasons other than lack of effect of study medication (lack of effect of study medication was documented as "Anemia of CRF" on the case report form) between weeks 25 and 36, inclusive.; Had not received any ESA other than the assigned study treatment between weeks 25 and 35, inclusive.; Had adequate iron status at baseline and at week 36 (defined as serum ferritin ≥ 100 ng/ml and TSAT ≥ 20%). This population included all patients who underwent randomisation and who completed the study procedures to month 6. 1 (1%) We also performed a per-protocol analysis, which notably excluded patients in the antibiotic group who had been switched from amoxicillin plus clavulanic acid to another antibiotic.

(1%)
We did a per-protocol snapshot analysis, which included all participants who were randomly assigned treatment, received at least one dose of study drug, and did not meet any of the following prespecifi ed criteria: discontinuation of study drug before week 48 or HIV RNA results missing in the week 48 analysis window, and adherence in the bottom 2.5th percentile.

(1%)
Patients were included in the per-protocol population if they met the criteria for inclusion in the modified intention-to-treat population, underwent an adequate assessment of venous thromboembolism not later than 2 days after administration of the last dose of study drug, and had no major protocol violations.

(1%)
The perprotocol population comprised patients in the modified intention-to-treat group who received treatment for at least 3 days (in the case of patients with treatment failure) or at least 8 days (in the case of patients with clinical cure), had documented adherence to the protocol, and underwent an end-of-therapy evaluation.

(1%)
The per-protocol analysis set consisted of participants with exposure to treatment for at least 12 weeks who did not have any major protocol violations that could affect the primary endpoint and had a valid glycated haemoglobin (HbA1c) assessment at baseline and at (or after) 12 weeks.

(1%)
Not defined 11 (12%) mITT 34 All patients randomised who received at least one dose of treatment/intervention 10 (29%) All patients randomised who received at least one dose of treatment/intervention, excluding missing data 6 (18%) All patients randomised with at least one dose of treatment/intervention excluding patients/site with violations of GCP 2 (6%) All randomised patients who received at least one dose of treatment/intervention excluding patients without disease or excluding patients resistant to one of the drug combinations. Excluding patients whose death was not related to the disease or had reinfection after being cured or patients who were classed as unassessable at the endpoint 1 (3%) Patients were excluded if they were resistant to two of the treatment combinations and patients who were unassessable and had not reached endpoint 1 (3%) On-treatment which included events that occurred within 30 days after the last dose of study medication was administered 1 (3%) Patients were excluded if they had missing/contaminated outcome data or could not produce an assessment or were lost to follow up or had death not related to disease or had confirmed reinfection 1 (3%) Excluded if consent withdrawn, non-compliance, moved and other (other not defined) 1 (3%) Other 11 (32%)

Study conclusions
Of the articles that were designed as non-inferiority trials, two articles stated the trial was noninferiority, but had drawn equivalence graphs with two margins; one article stated the trial was for non-inferiority but states the sample size calculation is to determine equivalence; one article concluded that their study did not show equivalence; one concluded equivalence; one article stated that the margin was an equivalence margin; one stated that they would test for equivalence; one concluded non-inferiority as the confidence interval was within ±margin; one concluded equivalence in the abstract but non-inferiority in the main paper; one stated that "results were consistent with showing non-inferiority (i.e. equivalence)".