Introduction Overall, intra-arterial treatment (IAT) proved to be beneficial in patients with acute ischaemic stroke due to a proximal occlusion in the anterior circulation. However, heterogeneity in treatment benefit may be relevant for personalised clinical decision-making. Our aim is to improve selection of patients for IAT by predicting individual treatment benefit or harm.
Methods and analysis We will use data collected in the Multicenter Randomized Clinical Trial of Endovascular Treatment for Acute Ischemic Stroke in the Netherlands (MR CLEAN) trial to analyse the effect of baseline characteristics on outcome and treatment effect. A multivariable proportional odds model with interaction terms will be developed to predict the outcome for each individual patient, both with and without IAT. Model performance will be expressed as discrimination and calibration, after bootstrap resampling and shrinkage of regression coefficients, to correct for optimism. External validation will be conducted on data of patients in the Interventional Management of Stroke III trial (IMS III). Primary outcome will be the modified Rankin Scale (mRS) at 90 days after stroke.
Ethics and dissemination The proposed study will provide an internationally applicable clinical decision aid for IAT. Findings will be disseminated widely through peer-reviewed publications, conference presentations and in an online web application tool. Formal ethical approval was not required as primary data were already collected.
Trial registration numbers ISRCTN10888758; Post-results and NCT00359424; Post-resultsc.
Statistics from Altmetric.com
- ischaemic stroke
- intra-arterial therapy
- mechanical thrombectomy
- prediction model
- decision aid
- personalised treatment
Strengths and limitations of this study
Multiple characteristics will be evaluated simultaneously to show clinically relevant heterogeneity in treatment benefit between patients.
Multivariable prediction modelling substantially increases statistical power compared with other approaches and is more robust, especially in small data sets.
We will use a relatively small cohort for the development of a prediction model.
Using a proportional odds model requires the assumption that the ORs are the same for each cut-off of the modified Rankin Scale.
In 2015, five consecutive randomised controlled trials (RCTs) showed that intra-arterial treatment (IAT) improves functional outcome in patients with a proximal occlusion in the anterior circulation.1–6 This was a major breakthrough in the field, and IAT is now implemented in updated guidelines on acute ischaemic stroke (AIS) management.7
Ideally, IAT will be targeted at patients who are expected to have optimal benefit: personalised treatment. In this study protocol, we present seven steps for development and validation of a clinical decision aid to predict which individual patients with AIS will benefit most from IAT.8 ,9
Methods and analysis
Step 1: problem definition and data inspection
RCTs provide estimates of treatment effects for average patients. However, it is important to take potential heterogeneity of treatment effects into account. Clinically relevant differences in the absolute effect of a treatment can be caused by (1) differences in the relative treatment effect (predictive effects) and (2) differences in baseline risk on the outcome of interest (prognostic effects).10 ,11 For example, in the Multicenter Randomized Clinical Trial of Endovascular Treatment for Acute Ischemic Stroke in the Netherlands (MR CLEAN) trial, there is no predictive effect of age; the relative treatment effect is constant across age subgroups.1 This is demonstrated by a non-significant test for interaction between age and treatment (figure 1A). However, variation in baseline risk on favourable outcome according to age results in a larger absolute treatment benefit in younger patients (figure 1B).
Conventional subgroup analyses are focused mainly on predictive effects and assess the effect of only one variable at a time. If predictive and prognostic effects of multiple characteristics are evaluated simultaneously in multivariable prediction modelling, it is likely that larger heterogeneity in treatment benefit between individual patients will be found. Our aim is to improve selection of patients for IAT by predicting treatment benefit or harm for individual patients with stroke.
We will use data of the MR CLEAN trial (n=500), which was a phase 3 multicentre clinical trial with randomised treatment group assignment, open-label treatment and blinded end point evaluation. IAT plus usual care (which could include intravenous administration of alteplase) was compared with usual care alone. IAT consisted of arterial catheterisation with a microcatheter to the level of occlusion and delivery of a thrombolytic agent, mechanical thrombectomy or both.1
Severity of stroke was assessed at baseline with the National Institutes of Health Stroke Scale (NIHSS; range 0–42). Baseline CT was evaluated with the Alberta Stroke Program Early CT Score (ASPECTS; range 0–10). Baseline imaging (CT angiography) was used to determine the location of occlusion and to grade the quality of collateral flow to the ischaemic area with a four-point scale. Detailed information about the MR CLEAN trial can be found in the study protocol and the publication of the main results.1 ,12
End points of interest
The primary outcome will be the modified Rankin Scale (mRS), a seven-point scale ranging from 0 (no symptoms) to 6 (death) at 90 days after stroke.13 We will provide estimates of treatment benefit as the absolute increase in probability on functional independence (defined as mRS 0–2) and survival (defined as mRS 0–5).
Step 2: coding of variables
As variables, we will use patient characteristics that are expected to predict outcome, or that are expected to interact with treatment, based on expert opinion and the recent literature (table 1). Non-linearity of continuous variables will be tested by comparing the two log likelihood of models with linear and restricted cubic spline functions.14
Timing of treatment is an essential predictor of outcome. Since time to randomisation was not a reliable indicator for time to treatment in the MR CLEAN trial and will not be applicable in clinical practice, we will use time from stroke onset to groin puncture. Since time to groin puncture is not observable in the control group, we will explore imputation approaches based on the correlation with time to randomisation. All other baseline variable values are more than 98% complete in the MR CLEAN data, so we choose simple imputation by the mean for continuous variables and simple imputation by the mode for categorical variables.
Steps 3 and 4: model specification and estimation
We will test the effect of variables on functional outcome and treatment effect with proportional odds regression modelling. All variables from table 1 will be tested for effect on outcome and interaction with treatment effect. Prognostic variables (main effects) and predictive variables (interaction effects) with a p value of 0.15 in univariable and multivariable analyses will be included in our final model. A p value of 0.15 was chosen to make the predictor selection less data driven and prevent overfitting.14 ,31 We will perform shrinkage of all regression coefficients with ridge regression to prevent overfitting of the model.14 Predicted probabilities for each of the mRS categories, with and without treatment, will be derived from the ordinal model. All statistical analyses will be performed within the computing environment R V.3.2.2 (The R Foundation)
Step 5: model performance
Model performance will be expressed in discrimination and calibration. Discrimination will be quantified with the c-statistic. The c-statistic is similar to the area under the curve for binary outcomes and estimates the probability that out of two randomly chosen patients, the patient with the higher predicted probability of a good outcome will indeed have a better outcome. Calibration refers to the agreement between predicted and observed risks and will be assessed graphically with validation plots, and expressed as calibration slope and an intercept. The calibration slope describes the relative overall effect of the variables in the validation sample, and is ideally equal to 1. The intercept indicates whether predictions are systematically too high or too low, and should ideally be 0.32 We will calculate a general c-statistic to express the performance of our ordinal model and additional calibration plots with specific c-statistics for the predictions of favourable functional outcome (mRS 0–2) and survival (mRS 0–5).
Step 6: model validity
The c-statistic will be internally validated with a bootstrap procedure (500 samples with replacement) to estimate the degree of optimism in parameter estimates.8 After penalisation of the regression coefficients, we will externally validate the model on data of patients in the Interventional Management of Stroke III trial (IMS III) with an occlusion in the anterior circulation on non-invasive vessel imaging.33 Coefficients of the final model will be fitted on the combined development and validation data sets.
After validation, we will assess whether the model can be used to discriminate between patients with low and high expected benefit by making individual predictions of outcome for all patients included in the development and validation data.
Step 7: model presentation
The final model will be digitally available for use in clinical practice, both for mobile devices and as a web application. It will provide predictions of all mRS categories for each individual patient, both with and without IAT.
Ethics and dissemination
Findings will be disseminated widely through peer-reviewed publications, conference presentations and in an online web application tool. Formal ethical approval was not required for this study as primary data were already collected.
Compared with the current subgroup analyses on the effect of IAT, our modelling approach has multiple advantages. First, it accounts for the fact that patients have multiple characteristics that simultaneously affect the likelihood of treatment benefit.34 Thus, our model will show more clinically relevant heterogeneity in treatment benefit between patients. Second, a multivariable prediction model substantially increases statistical power to identify heterogeneity in treatment effects compared with other approaches.35 These include neural network and decision trees. We use regression modelling since it is considered more robust, especially in relatively small data sets.36 ,37
There are some differences between patients included in the MR CLEAN trial and the IMS III trial that may influence the external validity of our model. IMS III had different inclusion criteria, used older devices and older treatment paradigms than MR CLEAN. In order to overcome these limitations, we will use only those patients in IMS III with an occlusion in the intracranial anterior circulation on non-invasive vessel imaging. We will compare the baseline characteristics of the derivation and validation cohort and describe relevant differences that might lead to an underestimation or overestimation of the model performance. Interestingly, a substantial treatment effect in the IMS III patients with proven intracranial large vessel occlusion has been reported.38
Furthermore, even though the MR CLEAN trial has included most patients of the recent RCTs, the cohort remains relatively small for the development of a prediction model, especially for the selection of the main effect and interaction effects. We will reduce regression coefficients to prevent overfitting and also perform external validation. In the future, we will further validate and update our model in the pooled individual patient data of the Highly Effective Reperfusion evaluated in Multiple Endovascular Stroke Trials (HERMES) collaboration, harbouring data of all patients from recent randomised trials regarding IAT (over 1700 patients in total). Moreover, we aim to investigate the validity of our model predicting outcome after treatment in clinical practice. Our model will therefore be tested by applying it to recently treated patients in all Dutch neurovascular centres participating in the MR CLEAN Registry (mrclean-trial.org).
We will use a proportional odds model to analyse the full mRS score as outcome. Formally, this model requires the assumption that the ORs are the same for each cut-off of the mRS. However, previous studies have shown that even if the proportionality assumption is violated, proportional odds analysis is still more efficient than dichotomisation.39 In addition, all recent RCTs on the effect of IAT used the full mRS and analysed their results with proportional odds regression.
The proposed study will provide an internationally applicable clinical decision aid for the selection of patients for IAT. We consider this study an important next step towards personalised treatment of patients with AIS.
MJHLM and EV contributed equally.
Contributors MJHLM and EV were involved in literature search, study design, writing (authors contributed equally). BR and HFL were involved in study conception and design, and writing. EWS and DWJD were involved in study conception and design, and critical review of the manuscript. JPB, SDY, PK, OAB, YBWEMR, RJvO, WHvZ, CBLMM and AvdL were involved in study conception and critical review of the manuscript.
Competing interests Erasmus MC received funds from Stryker®, Bracco Imaging® for consultations by DWJD. AMC received funds from Stryker® for consultations by CBLMM, YBWEMR and OAB. MUMC received funds from Stryker® and Codman® for consultations by WHvZ. JPB received study medication for intra-arterial tissue-type plasminogen activator supplied by Genentech. Catheters were supplied during early years of the IMS III trial by EKOS Corp, Concentric Medical, Cordis Neurovascular. He currently receives research monies to Department of Neurology and Rehabilitation Medicine from Genentech for Role on Steering Committee for A Study of the Efficacy and Safety of Activase (Alteplase) in Patients With Mild Stroke (PRISMS) trial. SDY research monies from Genentech for statistical role in the PRISMS trial. PKs department of Neurology received research support from Genentech, Inc for her role as lead principal investigator (PI) of the PRISMS trial and Penumbra, Inc for her role as neurology PI of the Assess the Penumbra System in the Treatment of Acute Stroke (THERAPY) trial; she has also received royalties from UpToDate, Inc and provided consultation for Grand Rounds Experts, St Jude’s and Biogen (DSMB).
Ethics approval Medical and Ethical Review Committee Erasmus MC.
Provenance and peer review Not commissioned; externally peer reviewed.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.