Introduction Treatment decisions for aneurysmal subarachnoid haemorrhage patients should be supported by individualised predictions of the effects of aneurysm treatment. We present a study protocol and analysis plan for the development and external validation of models to predict benefit of neurosurgical versus endovascular aneurysm treatment on functional outcome and durability of treatment.
Methods and analysis We will use data from the International Subarachnoid Aneurysm Trial for model development. The outcomes are functional outcome, measured with modified Rankin Scale at 12 months, and any retreatment or rebleed of the target aneurysm during follow-up. We will develop an ordinal logistic regression model and Cox regression model, considering age, World Federation of Neurological Surgeons grade, Fisher grade, vasospasm at presentation, aneurysm lumen size, aneurysm neck size, aneurysm location and time-to-aneurysm-treatment as predictors. We will test for interactions with treatment and with baseline risk and derive individualised predicted probabilities of treatment benefit. A benefit of ≥5% will be considered clinically relevant. Discriminative performance of the outcome predictions will be assessed with the c-statistic. Calibration will be assessed with calibration plots. Discriminative performance of the benefit predictions will be assessed with the c-for benefit. We will assess internal validity with bootstrapping and external validity with leave-one-out internal-external cross-validation.
Ethics and dissemination The medical ethical research committee of the Erasmus MC University Medical Center Rotterdam approved the study protocol under the exemption category and waived the need for written informed consent (MEC-2020-0810). We will disseminate our results through an open-access peer-reviewed scientific publication and with a web-based clinical prediction tool.
- PUBLIC HEALTH
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.
STRENGTHS AND LIMITATIONS OF THIS STUDY
Use of a landmark trial for model development.
Modelling for heterogeneity of treatment effect.
Providing individualised estimates of functional outcome and durability of treatment and integrating these in a web-based clinical prediction tool.
The proposed tool could be used for individualised aneurysm treatment decision making.
Stringent selection criteria used in the trial used for model development may limit the generalisability and transportability of the study findings.
In the past decade, trial evidence showed that, in patients with aneurysmal subarachnoid haemorrhage (aSAH), endovascular aneurysm treatment leads to improved functional outcome in comparison to neurosurgical aneurysm treatment.1–4 Because of this, it is customary to provide endovascular aneurysm treatment when patients are, in the perception of the clinicians, equally eligible for both treatment approaches.5–7 This principle is referred to as ‘treatment equipoise’.
However, long-term follow-up revealed that patients who underwent neurosurgical aneurysm treatment had a higher degree of aneurysm occlusion, and lower rates of rebleeding and retreatment of the target aneurysm.2 8 There is a trade-off between short-term to medium-term expected functional outcome and the long-term risk of complications related to rebleeding and retreatment. In the International Subarachnoid Aneurysm Trial (ISAT), on average, the excess retreatment and rebleeding following endovascular treatment did not lead to a worse functional outcome at longest follow-up.9
However, evidence from randomised controlled trials (RCTs) applies to the population as a whole. Ideally, treatment effects are estimated for the individual patient. To assess individual patient treatment benefit it is necessary to model for heterogeneity of treatment effect. This means that the direction and magnitude of the treatment effect can vary depending on patient characteristics.10
Clinical prediction models accounting for this heterogeneity can enable personalised decision making and lead to improved patient outcome. For aSAH patients this could mean, weighing the individualised risk of rebleeding and retreatment against the individualised probability of favourable functional outcome. We present a study protocol for a study aiming to develop a clinical prediction tool to predict benefit of endovascular and neurosurgical aneurysm treatment in terms of functional outcome and durability of aneurysm treatment in individual patients with aSAH.
Methods and analysis
We will use data from the ISAT trial for model development. The ISAT trial was an international multicentre RCT that included 2143 patients with aSAH.11 The ISAT trial aimed to investigate the safety and efficacy of neurosurgical versus endovascular aneurysm treatment for patients with aSAH. Patient eligibility was based on the treatment equipoise policy. Patients were randomly assigned to neurosurgical aneurysm treatment or endovascular aneurysm treatment in a 1:1 ratio with a 24-hour telephone randomisation service. Detailed information about the study protocol can be found elsewhere.12
An advantage of using trial data for development of a prediction model is that the data are carefully and prospectively collected with a generally well-defined study population. Too stringent selection criteria may, however, limit generalisability.13 Since ISAT was published there has been extensive debate regarding the generalisability of the study population.14 15 Because of the treatment equipoise policy in the ISAT study, 80% of the initially screened patients were excluded.14 However, in this study, we specifically target the remaining 20%.
Ultimately, in ISAT, there was an underrepresentation of elderly and poor-grade patients, as well as aneurysms located at the middle cerebral artery or in the posterior circulation. Also, aneurysms in the ISAT study population were smaller. This could lead to increased uncertainty on the effect of a predictor with fewer observations.
Outcomes of interest
The outcomes of interest are the modified Rankin Scale (mRS) score at 12 months and any rebleed or retreatment of the target aneurysm after aneurysm treatment during follow-up. The mRS is a seven-point scale ranging from 0—no symptoms to 6—death.16 In ISAT the mRS-scores were collected with a standardized postal questionnaire.17 For presentation purposes, favourable functional outcome will be defined as mRS 0–2. The total duration of follow-up of the ISAT trial was 18 years.
We will define rebleed as any clinically or radiologically confirmed SAH after the first (partial) occlusion of the aneurysm. Retreatment will be defined as any endovascular or neurosurgical reintervention of the target aneurysm. The target aneurysm will be defined as the aneurysm which was identified as the origin of SAH and subsequently treated. If a patient was retreated because of a rebleed we will consider this a ‘rebleed’. Cross-over or a second treatment attempt after initial failed treatment without (partial) occlusion will not be considered retreatment.
All patients in the development cohort are eligible for inclusion in the model predicting functional outcome. In the model predicting any rebleed or retreatment during follow-up, we will exclude patients that have not had aneurysm treatment.
Potential predictors are selected based on clinical expertise and literature review. To fit the purpose of guiding aneurysm treatment decision making, we will only consider predictors that are available during early admission in a standard clinical setting. For both models, we consider age, World Federation of Neurological Surgeons (WFNS) grade, Fisher grade, vasospasm at presentation, aneurysm lumen size, aneurysm neck size, aneurysm location, aneurysm treatment and time-to-aneurysm treatment.
Aneurysm lumen size will be defined as the maximum lumen size of the aneurysm dome. Aneurysm location will be categorised as anterior cerebral artery, anterior communicating artery, middle cerebral artery, posterior communicating artery, internal carotid artery and other posterior circulation aneurysms. Aneurysm treatment will be entered into the model as the allocated or assigned treatment. Vasospasm at presentation will be dichotomised into present or absent.
We will use multiple imputation to account for missing data (table 1). The proportion of missing data in ISAT was negligible. We assume that data are missing at random. The imputation model will contain the predictors and the outcomes, with the addition of sex. We will inspect patterns of missingness and assess the imputed data for adequacy.
Possibly, patients that did not receive aneurysm treatment may have had an unfavourable prognosis (not justifying further treatment) or died beforehand. Because of this, we anticipate missing values for the time-to-aneurysm-treatment variable. In the model predicting durability of treatment, we will exclude patients without time-to-aneurysm-treatment because the model will only be used for patients that will receive aneurysm treatment.
However, in the model predicting functional outcome, this will lead to selection bias. Additionally, we cannot perform multiple imputation because time-to-aneurysm treatment is missing-not-at-random (ie, missingness is related to the outcome). To account for this, we will truncate time-to-aneurysm treatment at the 95th percentile. We will assign the value of the 95th percentile to patients that for whatever reason did not receive aneurysm treatment. This approach may lead to a (slight) overestimation of the effect size of time-to-aneurysm treatment.
Model specification and estimation
We will use ordinal logistic regression to develop a model for the mRS. Effect size estimates will be expressed as common ORs with 95% CIs. We will use Cox regression to develop a model for the time-to-event outcomes. Censoring occurs when patients are lost to follow-up or in case of death. Effect size estimates will be expressed as hazard ratios with 95% CIs.
To reduce the full model to the preliminary main effects model we will eliminate all predictors with a significance level above the threshold of p>0.20, and assess the changes in the remaining coefficients. The potential non-linearity of continuous predictors will be assessed by likelihood ratio tests (LRTs) of restricted cubic splines. We will also use LRTs to assess interaction with treatment of predictors and of baseline risk. We will consider interaction with treatment for: age, vasospasm at presentation, aneurysm lumen size, aneurysm location, aneurysm neck size and time-to-aneurysm treatment. If the omnibus LRT indicates additivity, the individual predictors will be tested one by one with a more stringent p<0.01, to avoid overfitting. Interactions with treatment of continuous predictors and baseline risk will also be assessed non-linearly. We will apply a threshold of p>0.05 for non-linearity. We will take several other measures to prevent overfitting. First, all predictors are preselected based on clinical knowledge and expertise. Next, we apply lenient p value to select predictors for the preliminary main effects models. Last, we will be parsimonious and test only those for interaction with treatment of predictors that are clinically plausible. We will comply with the PATH statement in modelling for heterogeneity in treatment effect.10 18
All statistical analyses will be performed with R software (V.4.1.1, R Foundation for Statistical Computing) using the rms (V.6.2.0), Hmisc (V.4.5.0), survival (V.3.3.1) and mice (V. 3.13.0) packages.
Benefit of treatment
We will derive predicted probabilities of favourable outcome at 12 months and of any retreatment or rebleed within 10 years follow-up for patients with aneurysms treated endovascular and neurosurgical. Treatment benefit will be defined as the absolute difference between the predicted probability of favourable functional outcome, and the predicted probability of retreatment or rebleed, with endovascular and neurosurgical aneurysm treatment. A benefit of ≥5% will be considered clinically relevant.
We will assess model performance in terms of discrimination and calibration. Prediction models need to discriminate between patients who experience the event and patients who do not. Furthermore, the model must have accurate risk estimates—the ratio between the predicted and the observed events for an ordinal outcome, or time to event for survival data.19 20 We will assess performance of the outcome predictions with the c-statistic and with calibration plots.19 To assess the performance of the benefit predictions we will use the c-for-benefit.21 22
We consider rebleed and retreatment as markers of revascularisation of the aneurysm and assume that predictors of rebleed and retreatment are equal. We will test this assumption by performing a sensitivity analysis. We will rerun the model with separate outcomes and evaluate the discriminative performance.
To assess internal validity, we will use bootstrapping.23 We will draw 200 random samples from the study population and analyse them as if they were an original sample. By subtracting the difference in performance, or mean optimism estimate, between the bootstrap and original sample, we obtain the optimism-corrected performance estimates.24 The final coefficients will be shrunk with penalised regression.
External validation is an underappreciated step in prediction modelling, and it has led to a sprawl of prediction models that are of low quality and sparsely used in the clinical context. Ideally, external validation of a model predicting treatment benefit is performed with randomised data.10 Besides the ISAT trial, at present, three trials investigated the safety and efficacy of endovascular versus neurosurgical aneurysm treatment.1 3 4 Taking into account the sample size and the need for long-term follow-up only the BRAT trial is eligible. At present, we do not have access to these data. Therefore, we will use leave-one-out internal-external cross-validation to assess external validity. Generalisability may be affected due to technological improvement or increased experience in endovascular techniques, and other supportive treatments. Possibly due to these improvements, since the publication of ISAT, the rates of retreatment and rebleeding of the target aneurysm have decreased.25
Sample size calculation
Many prediction models are underpowered for the number of parameters in the model.26 We used the pmsampsize package (V.1.1.12) to calculate the required sample size for the Cox model.27 28 Based on the number of considered parameters,20 the event rate of rebleed or retreatment (estimated at 0.05 per year), and the estimated r2 value based on previous models (30%), the total required sample size is 494 patients. In the development cohort, we have 2143 patients, meaning that our sample size is sufficient for reliable modelling. Because no similar tool exists for a model with an ordinal outcome, we apply the rule of thumb of a minimum of 10 events per variable,29–32 which would theoretically allow us to test for ≈ 200 parameters. For external validation, a minimum sample size of at least 100 events and non-events is proposed.33
Patient and public involvement
Ethics and dissemination
The medical ethical research committee of the Erasmus MC University Medical Center Rotterdam approved the study protocol under the exemption category and waived the need for written informed consent (MEC-2020-0810). We plan to disseminate our results through an open-access publication in a peer-reviewed scientific journal and conference presentations. We will adhere to the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) statement, a 22-item reporting checklist for prediction modelling studies.34 The R code of the models will be made publicly available for transparency and to enhance future external validation and model updating efforts. The R code will be accessible via: https://github.com/WinkelJordi/SHARP. The data needed to conduct this study has been received and is prepared for analysis. We anticipate finishing the analysis and ready the manuscript for submission no later than 1 August 2023.
The developed models will be integrated into a web-based clinical prediction tool. The web-based clinical prediction tool will be developed using the Shiny package (V.1.7.0). This tool will provide absolute estimates, based on baseline patient characteristics, of benefit of treatment in terms of functional outcome and durability of treatment. In the future, this tool could potentially be used to choose the optimal treatment strategy, maximising favourable functional outcome and durability of treatment. Previously, a similar tool has been proposed for intra-arterial treatment for acute ischaemic stroke.35 The proposed study will provide much-needed individually tailored evidence in the long-lasting discussion of neurosurgical versus endovascular aneurysm treatment. We believe that this study will prove to be an important addition to personalised medicine in the field of aSAH.
Patient consent for publication
Twitter @WinkelJordi, @mvanderjagt1, @DavidKlaveren, @hesterlingsma
Contributors Study conceptualisation (SAD, DWJD, DvK, HFL, BR and JdW); literature review (SAD and JdW); formal analysis (JdW); visualisation (JdW); writing original draft (JdW); critical review of manuscript (RD, SAD, DWJD, P-JvD, MvdJ, DvK, HFL, BR and JdW); supervision (HFL and BR).
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.