Article Text
PDF
Abstract
Introduction Endovascular aneurysm repair (EVAR) has a marked short-term advantage over open surgical repair in managing abdominal aortic aneurysms (AAA); however, this benefit is lost in the long term. The current trend towards stratified medicine has given rise to diverse prognostic prediction models and scoring systems for EVAR. These models could act as decision support tools that employ patient and operative factors, to improve long-term outcomes. Past literature evaluated and compared model performance for predicting one outcome, for example, mortality. None were deemed competent for clinical application. The proposed study will use a scoping review approach to capture literature on prognostic modelling in EVAR for all predictable outcomes. The results are anticipated to inform future research, identify knowledge gaps, and assist in determining the potential of models for clinical use.
Methods and analysis The proposed study will use the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) extension for scoping Reviews as a framework for conducting the review. PubMed Central, Embase and Cochrane Library will be searched and screened for peer-reviewed studies on prognostic modelling for EVAR, published between 2000 and 2022. No limits exist on predictor variables used and outcomes predicted by the model for inclusion, provided they apply to AAA patients managed with EVAR. Data will be abstracted using a charting form based on the Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies guidelines and PRISMA guidelines for systematic reviews. The Prediction model Risk of Bias Assessment Tool and the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis checklist will be used to critically appraise included studies.
Ethics and dissemination Since scoping reviews cover secondary data from published literature, ethical approval is not required. The findings will be disseminated via peer-reviewed publications and presentations at key conferences.
- vascular surgery
- Statistics & research methods
- Vascular surgery
- Risk management
- Protocols & guidelines
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
STRENGTHS AND LIMITATIONS OF THIS STUDY
This study will adhere to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines for scoping reviews to maintain methodological rigour.
Although optional in scoping reviews, a critical appraisal of the included studies will be performed to assess the risk of bias and applicability of model studies using the Prediction model Risk of Bias Assessment Tool and completeness of reporting using the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis checklist.
The outcome predicted by the model is open to inclusion, unlike previous systematic reviews which only included studies predicting one specific outcome (eg, postoperative mortality).
No meta-analysis will be performed on the included studies.
Introduction
Endovascular aneurysm repair (EVAR) has superseded open surgical repair (OSR) in recent years as the operative modality of choice for managing abdominal aortic aneurysms (AAA). According to the National Vascular Registry annual report 2021, EVAR made up 59.6% of the surgical repairs performed for all infra-renal AAA in the UK.1 Randomised control trials have consistently shown EVAR to have markedly reduced short-term mortality and morbidity compared with OSR, particularly in older patients with comorbid disease.2–4 However, the benefits of EVAR are lost in the long term as mortality rates converge with OSR within 2–3 years and high reintervention rates become a significant cause of morbidity.5–8 Postoperative follow-up plays a role in decreasing mortality and morbidity long term, but population studies have shown that complying with recommended protocols for post-EVAR surveillance provided no statistically significant survival benefit.9 10 This is consistent with the results of a 2018 systematic review and meta-analysis, which additionally found no benefit in all-cause mortality as well as higher reintervention rates for patients that were compliant with surveillance guidelines.11 These findings underscore the need for more refined criteria to guide clinical decision-making, to improve the outcomes of EVAR long term.
There has been a growing interest in the development of prediction models to address these challenges. Clinical prediction models use multiple variables (predictors) to estimate either an existing outcome (diagnostic model) or a future outcome (prognostic model). Prognostic prediction models could be employed as stratified medicine tools to support decision-making on suitability for endovascular repair, controlling procedural complications and post-EVAR surveillance requirements. Despite their potential utility, their use in practice is currently limited. The Society for Vascular Surgery (SVS) practice guidelines reference the Vascular Study Group of New England (VSGNE) risk model, which uses anatomical features such as aneurysm diameter, neck length and level of clamp placement to calculate perioperative mortality. Although the model was validated and endorsed by the Vascular Quality Initiative database, the guidelines gave it a recommendation level of ‘2’ (weak) and graded the quality of evidence as ‘C’ (low).12 A 2017 systematic review on prediction models in EVAR compared the performance of 13 prediction models for mortality after EVAR.13 The authors found that the British Aneurysm Repair score and the Vascular Biochemistry and Hematology Outcome Model came out on top, even outperforming the VSGNE risk model. However, they reported that both these models lacked enough validation to be used in clinical practice. The study also highlighted that although there were many new models, the existing models were not getting updated, nor were they externally validated well enough for generalised use. Another recent paper systematically reviewed 29 studies on predictors of reinterventions after EVAR to guide risk-stratified surveillance for patients.14 The study reported that models had a critical weakness in their development process, in that most were based on retrospective studies rather than prospective studies. It similarly concluded that existing models needed to be subjected to more external validation to be used for stratified surveillance.
Due to several gaps in the existing literature, it is difficult to assess whether EVAR prognostic models can currently make the transition into clinical practice. Systematic reviews on the subject have been performed; however, the most recent papers were published in 2017 and only included data up until 2015. It is likely that advancements have been in model development, validation and study conduct over the years; an update is necessary. Furthermore, previous systematic reviews only evaluated models for mortality and reintervention risk prediction.13 14 There are no reviews on models predicting other important outcomes such as case complexity, costs and specific preoperative, perioperative or postoperative complications. Although both the reviews and SVS guidelines highlighted a need for improvement in the quality of evidence for EVAR prognostic models, there is no current review that assessed and compared the risk of bias (RoB), applicability, or completeness of reporting for modelling studies.
Filling these gaps requires a thorough review of the literature that captures the present state of research into prognostic modelling for EVAR. This will allow for a full dissection of current research practices and an indication of whether the key improvements have been made. Subsequently, it will assist in identifying knowledge gaps and expediting factors for bringing models into clinical practice. The results of the study could inform the value of performing a systematic review, developing new models or validating existing models. Considering the broad nature of the goals to be met, a scoping review is the most appropriate method.15
The study will use a scoping review approach to explore and describe the literature on EVAR prognostic modelling. It will cover methods that are currently being used to select candidate predictors, develop prognostic models, evaluate model performance and validate models across various population samples. Furthermore, the literature will be critically appraised to help uncover possible areas of improvement so that recommendations can be made to inform future research into predictive modelling. Overall, the target is to deliver a full picture of how evidence on EVAR prognostic models has evolved and provide context to their potential use in clinical practice. A preliminary search for scoping reviews on the subject was conducted on PubMed Central, Embase and Cochrane Library. No studies were identified that are current or underway.
Methods
Protocol development
This review protocol was developed a priori and will be available on request from the corresponding author. The methodological framework for the proposed scoping review will be guided by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses protocols extension for Scoping Reviews (PRISMA-ScR) (accessed online in 2021 at: https://doi.org/10.7326/M18-0850).16 The PRISMA-ScR checklist will be used to ensure compliance with the guidelines (online supplemental appendix 1).
Supplemental material
Eligibility criteria
Population
There is no upper or lower limit for the sample size described in the paper to be included. Patients from all demographics will be included, so long as they have been managed or will be managed with endovascular repair for AAA. There are no limitations on the type of stent-graft used for inclusion, and both elective and emergency patient groups are eligible. However, studies that do not involve patients with infra-renal aneurysms will be excluded.
Concept
For inclusion, the study must focus on a prognostic predictive model. Diagnostic prediction model studies will be excluded. Predictive variables of all types and combinations used in the model will be included in the review. Unlike previous reviews, outcomes (or cumulative outcomes) that the models are predicting and the time span of the prediction are also open for inclusion. If a study did not report a model and only examined the risk of individual factors in isolation, they will be excluded. If the model described in the study is not explicitly applicable to AAA patients managed with endovascular repair, it will also be excluded.
Context
The inclusion of grey literature is one area that differentiates scoping reviews from systematic reviews; however, it is by no means a requirement.16 Low quality of evidence, rather than model variety, is the primary barrier to models entering clinical practice.12 The proposed review aims to fill this gap by mapping high-quality literature on EVAR prognostic models and understanding the nuances in their methodology to guide future high-quality research. Given these objectives and the variability of grey literature when it comes to reporting study methods, offering reliable results and providing adequate data, the feasibility cost outweighs the potential bias reduction that is gained from including them. Therefore, the review will only include studies and journal articles that are available as peer-reviewed full texts.
Scoping reviews allow researchers to explore a wider variety of evidence than systematic reviews and better appreciate trends in research practices for a given field.15 Including systematic reviews in the study might provide useful insights into the conduct of review literature on EVAR prognostic models, which predictive outcomes are being studied, and how models or studies are being compared systematically. Additionally, informing the value of future systematic reviews is one of the objectives of this study, and including previous systematic reviews could be beneficial. Hence, development studies, validation studies (both internal and external), and systematic reviews will all be eligible for inclusion. This encompasses both retrospective and prospective study designs.
There are no restrictions on the language in which the papers are published. During the screening process, the full texts of all records will be translated to English via online translation tools or professional translation services when required, then assessed for eligibility. In terms of the year of publishing, all references that are published before 2000 will be excluded on the basis that abdominal aortic stent-grafts only received regulatory approval in Europe in 1996 and the USA in 1999.
Information sources
A preliminary limited search of PubMed Central was performed on December 27 2021 using key terms derived from the population, concept and context of the review: ‘abdominal aortic aneurysm’, ‘EVAR’, ‘prediction’ and ‘model’. Additional search terms, thesaurus terms and MeSH terms were elucidated using the results. Truncation devices were subsequently inserted where appropriate, and Boolean operators were applied to optimise the search strategy. Finally, the search strategy was adapted for Embase and Cochrane Library, then peer-reviewed by an experienced librarian (D Stokes). The complete search strategy for all three databases is included in online supplemental appendix 2. The reference lists of all the included documents after the screening process will also be snowballed to retrieve other potentially eligible studies.
Supplemental material
Selection of sources of evidence
The citations identified by database searching will be uploaded onto EndNote to remove duplicates. Two reviewers will be recruited to independently evaluate the citations on an online reference management programme (Rayyan). As a calibration exercise, 100 articles will be selected by convenience sampling to pilot test the screening protocol. The reviewers will first screen the titles and abstracts of these records against the inclusion criteria, after which, inter-rater agreement will be assessed. If agreement is more than 80%, the screening protocol for titles and abstracts will be applied to the rest of the uploaded citations. Conversely, if agreement is under 80%, the reasons for conflict of opinion will be deliberated and the calibration exercise repeated until inter-rater agreement is satisfactory. Following this, the reviewers will examine the full texts of the selected articles in detail for eligibility. When articles are excluded at this stage, the reasons for exclusion will be recorded and outlined in the review. Additionally, inter-rater agreement will be tested again, as described previously, after full-text screening. The reviewers will perform snowballing on the reference lists of included articles and repeat the screening protocol to retrieve more potentially eligible studies. Conflicts or disagreements that arise will be resolved by consulting a third reviewer and gaining consensus through discussion. Results of the evidence selection process will be illustrated using a PRISMA flow chart once the screening process is complete (online supplemental material 1).
Supplemental material
Data charting process
No checklists of data items exist specifically for scoping reviews on prediction models. Although the primary objectives of systematic reviews are different, they resemble scoping reviews relatively closely in methodology. Therefore, the Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS) guidelines were consulted to determine data items to be extracted for developmental and validation studies.17 The PRISMA guidelines for systematic reviews were used to source data items for extraction in included review studies.18 A charting form was developed on Microsoft Excel as a standardised tool for data extraction; available in online supplemental appendix 3. Two reviewers will independently abstract data from all the included sources. As a calibration exercise, a random sample that represents 10% of the total number of eligible papers will be selected for pilot testing the data extraction form. This is to determine whether there are inconsistencies in the data extraction process and if modifications need to be made to the form. Any alterations to the charting form will be reported in the scoping review. This iterative process will be repeated until the team is satisfied with the volume of data collected from each article. Disagreements and conflicts that may arise later during the charting process will be resolved by consulting a third reviewer and gaining consensus through discussion. If there is additional data required that is missing from a retrieved paper, the authors will be contacted when appropriate.
Supplemental material
Data items
The CHARMS guidelines were used to determine which quantitative and qualitative data items needed to be abstracted from the included sources.17 Although this will vary between different study types, the items were broadly grouped into six domains. The PRISMA guidelines for systematic reviews were consulted to determine the data items required for reviews included in the study.18 These were grouped into four domains. All data items are extracted at the individual study level. These are still subject to change and may be further refined during the scoping review.
Developmental and Validation Studies:
General information: first author, title, journal, year of publication.
Study characteristics: setting, study type, study design, objectives, methods, sources.
Study findings: presentation of models, interpretations of study, reviewers’ comments.
Population characteristics: sample size, participants managed with EVAR, participant selection, notable demographics.
Model characteristics: models reported, selection of predictors, shrinkage of predictor weights, candidate predictors, predictor handling, outcomes predicted, events per variable, missed data (predictors, outcomes), modelling method.
Model performance: overall performance, discrimination, calibration, sensitivity, specificity, predictive values, model validation.
Systematic reviews:
General information: first author, title, journal, year of publication.
Study characteristics: setting, objectives, methods.
Data Characteristics: sources of data, eligibility criteria, data items, outcome effect measures, RoB assessment, models reported.
Synthesis: eligibility criteria, methods, synthesis summary, statistical heterogeneity, sources of heterogeneity, sensitivity analysis, reporting bias assessment, certainty assessment, results, interpretations, comments.
Critical appraisal of individual sources of evidence
Currently, there is a shortage of literature that provides a comparative assessment of RoB, applicability and completeness of reporting for prognostic models in EVAR. Such an assessment will help identify methodological gaps in model development conduct and offer a perspective on which models have the potential for use in clinical practice. As this is in line with the objectives of the scoping review, a critical appraisal of the included sources will be performed using standardised rating tools. The Prediction model Risk of Bias Assessment Tool (PROBAST) will be used to evaluate the RoB and applicability of the included sources.19 This will involve two independent reviewers assessing RoB and applicability of the studies through the signalling questions outlined in the tool, then carrying out an overall judgement based on the results. RoB is judged as ‘low’ when there are no shortcomings in all domains and ‘high’ if there is a shortcoming in at least one domain. Similarly, applicability is rated as ‘high concern’ when at least one shortcoming is present and ‘low concern’ if no shortcomings are found. The results of RoB and applicability appraisal will be tabulated as recommended in the PROBAST explanation and elaboration article (online supplemental material 2).19 The Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) checklist will be used as a tool to investigate research conduct across the modelling studies. This uses 22 items, with various subitems, to assess the transparency and completeness of reporting in prediction modelling studies.20 Again, two reviewers will independently assess the studies through the checklist items answering with ‘yes’, ‘no’ or ‘not applicable’. As a calibration exercise, both the tools will be pilot-tested on a random sample representing 10% of the included articles. If inter-rater agreement is found to be over 80%, the tool will be subsequently applied to the rest of the eligible studies. If this threshold is not reached, then the reasons for conflict of opinion will be deliberated and the calibration exercise repeated until agreement is satisfactory. The results will be tabulated as recommended in the PROBAST explanation and elaboration article (online supplemental material 2).18
Supplemental material
Synthesis of results
As recommended in the PRISMA-ScR guidelines,16 the results of scoping review will be synthesised and presented in accordance with the objectives of the paper:
Results of screening: A PRISMA flow diagram will be used to report the number of records identified from selected databases, identified from reference list snowballing, screened for eligibility and included in the scoping review.
Range and volume of literature: The study types and study designs will be tabulated, stating the numbers (and percentages) of papers in each category. This will be executed for each individual model and across all models included in the review.
Critical appraisal: A PROBAST table will be constructed to present the critical appraisal performed for RoB and applicability. For the results of the TRIPOD assessment, individual studies will be tabulated as a list on the x-axis, grouped together by study type (developmental or validation). The TRIPOD checklist items applicable to the study type, which the study fulfilled, will be tabulated on the y-axis. Finally, the percentage adherence to all the applicable items on the checklist will also be calculated for each study.
Comparative summary of models: The models will first be categorised by outcomes predicted, then subcategorised by the time span of predictors. Tables will be constructed for each category to map all the extracted data on model development or validation and compare performance between the models included in the review. A separate subcategory will include the results of systematic reviews and meta-analyses performed for the outcome that the model was predicting.
Discussion of results: Narrative summaries will be written for each of the categories above. Although meta-analysis will not be performed, a discussion section will be written which assimilates the key findings of the study and provides answers for the research questions in relation to the objectives of the scoping review. This aspect of the review will be tailored to healthcare providers and researchers, with emphasis on interpreting whether models may have the potential for use in practice and recommending directions for future research.
Patient and public involvement
This study did not involve patients or members of the public.
Ethics and dissemination
Since a scoping review covers secondary data from published literature, ethical approval is redundant. The findings of the proposed review will be published in peer-reviewed journals whose audience consists of healthcare providers and researchers specialising in vascular surgery. Further dissemination is planned by presentations at key conferences.
Ethics statements
Patient consent for publication
Acknowledgments
The authors acknowledge the contribution of Diarmuid Stokes, a liaison librarian at University College Dublin for offering guidance on developing an appropriate search strategy and peer-reviewing the search strategy created for the proposed review.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Contributors Conceptualising the study and providing expertise in endovascular management of AAA: GAA. Design of the protocol: GAA, ZU-M. Drafting of the manuscript: ZU-M. Final review and approval of the manuscript: GAA, ZU-M.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.