Article Text
PDF
Abstract
Introduction Acute kidney injury (AKI) has high morbidity and mortality in intensive care units, which can lead to chronic kidney disease, more costs and longer hospital stay. Early identification of AKI is crucial for clinical intervention. Although various risk prediction models have been developed to identify AKI, the overall predictive performance varies widely across studies. Owing to the different disease scenarios and the small number of externally validated cohorts in different prediction models, the stability and applicability of these models for AKI in critically ill patients are controversial. Moreover, there are no current risk-classification tools that are standardised for prediction of AKI in critically ill patients. The purpose of this systematic review is to map and assess prediction models for AKI in critically ill patients based on a comprehensive literature review.
Methods and analysis A systematic review with meta-analysis is designed and will be conducted according to the CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS). Three databases including PubMed, Cochrane Library and EMBASE from inception through October 2020 will be searched to identify all studies describing development and/or external validation of original multivariable models for predicting AKI in critically ill patients. Random-effects meta-analyses for external validation studies will be performed to estimate the performance of each model. The restricted maximum likelihood estimation and the Hartung-Knapp-Sidik-Jonkman method under a random-effects model will be applied to estimate the summary C statistic and 95% CI. 95% prediction interval integrating the heterogeneity will also be calculated to pool C-statistics to predict a possible range of C-statistics of future validation studies. Two investigators will extract data independently using the CHARMS checklist. Study quality or risk of bias will be assessed using the Prediction Model Risk of Bias Assessment Tool.
Ethics and dissemination Ethical approval and patient informed consent are not required because all information will be abstracted from published literatures. We plan to share our results with clinicians and publish them in a general or critical care medicine peer-reviewed journal. We also plan to present our results at critical care international conferences.
OSF registration number 10.17605/OSF.IO/X25AT.
- adult intensive & critical care
- acute renal failure
- risk management
- intensive & critical care
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
This study will provide an overall mapping of the available studies on prediction models for acute kidney injury (AKI) in critically ill patients.
This study will be carried out and reported according to the CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies.
Prediction models for AKI in critically ill patients will be evaluated using the Prediction Model Risk of Bias Assessment Tool.
Meta-analysis of C-statistics will be conducted for prediction models that are externally validated in different individual populations.
Several potential sources of heterogeneity including AKI definition, AKI type, window of prediction and other study characteristics will need further investigation.
Introduction
Acute kidney injury (AKI) is a common condition among hospitalised critically ill patients, especially in intensive care units (ICUs), and has been a major healthcare burden worldwide.1–4 AKI is also associated with serious complications, increased healthcare costs, length of stay and mortality. More than 1.7 million deaths have been reported indirectly due to AKI annually related to chronic kidney disease (CKD), cardiovascular and cerebrovascular events.5–8
AKI can originate from heterogeneous causes, and stratifying cases according to characteristics and biomarkers would raise possibility of early prediction of AKI. Currently, there is a great need for multimodal data in the development of these models as the clinical trajectory of critical illness involves multiple organ dysfunction and organ crosstalk, which can be captured with different data types. Biomarkers including serum creatinine and urine output are commonly used ones to define AKI.9 More recently, several other frequently used and new candidate biomarkers have been found to predict AKI in clinics at different stage of the disease condition, but many of them are found to have low sensitivity and specificity.10–16
Clinical prediction models are widely used in real-world clinical practice. They are proved to be useful for informing healthcare systems to distinguish high-risk patients, guide diagnostic and therapeutic intervention selection; thus, early measurements could be taken to improve outcomes.17 The application of multidimensional indicators to predict the risk of AKI in critically ill patients may provide a more comprehensive approach of disease assessment. Furthermore, in critically ill patients, multivariable risk prediction models for AKI could be used in clinical practice to assist decision making on hospital admission or admission to ICUs and treatment options.18–20
Several prediction models, incorporating multiple predictors for the prediction of AKI, have been developed. Wang et al21 found that hypertension, CKD, acute pancreatitis, cardiac failure, shock, pH ≤7.30, creatine kinase >1000 U/L, hypoproteinemia, nephrotoxin exposure and male gender were independent predictors of AKI. Ferrari et al22 established a novel prediction score to quickly predict AKI at any stage up to 7 days. However, to the best of our knowledge, no prognostic model for AKI has been endorsed. Moreover, in routine clinical practice, there are no current risk-classification tools that are standardised for prediction of AKI in critically ill patients.
In this study, we aim to systematically summarise the reported multivariable models developed for predicting AKI in critically ill patients, to map their characteristics and laboratory features, and to test whether they have been carried out external validation. We will apply the Prediction Model Risk of Bias Assessment Tool (PROBAST) to assess the risk of bias of the methodological aspects of the included studies developing or validating prediction models. For prediction models involving several validation studies, we will perform a meta-analysis for performance and calibration of each model to yield more accurate effect estimates.
Methods and analysis
We will design and conduct this systematic review according to Preferred Reporting Items for Systematic Review and Meta-Analyses Protocol guideline23 and the CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS)24 and the guidance by Debray et al.25 We have registered the protocol on the website of open science framework (https://osf.io/x25at/).
Literature search
We systematically searched PubMed, Embase and Cochrane Library from inception to October 2020 to capture all relevant studies developing and/or validating a prediction model for AKI in critically ill patients. The following search strategy with related key words was developed: (predict* OR progn* OR “risk prediction” OR “risk score” OR “risk calculation” OR “risk assessment” OR “C statistic” OR discrimination OR calibration OR AUC OR “area under the curve” OR “area under the receiver operator characteristic curve”) AND (“acute kidney failure” OR “acute tubular necrosis” OR “acute kidney tubule necrosis” OR AKI OR ARI OR AKF OR ARF) AND (“emergency care unit” OR “intensive care unit” OR “critical ill patient” OR “acute ill*” OR ICU). Two independent investigators will undertake the literature search and screening, and discrepancies will be resolved by a senior author. We will further hand-search the reference list of each eligible study for potential missing eligible studies.
Eligibility criteria
We will include all cohort studies that described development and external validation of original multivariable models for predicting AKI in criticaly ill patients.
We present the detailed description of the PICOTS (population, intervention, comparator, outcomes, timing, setting) for this systematic review in table 1. Based on the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis guideline,26 we will screen and select eligible prognostic model studies when the following inclusion criteria are satisfied.
Primary elements for formulating study purpose, search strategy, inclusion and exclusion criteria for the study according to the following PICOTS guidance
(1) studies that reported the development or validation multivariable model(s) of AKI with or without external validation; (2) studies reporting AKI models involving medical-AKI-related critically ill patients and using AKI definitions of Kidney Disease Improving Global Outcomes (KDIGO),27 Acute Kidney Injury Network (AKIN),28 and Risk, Injury, Failure, Loss and End-Stage Kidney Disease (RIFLE)29; (3) studies that yielded at least two predictors and (4) studies that evaluated or updated the quantitative measure of model performance of an existing model in an independent population in terms of overall performance, discriminative ability and calibration of a certain prediction model. We will exclude conference abstracts, editorials, clinical case reviews, letters, commentaries, book chapters and surveys. Studies involving only postsurgical critically ill patients will also be excluded.
Data abstraction
Data extraction will be conducted using a standardised data extraction form by at least two independently reviewers based on the recommendations in the CHARMS checklist.24 If the needed data are not reported or unclear, the corresponding authors will be contacted for detailed information. The following general study information will be extracted including first author, publication year, model name, publication source and research country. For model development study, we will extract the following specific data: modelling method, method for selection of predictors for inclusion in multivariable modelling combined with criteria used and shrinkage of predictor weights or regression coefficients. For model performance study, measures of calibration and discrimination with CIs will be abstracted. For studies reporting model evaluation, method used for testing model performance will also be abstracted. Besides, the method for treating the missing data involving the prediction model of each eligible study will also be abstracted.
Critical appraisal
We will critically appraise each included prediction model using the PROBAST technique, a tool to assess risk of bias and applicability of prediction model studies.30
Based on the checklist of PROBAST, 20 separate questions across four domains (participants, predictors, outcome and analysis) will be asked. Details for the assessment rules are summarised in box 1. Moreover, we will also use the Modified Downs and Black Checklist and Sackett’s level of evidence for assessment of risk of bias and methodological quality of included studies.31 32 Two authors of the research team will independently assess the risk of bias of the included studies and crosscheck the results. Any discrepancies will be resolved by discussion or by a senior author.
Twenty key questions assessing the risk of bias for four domains of participants, predictors, outcome and analysis
Domain 1: Participants
1.1 Were appropriate data sources used, for example, cohort, randomised controlled trial (RCT) or nested case–control study data?
1.2 Were all inclusions and exclusions of participants appropriate?
Domain 2: Predictors
2.1 Were predictors defined and assessed in a similar way for all participants?
2.2 Were predictor assessments made without knowledge of outcome data?
2.3 Are all predictors available at the time the model is intended to be used?
Domain 3: Outcome
3.1 Was the outcome determined appropriately?
3.2 Was a prespecified or standard outcome definition used?
3.3 Were predictors excluded from the outcome definition?
3.4 Was the outcome defined and determined in a similar way for all participants?
3.5 Was the outcome determined without knowledge of predictor information?
3.6 Was the time interval between predictor assessment and outcome determination appropriate?
Domain 4: Analysis
4.1 Were there a reasonable number of participants with the outcome?
4.2 Were continuous and categorical predictors handled appropriately?
4.3 Were all enrolled participants included in the analysis?
4.4 Were participants with missing data handled appropriately?
4.5 Was selection of predictors based on univariable analysis avoided?
4.6 Were complexities in the data (eg, censoring, competing risks and sampling of control participants) accounted for appropriately?
4.7 Were relevant model performance measures evaluated appropriately?
4.8 Were model overfitting and optimism in model performance accounted for?
4.9 Do predictors and their assigned weights in the final model correspond to the results from the reported multivariable analysis?
Statistical analysis
We will calculate and report descriptive statistics to summarise the characteristics of the AKI models. For binary or categorical variables, we will calculate frequencies or percentages, while for continuous variables, mean values, medians and IQRs will be calculated. For the prediction model of AKI developed from different populations, a random effects meta-analysis will be applied to calculate a summary estimate for models’ performance and calibration. For studies that did not provide measurements of mean C-statistics, we will use a formula to estimate the SE of mean C-statistic according to the methods proposed by Sackett et al.32
Owing to the relatively small sample size of validation studies for each prediction model, we will meta-analyse C-statistic with its 95% CI using a random-effects model based on the restricted maximum likelihood estimation and the Hartung-Knapp-Sidik-Jonkman method.25 33 95% prediction interval integrating the heterogeneity will also be calculated to pool C-statistics to predict a possible range of C-statistics of future validation studies. Heterogeneity between studies will be quantified using the I2 statistic, defined significant heterogeneity when I2 statistic more than 50%.34 To explore the sources of potential heterogeneity, we will conduct stratified analyses by summarising estimates based on AKI definition (KDIGO vs AKIN vs RIFLE), AKI type (any AKI vs severe AKI or stage 1 AKI vs stage 2/3 by KDIGO criteria), window of prediction (first 24 hours vs 48–96 hour) and lack of evaluation of key characteristics of AKI such as duration, need for renal replacement therapy (yes vs no). The potential of publication bias will be assessed by funnel plots when >10 studies are meta-analysed for the prediction model. All statistical analyses will be carried out using R Statistical Software V.3.2.3 (R Core Team, R Foundation for Statistical Computing, Vienna, Austria) and Stata V.15.0 (Stata Corporation, College Station, TX, USA).
Patient and public involvement
This study is a systematic review of what is already reported in the literature. It does not involve patient and public in the design, conduct or reporting of this study.
Ethics and dissemination
Ethical approval and patient informed consent are not required because all information will be abstracted from published literature works. We plan to share our results with clinicians and publish them in a general or critical care medicine peer-reviewed journal. We also plan to present our results at critical care international conferences.
Amendments
The protocol for this systematic review will be amended when necessary during the peer-review process.
Discussion
Although there have been numerous original reports and narrative reviews focusing on the prediction model of AKI,35–43 several factors may limit the interpretation and application of these prediction models. To the best of our knowledge, this will be the first systematic review that aims to evaluate the published evidence on the prediction models for AKI. This study will provide a clear overview for clinicians to identify some most effective prediction models for AKI among critically ill patients or patients in ICUs. By synthesising data including predictive accuracy such as C-statistics across studies, we may get some evidence-based data to stratify disease severity and help inform the clinical management of critically ill patients.
Strengths and limitations
There will be several strengths of this study. First, we will strictly adhere to the Cochrane Handbook’s method recommendations during the conduct and reporting of this systematic review to make the results more reliable.25 44 The purpose of this study is to achieve high-quality evidence regarding the prediction model of AKI in critically ill patients and provide practice recommendations on its applicability for policy makers. Second, we will present a detailed description of the characteristics of the reported prediction models for AKI. Moreover, another important strength is the critical appraisal of prediction models for AKI by using the PROBAST tool. Finally, we will perform a metaanalysis of C-statistics for prediction models that are externally validated in different independent cohorts.
There are also limitations to this study. One is that large between study heterogeneity is expected in the meta-analyses. There may be several potential sources of heterogeneity including the differences in clinical scenarios, patients’ characteristics, cohort regions or races and statistical methods. However, due to the small number of development or validation studies, subgroup analyses or metaregression analyses cannot be performed.
In summary, this study will provide an overall mapping of the available research on prediction models for AKI in critical ill patients.
References
Footnotes
DW and ZM contributed equally.
Contributors ZM and DW: full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. ZM and DW: study concept and design. DW, WZ, JL, HF, SJ and ZM: acquisition, analysis or interpretation of data. ZM and DW: drafting of the manuscript. DW, WZ, JL, HF, SJ and ZM: critical revision of the manuscript for important intellectual content. DW, WZ and ZM: statistical analysis. ZM and DW: administrative, technical or material support. All authors: study supervision.
Funding This study was supported by the Commonweal Project of Science and Technology Department of Zhejiang Province (grant no. LGF19H150002), the National Natural Science Foundation of China (grant no. 81774112) and a grant from Siming Scholars from Shuguang Hospital (grant no. SGXZ-201913).
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.