Identification of predictors

Identification of predictor variables was based on data in the RAI-HC, the extant theoretical and empirical literature, clinical experience and consultation with subject-matter experts. We began by compiling predictors identified in prior studies and systematic reviews of mortality prediction models for an older, community-dwelling adult population.8 ,10–13 Then, we reviewed several frailty indices that were developed from mortality prediction models for additional factors that were associated health declines leading to mortality.35–38 We adopted a multidimensional framework by considering overlapping domains of physical functioning (eg, difficulties with ADLs and reduced mobility), cognitive impairment (eg, memory decline and psychosis), sociodemographic factors (eg, marital status and level of education) and biological diseases (eg, diabetes, heart disease and cancer).39–41 We included additional measures of prior healthcare use (eg, number of hospital admissions or emergency room visits in the last 90 days), dependency on life-sustaining therapies (eg, dialysis and respirator) and symptoms of reduced health and physiological reserve (eg, weight loss, oedema and vomiting) to improve the discriminatory power of our algorithm, especially for people nearing the end of life. We also incorporated cohort characteristics (eg, year of the RAI-HC assessment, the type of and reason for assessment) that may account for remaining heterogeneity in the estimated risks. We will select variables based on their clinical relevance and may exclude some that are difficult to understand by the general user. A list of predictors—comprising of 93 df from 47 variables—that will be operationalised in our development model is presented in table 1. Figure 1 presents a conceptual map that groups predictors based on evidence-supported associations to mortality found in the existing literature and clinical experience among members of the research team.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Concept framework of predictors in RESPECT, grouped by evidence-supported contribution to mortality risk found in the existing literature. RESPECT, Risk Evaluation for Support: Predictions for Elder-life in the Community Tool.

Functional capacity will be represented by three commonly used functional and cognitive scales. The IADL Difficulty Scale is a hierarchical index reflective of difficulties with performing housework, preparing meals and using the telephone.6 The ADL Hierarchy Scale includes difficulties with personal hygiene, toileting, locomotion and eating. The extent of a person's limitations in performing these tasks will capture the progressive nature of senescence and is representative of early, middle and late loss functions. Moreover, the ADL Hierarchy Scale is suitable when significant functional deterioration is expected in the short term.42 The Cognitive Performance Scale (CPS) is constructed from items that assess home care clients' short-term memory, procedural memory, the capacity to communicate and be understood and their decision-making capacity. The CPS has been shown to correspond closely with scores generated by the Mini-Mental State Examination.43 We will, first, consider specifying each of the tasks that form the scales as separate and independent categorical variables in our model; this will enable us to examine their independent effects on mortality. In application, this approach will offer clearer directions for intervention and care planning. However, as the variables are conceptually related, they are susceptible to multicollinearity. If multicollinearity among the scale items is evident, we will correct it by replacing the individual items with the scale scores.

Data cleaning and coding of predictors

Data cleaning and coding will proceed without examining predictor–outcome associations. Continuous variables will be inspected using boxplots and descriptive statistics to identify values outside of plausible ranges. Values that are clearly erroneous will be corrected, where possible, or otherwise set to missing. To avoid instability in the regression analyses, frequency distributions for categorical predictors will be examined, and categories with small numbers of respondents will be combined. Variables with narrow distributions or insufficient variation will also be excluded. Missing data are rare and does not present a significant risk to selection bias; only 184 of nearly 1 138 000 assessments (0.02%) from our derivation cohort contained missing data. Therefore, assessments with missing data will be excluded from our analysis.

Model specification

Age is the only continuous candidate variable and will be flexibly modelled using restricted cubic splines (given its known association with mortality). Following Harrell's30 recommendation, we will consider the placement of knots at fixed quantiles of the distribution (eg, at the 5th, 27.5th, 50th, 72.5th and 95th centiles for a five-knot spline function). All other predictors will be modelled as linear terms. Ordinal variables with few levels will be specified as individual categories (ie, as multiple indicator variables), or appropriately collapsed based on sample size and interpretability.

Prespecification of predictors has advantages in limiting the risks of overfitting and spurious statistical significance, but may result in a final model that is overly complex and difficult to interpret. Unnecessary variables will also distort the estimated effects of other predictors and, given the size of our sample, may make the model more computationally intensive. It may be possible to derive a more parsimonious model that retains most of the prognostic information and performs as well or better than the full model, without increasing the type I error rate.30 ,44 We will use the stepdown procedure described by Ambler et al44 to identify a more parsimonious model. This procedure involves deleting variables to a desired degree of accuracy based on contribution to model R². We may consider further reductions, beyond the stepdown procedure, if the completion time for our web calculator exceeds our goal of 5–10 min, or if the questions are found to be abstruse in our pilot study with community care providers. We will evaluate the reduced model against the full model using Akaike's Information Criterion and by examining the effect on discrimination and calibration.

Model estimation

Our main estimation model will be proportional hazards regression with death as the outcome of interest. We will use centred values of all predictors; this will allow us to generate a predicted value in the event of incomplete or partially completed web questionnaires. We will conduct a formal check for multicollinearity using the VARCLUS procedure in SAS.45 Although the risk of overfitting will be minimal, due to prespecification of our model and the large sample size, we will nevertheless assess if this is a concern. The degree of overfitting in the model will be estimated using the heuristic shrinkage estimator, based on the log likelihood ratio χ² statistic for the full model.46 If shrinkage is <0.90, adjustment for overfitting will be required.

Assessment of model performance

A key assumption underlying the proportional hazards model is that the relative risk of the outcome between strata of exposure and the baseline risk must be constant over time. Violation of this assumption has been shown to produce biased results,47 although Allison48 has argued that estimated coefficients of time-varying variables can simply be interpreted as an ‘average’ rather than instantaneous hazard. We will test this assumption by visually examining the plots of raw and smoothed-scaled Schoenfeld residuals against time to identify non-proportionality. We will further examine the existence of non-proportionality by including time-interacted variables in our regression model. If a violation of this assumption is identified—by non-zero correlations between the Schoenfeld residuals and time, or statistical significance of the time-interacted terms—we will implement corrective measures, such as the inclusion of time-interacted predictors or stratification, that are appropriate for proportional hazards regression. We may consider alternate estimation approaches, such as estimating an accelerated failure time model, to account for time-varying effects if these methods are unsuccessful.49 ,50

Predictive performance will be assessed and reported using overall measures of predictive accuracy (eg, Nagelkerke's R² and the Brier score), discrimination (eg, the c-statistics) and calibration (eg, comparing the observed to predicted deaths by vigintiles of predicted probability). Steyerberg33 and Cook51 ,52 suggest that calibration should receive more attention when evaluating prediction models and that assessment of recalibration tests and calibration slopes should be performed routinely. We will emphasise visualisation of model performance using plots, rather than formal statistical testing, since the significance of traditional Hosmer-Lemeshow goodness-of-fit tests, for example, may be an artefact of large sample sizes rather than true miscalibration.30 We will generate the calibration slope by regressing the outcome in the validation cohort on the predicted mortality risk. This approach reflects the combined effect of overfitting to the derivation data as well as true differences in effects of predictors.46 A well-calibrated model should have a slope of ‘1’.30 ,53 Deviations from this will be tested using a Wald or likelihood ratio test. Additionally, we will create calibration plots comparing mean predicted probabilities with Kaplan-Meier estimates of observed rates across intervals of predicted risk. Subgroup validation will also be implemented as a conceptually easy check of calibration. This entails comparing observed and predicted risks across vigintiles and within predefined subgroups of importance to clinicians and policymakers (eg, age groups, sex, diseases as well as physical and cognitive functioning scale scores). The clinical or policy relevant standard of calibration is defined as <20% difference between observed and predicted estimates for categories with prevalence higher than 5%.54

Model presentation

First, we will present the final regression model—that is, a model estimated from the combined sample of the derivation and validation cohorts—showing estimated HRs and 95% CIs. However, given the anticipated complexity of the final regression model, the usual presentation may be less meaningful. We will complement the presentation of the regression models with plots of the survival functions for the entire sample and by subgroups of interest (eg, age groups, sociodemographic groups, diseases, levels of functional and cognitive impairment). Second, in addition to the predicted median survival time at 6 and 12 months, we will aim to estimate and report the survival times at other risk percentiles (eg, 25th and 75th). Third, we will present measures of functional performance (eg, scores on the ADL Hierarchy Scale or the CPS) in our web report on http://www.projectbiglife.ca to aid the interpretation of the predicted mortality risk, since these are commonly used indicators for determining the need for support in the community.

Additional considerations

As with all regression models, the omission of key predictors and misspecification of non-linear relationships can bias the model estimates. Potentially influential predictors that are not captured by the RAI-HC include earned income and rurality, which are indicators of access to healthcare and may serve as indirect proxies for health status. We will consider examining the effects of omitted variable by re-estimating the models with inputs from other health administrative data sources that contain these variables. In addition to the RAI-HC, ICES houses population-level data from physician billings, hospital discharge records and records from individuals receiving care in nursing homes. However, our primary model will only consider RAI-HC data; this is to support external validations and the replication of our study in provinces or countries that do not have linked RA-HC and other health administrative data.

Most existing predictive algorithms in older people have not accounted for non-linear relationships or interactive effects among predictors. For example, to account for individuals with varying severity of chronic obstructive pulmonary disease (COPD), we may interact COPD with receipt of oxygen therapy. We will consider the inclusion of interaction terms if the algorithm performs poorly for subgroups of interest. This process will be guided by relevant literature, such as the interactive effect of age with cancer diagnosis9 or disease by functional status.55

Following the development of our full algorithm, we aim to create a set of screening questions to reduce response burden for people who are at a low risk of death. RESPECT-lite will be an abridged version of the full algorithm, created by recursive partitioning techniques. We will identify main branches within the classification tree that separates low-risk, medium-risk and high-risk individuals. These nodes will then serve as the screening questions. For example, we expect that people who are at a low risk of dying will have few restrictions in ADLs, whereas people who are at a high risk will have multiple functional limitations and symptoms that are worsening (with possible cumulative effects involving multiple predictors). The ADL items or scale could then be used to classify respondents into the risk groups. We anticipate that RESPECT-lite will take <1 min to complete, based on our experience with other calculators implemented on http://www.projectbiglife.ca. We will assess the validity of RESPECT-lite by comparing it against the full model.

The proportional hazards regression was developed for estimating relative risks associated with a set of covariates, and generating survival time post hoc may be computationally challenging.48 We will consider taking a parametric approach to modelling (eg, estimating an exponential or Weibull model) if we are unable to derive survival estimates using proportional hazards regression.

Finally, since the RAI-HC assessments are usually completed by or in the presence of a trained healthcare professional (eg, nurse), self-reporting or the lay interpretation of the items may result in inaccurate estimates for the individual. For example, it has been shown that dementia patients often underestimate their own level of cognitive deficit, in comparison to their caregivers; though this is not a concern for self-reported functional status.10 ,56 In this example, an underestimation of their cognitive deficits may lead to an lower risk of mortality (ie, longer survival) than an accurate assessment. At the time of our algorithm development, the magnitude of the deviation between professional and lay responses is unknown. Nonetheless, we recognise this limitation and the implications of making this tool publicly accessible. We are conducting on-going assessments of the usability and impact of using the RESPECT web calculator through focus groups involving healthcare professionals, patients and their families or caregivers. Our future research involving RESPECT will also include a validation study comparing responses provided by healthcare professionals to those provided by patients.