Article Text

Download PDFPDF

Predictive risk modelling under different data access scenarios: who is identified as high risk and for how long?
  1. Tracy L Johnson1,
  2. Jill Kaldor2,
  3. Michael O Falster3,
  4. Kim Sutherland2,
  5. Jacob Humphries4,
  6. Louisa R Jorm3,
  7. Jean-Frederic Levesque2
  1. 1 Centre for Health Economics Research and Evaluation, University of Technology Sydney, Ultimo, New South Wales, Australia
  2. 2 Bureau of Health Information, Chatswood, New South Wales, Australia
  3. 3 Centre for Big Data Research in Health, University of New South Wales, Sydney, New South Wales, Australia
  4. 4 NSW Ministry of Health, North Sydney, New South Wales, Australia
  1. Correspondence to Dr Tracy L Johnson; tracy.johnson.hps{at}


Objective This observational study critically explored the performance of different predictive risk models simulating three data access scenarios, comparing: (1) sociodemographic and clinical profiles; (2) consistency in high-risk designation across models; and (3) persistence of high-risk status over time.

Methods Cross-sectional health survey data (2006–2009) for more than 260 000 Australian adults 45+ years were linked to longitudinal individual hospital, primary care, pharmacy and mortality data. Three risk models predicting acute emergency hospitalisations were explored, simulating conditions where data are accessed through primary care practice management systems, or through hospital-based electronic records, or through a hypothetical ‘full’ model using a wider array of linked data. High-risk patients were identified using different risk score thresholds. Models were reapplied monthly for 24 months to assess persistence in high-risk categorisation.

Results The three models displayed similar statistical performance. Three-quarters of patients in the high-risk quintile from the ‘full’ model were also identified using the primary care or hospital-based models, with the remaining patients differing according to age, frailty, multimorbidity, self-rated health, polypharmacy, prior hospitalisations and imminent mortality. The use of higher risk prediction thresholds resulted in lower levels of agreement in high-risk designation across models and greater morbidity and mortality in identified patient populations. Persistence of high-risk status varied across approaches according to updated information on utilisation history, with up to 25% of patients reassessed as lower risk within 1 year.

Conclusion/implications Small differences in risk predictors or risk thresholds resulted in comparatively large differences in who was classified as high risk and for how long. Pragmatic predictive risk modelling design decisions based on data availability or projected high-risk patient numbers may therefore influence individuals identified as high-risk, overall case mix and risk persistence. Routine data linkage would enable greater flexibility in developing and optimising predictive risk models appropriate to both case-finding and performance measurement applications.

  • health informatics
  • health economics
  • primary care
  • health policy

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See:

Statistics from

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Strengths and limitations of this study

  • This simulation illustrates the extent to which predictive risk models (PRMs) that rely on different data or models specifications will appraise patient risk status differently.

  • Linked population and service use data facilitated simulation of several ‘real world’ PRM case-finding and performance measurement applications.

  • Simulation findings are intended to be broadly illustrative, not generalisable.

  • In practice, case mix, risk persistence and high-risk group agreement across alternative PRM applications could differ due to differences in risk factor availability or measurement, base populations, prediction periods and other modelling specifications.

  • The study population is not representative of New South Wales due to oversampling of elderly and rural residents and a low survey response rate (18%) limit.


To address population health objectives and expenditure growth associated with ageing populations and increased chronic disease prevalence, governments and health systems in the UK, Australia, the USA and elsewhere are exploring the use of predictive risk modelling (PRM) to better target and integrate services.1–5 PRM algorithms calculate the probability that a specific patient will experience a future event, such as hospitalisation, based on their unique risk profile. Two different but related applications include identifying individual patients for intervention (‘case-finding’) and creating high-risk population segments for focused healthcare performance analysis.

Internationally, considerable variability exists in PRM implementations due to differences in health system organisation and financing, which affects perceptions of accountability and data access. The entity responsible to administer PRM can be hotly debated, as exemplified by the UK’s shift from a centrally administered algorithm to practice-specific adaptations.6 PRM requires access to detailed, patient-level risk factor and health service information, ideally, across clinical and community settings and over time. However, for many countries, data are non-linked or partially linked. Although important to intervention design, limited comparative information exists regarding high-risk patient characteristics identified in different data environments.7

A number of review studies compare alternative PRMs with focus on predictive statistics rather than resultant patient profiles and risk persistence8–12 Validation studies of specific instruments also focus on statistical performance, with some also estimating future per-person spending of identified high-risk patients13–16 Billings et al’s study17 is one of the few studies to quantify gains in prediction performance of a hospital-oriented PRM through sequential addition of emergency, outpatient and general practice (GP) information. They also observe that PRMs using non-hospital data identified more lower acuity patients which could present earlier intervention opportunities.17 Two studies have found that high-risk population subgroups differ according to the persistence of high-risk status over time.18 19

Australia’s planned use of PRM in state and national trials provides an opportunity to examine patient profiles under different data access scenarios. For example, the Commonwealth’s health care homes initiative will employ an automated PRM using GP data to identify primary care patients who are at risk of hospitalisation and assess them for allied health service needs and develop multidisciplinary care plans.20 In parallel, New South Wales (NSW) Health will implement an integrated care initiative that will use a hospital algorithm to identify recently discharged patients who need similar supports.21 Both programmes will require that GPs use a standardised care clinical screening tool to determine eligibility for specific services. Despite patient identification efforts that rely on different PRMs using different combinations of GP data, hospital data and patient survey data, both programmes focus on enhanced outpatient care provision to high-risk patients. The likelihood of patient overlap in separately administered high-risk patient identification efforts is unknown.

This research critically explored the comparative patient identification performance of different PRM algorithms, simulating three common data access scenarios: a ‘full’ model using all available information, a primary care data only (‘GP’) model and a hospital data only (‘hospital’) model. Using models that draw elements from planned Australian PRMs and patient assessment tools, we assessed: (1) sociodemographic and clinical profiles; (2) consistency in high-risk designation across models; and (3) persistence of high-risk status.


Data sources

The PRMs used population survey and linked health administrative data for participants in the 45 and Up Study. The Sax Institute’s 45 and Up Study is drawn from the population of the state of NSW, Australia. Prospective participants were randomly sampled from the Department of Human Services (DHS; formerly Medicare Australia) enrolment database, which provides near complete coverage of the population. People 80+ years of age and residents of rural and remote areas were oversampled. A total of 266 942 participants joined the study by completing a baseline questionnaire (between January 2006 and December 2009) and giving signed consent for follow-up and linkage of their information to routine health databases. With approximately 18% of those invited responding, participants represent about 11% of the NSW population aged 45 years and older.22

This analysis also incorporated information about respondents’ health service use and mortality, obtained with patient consent from administrative health databases and linked to their survey responses. This included public and private sector hospital separation and emergency department (ED) presentation information from the NSW admitted patient data collection (APDC) and emergency department data collection (EDDC). It also included information about subsidised GP care from the Medicare Benefits Schedule (MBS) and prescription drug use from the Pharmaceuticals Benefits Scheme (PBS). Fact of death information was obtained from the NSW Registry of Birth Deaths and Marriages (RBDM).

The Sax Institute used a unique identifier provided by the Australian DHS to link survey responses to the MBS and PBS that the DHS provided. Using probabilistic methods, the Centre for Health Record Linkage conducted the data linkage of APDC, EDDC and RBDM data sets.

Cohort definition

We created a NSW cross-sectional, population-based cohort (n=263 328) that includes both primary care users as well as those with recent hospitalisations, as these are common target populations in medical home and integrated care initiatives, including those in Australia.

Prediction outcome and prediction period

The PRM algorithms estimated each respondent’s probability of experiencing one or more acute emergency admissions during Fiscal Year 2009/2010. High-risk patients were identified as those in the highest quintile (top 20%) of predicted probabilities of hospitalisation. Alternative high-risk thresholds, that is, the top 5% or 10%, were also examined.

Predictor and descriptive variable definitions

A consolidated list of predictor variables was drawn from tools in use in the Australian integrated care and health care home trials, which included a validated Canadian PRM that uses hospital data only, a UK PRM that includes GP data and an Australian-developed clinical assessment tool23–26 (see online supplementary appendix A). Where data limitations prevented matching measure specifications from the three tools, variable definitions drew from previous work associating the predictor with hospitalisation.27–30 Covariates included self-reported measures from the 45 and Up Study (including sociodemographics, social support, health status, health behaviours and functional status) as well as utilisation history from the APDC, EDDC and MBS data sets (see table 1).

Supplementary file 1

Table 1

Variables used in predictive risk models

Missing survey responses were included as discrete values under the assumption that programmes would likely attempt to estimate risk for patients with missing information, rather than exclude them from consideration for services. Predictor variables were used as PRM covariates and as descriptive variables to characterise the resultant high-risk populations. Additional analysis variables identifying clinical subgroups were created by collapsing or combining PRM predictors.

Sociodemographics and social support predictors

Sociodemographic and social support variables (age, gender, indigenous status, marital status, language, geographic remoteness, socioeconomic status, income and social isolation) were obtained from the baseline questionnaire. Using birth date, baseline age was updated to reflect respondent age during prediction and subsequent measurement periods. Quintiles of socioeconomic status were derived from residential postcode using the Australian Bureau of Statistics (ABS) Index of Relative Socioeconomic Disadvantage (Socio-Economic Indexes for Areas).31 Similarly, the ABS Accessibility Remoteness Index of Australia Plus was used to classify remoteness into major city, inner regional, outer regional, remote and very remote categories.32 Indigenous status (yes/no) included those who self-identified as Aboriginal or Torres Strait Islander, or both. Income was based on ‘usual yearly household income before tax, from all sources’ and categorised into $10 000 increments through $70 000. Social isolation was identified as a response of ‘no one’ to a survey question that asked, ‘how many people outside your home, but within one hour of travel, do you feel you can depend on or feel very close to?

Health status and health behaviour predictors

To ensure consistency in covariate measurement across the three PRM models, all health status and health behaviour measures were calculated from the baseline survey even when also available in administrative data. Several predictors consisted of risk factor counts. The health condition count summed self-reported chronic conditions (0, 1, 2, 3+) according to participants’ responses to the questions ‘Has a doctor ever told you that you have…?’ or ‘In the last month have you been treated for …?’ The eight conditions counted were: heart disease, diabetes, high blood pressure, stroke, blood clots, cancer, asthma or Parkinson’s disease. The depression/anxiety measure (yes/no) was also derived from these same questions. The polypharmacy variable totalled the number of medications (0, 1–4, 5+) that respondents had ‘taken most of the last 4 weeks’, as selected from a list of common medications. The falls count (0, 1, 2+) was based on the question, ‘during the past 12 months, how many times have you fallen to the floor or the ground?

Additional predictors were derived from validated items or short standardised scales included in the 45 and Up baseline questionnaire. Self-rated health (SF-1) was reported as excellent, very good, good, fair or poor. The body mass index was collected in kg/m2 and responses were classified as underweight (<18.5), normal weight (18.6–25), overweight (26–30) and obese (>30). The question ‘are you a regular smoker now?’ was used to assess baseline smoking status. Alcohol use was estimated from self-reports of ‘about how many alcoholic drinks do you have each week?’ with unsafe use defined as more than 14 per week (adjusted for 9% under-reporting).33 Functional capacity was defined according to the Medical Outcomes Study, Short Form 36 Physical Functioning Scale scores, with no limitation corresponding to a score of (100), minor limitation (95–99), mild (85-94), moderate (60–84) and severe limitation (0–59).34

Prior hospital/ED utilisation and primary care accessibility

APDC, EDDC and MBS administrative data sets were used to calculate prior utilisation predictors. Variable specifications for the hospitalisation-related covariates were based on the Ontario Hospital Admission Risk Prediction. Utilisation predictors included: acute admission 6 months prior (0, 1, 2, 3+), length of stay of prior acute admission (0–2, 3–7, 8–14, 15–30, 31+), admission via ED (yes/no) and ED visits 6 months prior (0, 1, 2, 3, 4+). GP visits in the prior 12 months were also calculated and classified as (0, 1–3, 4–5, 6–11, 12+).35 Look-back periods for prior hospital, ED and GP service use were calculated from the start of the prediction period (1 July 2009).

Population subgroups

Frail elderly were defined as individuals 65+ years that reported severe physical limitations and/or two or more falls in the last yeari. 36 37 Superutilisers were those with 2+ acute admissions in the previous 6 months. Fair/poor health indicates those whose self-rated health was fair or poor and multimorbid individuals were those with 2+ health conditions. Those who died during the prediction year are classified as end of life. Although these subpopulations were selected because they are often considered clinically distinctive, they are not mutually exclusive designations.

PRM modelling scenarios

Three PRM models were developed to simulate high-risk population identification within alternative data environments. These simulations used a combination of service use and population survey data to approximate data commonly captured through hospital electronic health records, GP practice management systems and clinical assessment tools.

PRM-1 (‘full model’) was built with comprehensive, linked data and used the full, consolidated list of predictor variables to define high-risk status. In addition to data elements routinely available in GP and hospital settings, it included less commonly collected risk factors such as income, social isolation and functional status. PRM-2 (‘GP model’) simulates a primary care-based implementation. Based on the capabilities of the most common practice management software in Australia, GPs were assumed to have electronic access to sociodemographics, language, health/mental health status and health behaviours. Conversely, the GP model did not include prior hospitalisation, ED or GP visit information because common GP practice management software does not track utilisation history. While practices often have separate business software that includes GP visits, only very sophisticated practices have the data skills to link business information and clinical information for analysis. We did, however, conduct sensitivity analyses on our decision to exclude past GP use via a GP+ model that included GP utilisation. Finally, PRM-3 (‘hospital model’) assumed that primary care patient rosters were matched with hospital administrative data, providing access to patient demographics, health status, as well as past hospital and ED use. This data matching assumption reflected actual practice in Australia’s integrated care pilot and ensured consistency in the base population across the three PRM test scenarios. (See table 1 for model-specific risk predictors.)

All models used a logistic regression to predict the outcome of any acute emergency hospitalisation in Fiscal Year 2009/2010. The extent to which high-risk individuals were admitted to hospital as predicted (positive predictive value) was assessed and the c-statistic was calculated.38 All statistical analyses were performed in SAS V.9.4 (SAS Institute).

Descriptive analysis and assessment of persistence

We compared case mix, identification consistency and risk persistence among the high-risk populations identified by each model. High-risk persistence refers to the proportion of high-risk individuals who continue to be (or reachieve) high-risk status. To assess the persistence, we identified individuals in the high-risk quintiles for four models. Two GP models were included in this analysis, with and without GP utilisation history as a predictor variable. We recalculated the risk quintile of these high-risk people on a monthly basis over a 2-year period, applying updated age and utilisation risk factor information. All other risk predictors were measured at solely baseline, so are not time-varying covariates. Individuals who died during the prediction period were removed from this analysis in the month of their death.


Of the 266 942 participants in the 45 and Up Study, n=266 519 were eligible for inclusion because they used a hospital/ED/medical/pharmacy service between July 2006 and June 2009. Of these, n=3182 were excluded as they died prior to the 1-year prediction period (30 June 2009). An additional n=9 were excluded for possible linkage errors, leaving 263 328 patients for analysis. Characteristics of the full population are provided in online supplementary appendix B.

Supplementary file 2

The probability threshold for the high-risk quintiles was low for all models (12.0%–13.1%). Approximately one-quarter (22.2%–24.9%) of patients in the top risk quintile experienced an emergency admission during the prediction year (positive predictive value) and these high-risk patient admissions represent one-half (48.9%–54.8%) of total admissions (sensitivity). Statistical performance of the three PRM scenarios was very similar, with moderately strong c-statistics for all models, ranging from 0.74 to 0.77.

Despite similarities in predictive accuracy, the three PRM scenarios yielded different high-risk individuals and population characteristics (figure 1supplementary appendix B.

Figure 1

Comparison of high-risk quintiles from the full, GP and hospital scenarios. GP, general practice.

Sharing three-quarters (74%) of patients in common with the full model, the primary care model high-risk group includes more multimorbid individuals (62.8% vs 55.9%) and those in fair/poor health (43.2% vs 39.5%) but fewer superutilisers (8.0% vs 13.4%) and those at the end of life (3.9% vs 4.4%). The hospital high-risk quintile also agreed with the full model one three-quarters of the time (77%). Across the three models, the hospital high-risk group includes the lowest proportion of frail elderly (34.4% vs 42.3%) and those in fair/poor health (27.5% vs 43.2%) and highest proportion of superutilisers (14.3% vs 8.0%). These case mix differences across high-risk quintiles are driven by differences among the one-quarter of patients not shared in common (see figure 1).

Nearly two-thirds (59%) of the top quintile of high-risk individuals were consistently identified within all three models (see Figure 2). As compared with the full model, this overlap group has higher morbidity and mortality: with multimorbid individuals (69.6% vs 55.9%), frail elderly (50.1% vs 42.3%) in fair/poor health (42.7% vs 39.5%) and very old (31.4% vs 18.7%). They are also somewhat more likely to be at the end of life (5.7% vs 4.4%). As the high-risk threshold is increased to the 90th or 95th decile, positive predictive values increase in the range of 28.3–33.5 and 34.3–41.8, respectively, but the models’ intersection decreases, falling to 48% and 39% shared patients, respectively. This results primarily from the GP model diverging from the other two (online supplementary appendix B).

Figure 2

Intersection of models for different high-risk thresholds.

Table 2 summarises the extent to which characteristics of the high-risk patient cohort changed when using different models or risk prediction thresholds. High-risk group acuity increased dramatically as high-risk thresholds increased. Proportions of very old, frail elderly, multimorbid individuals, superutilisers and those who report fair/poor health increased by more than 10 percentage points. With the exception of age and marital status, there was little change (<5 percentage points) in the sociodemographic and behavioural profile of high-risk patients, irrespective of model or high-risk threshold. Although not all population characteristics were explicitly measured in all models (table 1), this did not consistently affect their prevalence in high-risk groups. For example, functional status was measured explicitly only in the full model, but individuals with severe functional limitations accounted for at least one-third of high-risk quintile patients (32.3%–45.5%) across models. In general, patient characteristics with high ORs (>1.3) were the most sensitive to modelling specifications. (See online supplementary appendix C for ORs and online supplementary appendix B for selected case-mix characteristics by model and threshold).

Supplementary file 3

Table 2

Variability in case mix among patients identified as ‘high risk’ when varying modelling approach or high-risk admission thresholds

Figure 3 illustrates that four in five (78.8%) high-risk individuals for the full model and three of four (74.3%) in the hospital model remain persistently high risk when re-evaluated over a 2-year period. For these models, approximately 20%–25% of high-risk group members gradually lose high-risk status throughout the first year following identification. High-risk group membership stabilises thereafter.

Figure 3

Persistence of high-risk group status over time. Per cent of original HR quintiles that are high risk in subsequent 24 months. GP, general practice; HR, high risk.

Reduction in risk status results from individuals no longer meeting the ‘prior ED’ or ‘prior hospitalisation’ criteria, which are used to assess risk in the full and hospital models. By contrast, all of the GP high-risk quintiles (100%) remained persistently high risk, because only age changed over the 2-year period. Unlike the other two models, the GP model did not include utilisation history, and all other risk factors were measured solely at baseline. While advancing age increases risk status, this analysis focuses on individuals already at highest risk. As a result, for a fixed survey cohort, high-risk status did not vary over time. In the GP+ model that included GP usage history, 12.9% of high-risk individuals identified with the GP+ model lost high-risk status over 2 years.


To effectively leverage PRM as part of the implementation of medical home, integrated care and other quality improvement efforts, stakeholders need to know whether they are targeting the ‘right’ patients and to understand how well health systems are currently performing for them. Who is the ‘right’ patient depends on the clinical or measurement context. Our study demonstrated sensitivity of case mix and risk persistence to PRM specifications, resulting in somewhat different target populations with different hospitalisation risks. One cannot rely solely on predictive performance to assess model suitability for either clinical or evaluative purposes.

Among our population-based cohort, all three models resulted in high-risk quintiles that included relatively low cut points for the risk of admission (12%–13%), with one in four individuals subsequently hospitalised. The large number of people with lower risk scores meant that the high-risk population was sensitive to varying risk predictors and thresholds. As high-risk thresholds increased, case mix was increasingly characterised by high proportions of very old, frail elderly, multimorbid individuals, and those who report fair/poor health. At the 95th percentile, for example, more than one in four high-risk group members had a history of repeat hospitalisations and nearly 1 in 10 would die during the prediction year. Differences between high-risk groups reflected the strongest PRM predictors, highlighting the need to ensure the modelling specifications are optimised to ‘find cases’ appropriate for the intervention.

Risk status may be conceptualised as something that changes over time (eg, as an outcome) or it may be thought of as a relatively stable characteristic of a patient or a population. How long a person remained high risk depended on the PRM’s relative reliance on factors that could change over time—only age and recent utilisation history in our models. Risk status was least stable in PRMs that incorporated prior hospital use. It is common in clinical settings to have access to both time-variant and time-invariant patient data. Some patient information is regularly updated (eg, utilisation history, age). Other patient data do not typically change (gender, race/ethnicity). Still other information may be collected once for a specific purpose or infrequently reassessed (eg, eligibility screening data).

Case-finding implications

Case-finding PRM applications seek to target interventions by identifying patients at risk of hospitalisation. Our results demonstrate that fully or partially linked data identified high-risk patients who would not have been classified as such using GP data alone—for example, superutilisers with multiple prior hospitalisations. In a GP setting, such patients may not have been targeted for closer monitoring, especially if they were out of care or new to the practice. Sharing actionable information across data settings clearly improves case-finding potential and is often an explicit motivation for PRM.

Whether a patient’s high-risk status is relatively more persistent or episodic also has bearing on case-finding. For example, if risk manifests episodically, clinical interventions may be short term and time sensitive. To identify new at-risk patients in near real time, the PRM base population may need to be updated and reassessed more frequently. In practice, not all variables commonly used in PRMs are available on a real-time basis, including those (eg, hospitalisation history) that influence risk persistence.8 If data lags are long, the window of opportunity for intervention may close before providers become aware of a change in risk status.

Performance measurement implications

Additional considerations apply for the use of PRM in performance measurement applications. Here, the aim is to define high-risk population segments, for example, to understand how well health systems have been performing for high-risk populations and evaluate the impact of new models of care. PRM-defined population segments could be used to describe the regional distribution of high-risk populations and to assess past and current service use, morbidity, mortality and other medical home/integrated care outcomes. Fair and appropriate performance measurement therefore requires a reasonable match between target populations identified via case-finding and corresponding performance measurement populations. Yet in our findings, only three-quarters of patients, at best, were commonly identified using data available in different settings. While the minimum degree of agreement necessary for analysis may differ across performance measurement applications—for example, programme evaluation may differ from ongoing health system reporting—it is clear that clinical intervention and performance populations will not be fully equivalent unless they are using exactly the same data.

Even with identical data, pragmatic data analysis decisions could inadvertently introduce bias. For example, case-finding applications often set risk thresholds high due to clinical capacity constraints,39 while performance measurement applications may prefer larger risk groups to ensure adequate sample size for analysis. In our simulation, morbidity and mortality dramatically increased as risk thresholds increased. Therefore, creating a larger analysis population by relaxing the PRM high-risk threshold would increase sample size (intended) and change case mix (unintended).

Stratified approaches to performance measurement also need to reflect on the persistence of high-risk status relative to the outcome assessment time frame. Stratified analysis is well suited for short-term outcomes that are measured soon after patients are classified as high risk, such as care planning or readmission. However, medical home and integrated care initiatives often aim to reduce long-term hospitalisation rates and costs, measured years later.40 In our simulation, up to 20% of high-risk patients in the full model and 25% identified via the hospital model would change risk status within a year of identification. From a performance measurement perspective, this means that high-risk patients who become lower risk over time—whether due to usual care or programme interventions—would no longer be counted among the ‘high-risk’ strata for which outcomes are reported, unless methodological steps were employed to retain them.

Theoretically, variation in care could also be reflected in risk status, and persistence of high-risk status could potentially be explored as an outcome of care to be assessed as part of performance measurement. The GP model, with regularly updated risk factor information related to health behaviours and self-rated health, might work well for this purpose. By contrast, the hospital model is highly endogenous, with recent hospitalisation history a key determinant in establishing current (hospitalisation) risk status. Because risk scores in this instance track with recent hospital use, monitoring hospitalisation patterns may provide a more direct means for benchmarking and evaluating variation in care. Given the predominantly time-invariant variables in our study, we did not attempt to investigate the utility of risk status persistence as an outcome but highlight it as a promising area for future research.

Regional differences may emerge in stratified performance outcomes or in high-risk patients’ rates of return to lower risk status. Designing meaningful comparisons of performance across sites will require careful consideration. Within the broader high-risk population segment, case mix is likely to differ by clinical site, especially if they implement PRMs tailored to their populations. Even with a standard PRM, known regional variations in demographics and health status will likely result in some sites having high-risk groups dominated by frail elderly, others by superutilisers and still others by the multimorbid. For certain performance comparisons, it may be necessary to risk-adjust (within risk strata) or further stratify (eg, by subgroup) to account for this heterogeneity. This area is also ripe for additional research.


This research was completed using data collected through the 45 and Up Study (www. saxinstitute. org. au). The 45 and Up Study is managed by the Sax Institute in collaboration with major partner Cancer Council NSW; and partners: the National Heart Foundation of Australia (NSW Division); NSW Ministry of Health; NSW Government Family and Community Services–Ageing, Carers and the Disability Council NSW; and the Australian Red Cross Blood Service. We thank the many thousands of people participating in the 45 and Up Study. The authors also thank the NSW Ministry of Health for access to Admitted Patient Data Collection and the Emergency Department D ata Collection, the NSW Register of Births Deaths and Marriages for access to death data, the Australian Department of Health for access to the Medicare Benefits Schedule and the Pharmaceuticals Benefits Scheme datasets, and the Centre for Health Record Linkage for facilitating the record linkage.


  1. 1.
  2. 2.
  3. 3.
  4. 4.
  5. 5.
  6. 6.
  7. 7.
  8. 8.
  9. 9.
  10. 10.
  11. 11.
  12. 12.
  13. 13.
  14. 14.
  15. 15.
  16. 16.
  17. 17.
  18. 18.
  19. 19.
  20. 20.
  21. 21.
  22. 22.
  23. 23.
  24. 24.
  25. 25.
  26. 26.
  27. 27.
  28. 28.
  29. 29.
  30. 30.
  31. 31.
  32. 32.
  33. 33.
  34. 34.
  35. 35.
  36. 36.
  37. 37.
  38. 38.
  39. 39.
  40. 40.


  • i There is no single generally accepted clinical or operational definition of frailty. However, many conceptual frameworks and measures incorporate older age, poor functional status and a history of falls.

  • Contributors TLJ conceived the study, undertook the literature review, implemented the regression models, conducted descriptive analyses and wrote the first draft. JK provided important intellectual contributions to study design and led the statistical and descriptive analyses. MOF produced some of the regression predictors, contributed substantively to the analysis and intellectual content, and provided detailed revisions throughout the article drafting process. JH conducted analyses and critically reviewed the methodology sections of the paper. KS, LRJ and JFL served as project mentors, providing significant ideas to the development of the model and important intellectual content throughout the draft revision process. All authors have read and approved the final version of the manuscript.

  • Funding TLJ was supported by the Australian-American Health Policy Fellowship 2016-2017 from the Centre for Health Economics Research and Evaluation, University of Technology Sydney, supported by a grant from the Commonwealth of Australia as represented by the Department of Health (DOH). JH was employed as a trainee on the NSW Biostatistics Training Program funded by the NSW Ministry of Health.

  • Disclaimer The information and opinions contained do not necessarily reflect the views or policy of the Department of Health. Neither the NSW Ministry of Health nor the DOH had any role in study design; in the collection, analysis and interpretation of data; in the writing of the report; nor in the decision to submit the articlefor publication.

  • Competing interests All authors have declared grants from the Commonwealth Department of Health (TLJ, LRJ, MOF) and Centre for Health Economics Research and Evaluation (TLJ), infrastructure support from the Bureau of Health Information (TLJ, JFL, KS, JH) and NSW Ministry of Health (JH), and LRJ is a member of the governing board of the Bureau of Health information; research results may inform evaluation/performance measurement for the Department of Health and Bureau of Health Information.

  • Patient consent Obtained.

  • Ethics approval The 45 and Up Study was approved by the University of New South Wales Human Research Ethics Committee (HREC). Additionally, this research, inclusive of data linkage, was approved by the NSW Population and Health Services Research Ethics Committee (HREC/15/CIPHS/42, 04/10/16).

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Data sharing statement The patient level data from 45 and Up Survey are available to researchers according to its governance framework. See for further details.

  • Author note The fellowship and traineeship were hosted by the Bureau of Health Information (BHI).