Study design

This retrospective quasi-experimental study used data from the Integrated Population Health Management (PHM) database from the National Healthcare Group Health Services & Outcomes Research department. The Integrated PHM database contains data from NUH and three of other six major public acute hospitals in Singapore (Ng Teng Fong General Hospital, Alexandra Hospital and Tan Tock Seng Hospital), and includes data from over 2.3 million individuals.

One-year pre- and post-enrolment healthcare utilisation data from all four hospitals were extracted from the database for the time period of January 2013 to December 2016. The extracted data were for patients enroled in the NICE programme or NUHS TCP from June 2014 to December 2015, and also for potential controls that had at least one all-cause inpatient admission – includes emergency admissions – in NUH over the same period. This study was a component of a larger mixed-methods study to evaluate the process and outcome measures of the NICE programme and NUHS TCP.23

Patient and public involvement

Patients and the public were not involved in the designing or conducting of this study.

Setting

Singapore is a city-state nation located in Southeast Asia with a total population of 5.6 million, of which over 3.9 million are citizens and permanent residents.24 The population has an ethnic composition of 74.1% Chinese, 13.4% Malays, 9.2% Indians and 3.3% Others,25 and a life expectancy of 82.9 years.24 For the time period from which the healthcare utilisation data were extracted (January 2013 to December 2016), NUHS RHS was one of six RHS in Singapore. The funding for the NICE programme and NUHS TCP was supported by the Ministry of Health as two of its six nation-wide priority areas, covering the management of frequent hospital utilisers and the timely discharge from acute hospitals through transitional care, respectively.

Intervention

For both the NICE programme and NUHS TCP, there were variations in work processes and outputs due to the profiles of patients served, but in general they had similar objectives and workflows (table 1). For example, each of the programmes had teams conducting follow-up home visits and phone calls to monitor and manage patients’ post-discharge. Patients were referred to community-based care and services (for example, home nursing, home hospice, outpatient clinic, social workers and corporate volunteers) when necessary and appropriate. However, both programmes had their unique enrolment criteria. Patients with cancer enroled in the NICE programme (n=87) or NUHS TCP (n=30) were not included in this study, as these patients had a high post-enrolment mortality rate (over 65%). Additionally, cancer diagnosis information was not available in the Integrated PHM database and so potential control patients could not be appropriately selected to match on cancer types.

NICE programme

Patients with three or more inpatient admissions within a year and who had associated risk factors for frequent readmission such as functional decline, cognitive impairment and multiple comorbidities could be enroled under the NICE programme. Patients were assigned to a team of case managers who customised the care programme based on the needs of the patients and caregivers. Case management commenced during the inpatient episode and could continue for up to a year post-discharge from hospital, after which, a proper handover process could be carried out to the next healthcare provider to ensure continuity of care.

NUHS TCP

Patients who required high acuity post-discharge care and/or were home-bound with mobility issues but had an identified caregiver could be enroled under the NUHS TCP. Each patient was assigned to a care team led by a transitional care nurse. Patients were enroled for a post-discharge period of up to 3 months, and services included medical, nursing, rehabilitative and psycho-social support to meet the needs of the patients and caregivers.

Population

Four groups of patients were generated for comparisons: two intervention and two control groups. Patients enroled in the NICE programme and NUHS TCP from June 2014 to December 2015 were in the intervention groups, and these patients were matched to controls who were patients with at least one admission at NUH but not enroled in either programmes. Matching was conducted separately for the NICE programme and NUHS TCP, with one matched control for every intervention patient selected from the same pool of 50 855 potential controls by propensity score matching (detailed below).

Outcomes

Healthcare utilisation frequencies in the metrics of all-cause inpatient admission, emergency admissions, emergency department attendances, specialist outpatient clinic (SOC) attendances, LOS and the healthcare charges in the metrics of the full gross bill amounts from all-cause inpatient admission, emergency admissions, emergency department attendances and SOC attendances, were calculated for the 1 year period before and after the point of enrolment, which was usually at the end of an admission. Survival days were calculated by subtracting the point of enrolment from the date of death. Patients who survived beyond 1 year from the point of enrolment was assigned the maximal number of 365 days and were censoring events. Mortality rate for the 1 year period after the point of enrolment (1 year mortality) was also calculated.

For patients in the intervention group, the points of enrolment were actual enrolment dates into the NICE programme or NUHS TCP, which usually occurred when the patients were discharged from an admission. For controls, there was no actual point of enrolment into the intervention (or into a control group like a randomised control trial). Only discharge dates of admissions with at least one bed day in NUH which fell within June 2014 to December 2015 were considered as potential proxy for point of enrolment among the controls, in order to mimic the triggering event (ie, an inpatient admission to NUH) necessary for consideration for enrolment into either programmes among patients from the intervention group.

Matching for controls

To obtain a control for each patient in the intervention groups, we generated a population of potential proxies for point of enrolment that corresponded to the discharge dates of all admissions belonging to patients not in the intervention groups, where the discharge dates occurred between June 2014 and December 2015. Hence, patients who are not in the two intervention groups and have multiple admissions with discharge dates between June 2014 and December 2015 would contribute multiple potential proxies for point of enrolment. Although potential proxies for point of enrolment could belong to the same patient, the number of previous admissions in the 1 year period prior could vary across these potential proxies because they corresponded to discharge dates from different admission episodes. This provided a wide spread of potential proxies for point of enrolment over a period with varying pre-enrolment healthcare utilisation patterns. This approach could provide better match given the variety of patients from the intervention group with different points of enrolment and different pre-enrolment healthcare utilisation patterns. For example, patients could have chosen to be enrolled in the NICE programme after more than three all-cause inpatient admissions within a year, and patients enrolled in the NUHS TCP did not have to meet any criteria of minimum all-cause inpatient admissions a year for enrolment.

Propensity score matching26 was used to reduce bias in treatment effect estimates by reducing covariate imbalance between the intervention patients and control patients from quasi-experimental studies. Matching was conducted using the MatchIt package27 with R V.3.4.1 to generate a group of matched controls. The approach selected was nearest neighbour matching with a 1:1 ratio, whereby each case (a patient from the intervention group) was matched with the potential control (with a proxy for point of enrolment) with the closest propensity score that was generated from the matching variables used.26 The following were the matching variables used to construct the propensity score: age at enrolment, gender, ethnic group (Chinese, Malay, Indian, others), residential housing type (1- or 2-room, 3-room, 4-room, 5-room/executive, private/others; serving as proxy for socio-economical status28 29), comorbidities at enrolment (14 identified from the chronic disease management system30 as part of the database set-up: asthma, diabetes, hyperlipidaemia, hypertension, atrial fibrillation, heart failure, congestive heart failure, stroke, chronic obstructive pulmonary disease, chronic kidney disease, coronary heart disease, osteoporosis, spine fracture, hip fracture), (proxy) point of enrolment, and 1 year pre-enrolment all-cause inpatient admissions, emergency admissions, emergency department attendances, SOC attendances, LOS, all-cause inpatient admission charges, emergency admission charges, emergency department attendance charges and SOC attendance charges.

Statistical analysis

Statistical analysis was conducted using Stata V.14.2 (Stata Corp, College Station, Texas, USA). For count data of all-cause inpatient admissions, emergency admissions, emergency department attendances, SOC attendances and LOS, negative binomial regression was used to model these post-enrolment healthcare utilisation frequencies, adjusting for log-transformed pre-enrolment healthcare utilisation frequencies with a value of 0.5 added to the frequencies (to avoid taking the log of zero values which give invalid values) and propensity score.31 The model included an offset term to account for varying follow-up time due to death.31 Any clustering effect of different proxies for point of enrolment selected as controls and from the same patient was also accounted for with the clustered sandwich estimator which was used to obtain robust standard errors. The incidence rate ratios (IRRs) of the intervention group to the control group with their corresponding 95% CI were reported for the NICE programme and NUHS TCP respectively. Wilcoxon signed-rank test was conducted to compare pre- and post-enrolment utilisation frequencies for each type of group (intervention and control groups).

For continuous data of all-cause inpatient admission charges, emergency admission charges, emergency department attendance charges and SOC attendance charges, linear regression was used to model these log-transformed post-enrolment healthcare charges with a value of 0.5 added to the charges (to avoid taking the log of zero values which give invalid values), adjusting for log-transformed pre-enrolment healthcare charges with a value of 0.5 added to the charges, propensity score and the log-transformed survival days (or follow-up time). Any clustering effect of different proxies for point of enrolment selected as controls and from the same patient was also accounted for with the clustered sandwich estimator. The exponential of beta-coefficients, which will be termed mean ratios (MRs) of the intervention group to the control group, and their corresponding 95% CI were reported for the NICE programme and NUHS TCP, respectively.

For 1 year mortality, Cox proportional hazard regression analysis was performed, adjusting for propensity score, and any clustering effect of different proxies for point of enrolment selected as controls and from the same patient was accounted for with the clustered sandwich estimator. The hazard ratios (HRs) of the intervention group to the control group with their corresponding 95% CI were reported for the NICE programme and NUHS TCP, respectively.

Additional analyses were conducted to test for the presence of interaction effects between the intervention and sociodemographical variables. The age group variable was dichotomised into patients who were aged 60 years and above (≥60 years), and those who were aged below 60 years (<60 years). The residential housing type was used as proxy for socio-economical status28 29 with five categories: 1- or 2-room, 3-room, 4-room, 5-room/executive and private/others. The sociodemographical variables (gender, age group and residential housing type) were added separately into the regression models for healthcare utilisation frequencies and charges with their corresponding interaction with the intervention variable. This allowed us to assess whether the sociodemographical variable could be an effect modifier in the association of the intervention with healthcare utilisation frequencies and charges. If the interaction term was significant, the intervention effect for each category of the sociodemographical variable was calculated and presented.

NICE programme

Propensity score matching identified 554 controls for the 554 intervention patients enroled in the NICE programme. Among the 554 controls, there were 500 unique control patients, where 456 control patients contributed one unique proxy for point of enrolment, 35 control patients contributed two unique proxies each, eight control patients contributed three unique proxies each, one control patient contributed four unique proxies each and each of these 554 unique proxies was matched to a unique intervention patient. Visual review of the propensity score distribution plots for cases, matched controls and unmatched control pool of patients indicate the close distribution of propensity scores of cases and matched controls (online supplementary data 1). Demographical information and pre-enrolment hospital utilisation frequencies and charges of the two groups are presented in table 2.

No significant differences were observed between the intervention patients enroled in the NICE programme and control patients for post-enrolment all-cause inpatient admission charges, emergency admission charges, emergency department attendance charges, SOC attendance charges and LOS (table 3). The IRR for emergency admissions (IRR=1.15, 95% CI 0.99 to 1.33) and SOC attendances (IRR=0.90, 95% CI 0.80 to 1.01) both did not reach statistical significance. Significantly higher all-cause inpatient admission charges (MR=2.38, 95% CI 1.44 to 3.92), emergency admission charges (MR=2.71, 95% CI 1.60 to 4.58) and emergency department attendance charges (MR=2.17, 95% CI 1.50 to 3.13) among intervention patients were observed when compared with the control patients. SOC attendance charges among intervention patients were not statistical different from the control patients (MR=1.31, 95% CI 0.96 to 1.79).

For all-cause inpatient admission charges, emergency admission charges and emergency department attendance charges, the interaction term between intervention status and gender was significant (online supplementary data 2). Female intervention patients had significantly higher all-cause inpatient admission charges (MR=3.99, 95% CI 1.87 to 8.52), emergency admission charges (MR=5.47, 95% CI 2.50 to 12.00) and emergency department attendance charges (MR=3.39, 95% CI 1.95 to 5.89) than the female control patients.

Within intervention patients and within control patients, the post-enrolment all-cause inpatient admissions, emergency admissions, emergency department attendances and LOS were significantly lower than the pre-enrolment counterparts, indicating regression to the mean. The post-enrolment SOC was significantly higher than the pre-enrolment SOC among the control patients only. The mortality hazard of the intervention patients was 26 (95% CI 5 to 41) per cent lower than the control patients.

NUHS TCP

Propensity score matching identified 270 controls for the 270 intervention patients enroled in the NUHS TCP. Among the 270 controls, there were 266 unique control patients, where 262 control patients contributed one unique proxy for point of enrolment, four control patients contributed two unique proxies each and each of these 270 unique proxies was matched to a unique intervention patient. Visual review of the propensity score distribution plots for cases, matched controls and unmatched control pool of patients indicate the similar distribution of propensity scores between cases and matched controls (online supplementary data 3). Demographical information and pre-enrolment hospital utilisation frequencies and charges of the two groups are presented in table 4.

No significant differences were observed between the intervention and control patients for post-enrolment all-cause inpatient admissions, emergency admissions, emergency department attendances, SOC attendances and LOS (table 5). Significantly higher post-enrolment emergency admission charges (MR=2.64, 95% CI 1.15 to 6.06) and post-enrolment SOC attendance charges (MR=4.50, 95% CI 2.82 to 7.18), among intervention patients were observed when compared with the control patients.

For SOC attendances, the interaction term between intervention status and age group was significant (online supplementary data 4). Intervention patients that are ≥60 years had a significantly lower SOC attendance rates (IRR=0.80, 95% CI 0.67 to 0.94) than matched controls that are ≥60 years. The same interaction term was also significant for SOC attendance charges, where both younger and older age groups had a significantly higher SOC attendance charges (<60: MR=21.64, 95% CI 9.17 to 51.06; ≥60: MR=2.78, 95% CI 1.63 to 4.74) than their respective matched control patients. For LOS, the interaction term between intervention status and gender was significant, where male intervention patients had a significantly higher LOS (IRR=1.73, 95% CI 1.06 to 2.84) than male control patients. The interaction term between intervention status and housing type was also significant for LOS, where intervention patients living in 3-room housing had significantly lower LOS (IRR=0.49, 95% CI 0.25 to 0.97) than control patients living in 3-room housing, and intervention patients living in private/other housing types had significantly higher LOS (IRR=2.28, 95% CI 1.08 to 4.79) than control patients living in private/other housing types. Only the interaction term between intervention status and housing type was significant for all-cause admission charges, where intervention patients living in private/other housing types had a significantly higher LOS (MR=10.34, 95% CI 1.71 to 62.57) than control patients living in private/other housing types.

Within intervention patients and within control patients, the post-enrolment all-cause inpatient admissions, emergency admissions, emergency department attendances and LOS were significantly lower than their pre-enrolment counterparts, indicating regression to the mean. Within intervention patients and within control patients, the post-enrolment SOC attendances was significantly higher than the pre-enrolment SOC. The mortality hazard (HR=1.41, 95% CI 0.97 to 2.05) was not significantly different between the intervention and control groups.

Principal findings

The post-enrolment all-cause inpatient admissions, emergency admissions, emergency department attendances, SOC attendances and LOS were not statistically different in patients enroled in the NICE programme or NUHS TCP, when compared with their matched controls. However, the all-cause inpatient admission charges, emergency admission charges and emergency department attendance charges were significantly higher in patients enroled in the NICE programme, and the emergency admission charges and SOC attendance charges were significantly higher in patients enroled in the NUHS TCP, when compared with their matched controls. Female patients enroled in the NICE programme had significantly higher all-cause inpatient admission charges, emergency admission charges and emergency department attendance charges than the female matched control patients. Older patients (aged 60 years and above) enroled in the NUHS TCP had significantly lower SOC attendances but had significantly higher SOC attendance charges, while male patients enroled in the NUHS TCP had significantly higher LOS, compared with their respective matched controls. Patients living in private/other housing types enroled in the NUHS TCP also had significantly longer LOS and thus incurred greater all-cause admission charges than the matched control patients living in private/other housing types. Patients enroled in the NICE programme, of which over 30% had history of heart failure, had lower mortality than their matched controls, mirroring findings from another study with heart failure patients.13

Strengths

For this study, the follow-up period of 1 year allowed us to evaluate the long-term effects of the programmes. The use of healthcare administrative data avoided recall errors or missing data issues that would arise from self-reported healthcare utilisation, particularly for a long follow-up period. The healthcare administrative data contained patient-level information extracted from a large consolidated database of electronic records from four public hospitals where separate episodes of care across different hospitals were linked to each patient. Although this study did not include data from the other three major public hospitals and all other private acute hospitals in Singapore, patients having an encounter with a specific acute hospital would likely return to the same acute hospital for follow-up care, as previous research has found that less than 10% of patients have cross-utilisation across different hospitals.17

The use of patient-level administrative data allowed for selection of matched controls, which would otherwise not be possible if the administrative data was aggregated at hospital- or national-level. Previous evaluation of a transitional home care programme in Singapore32 found that post-enrolment SOC attendances rates had increased from pre-enrolment rates, where a lack of a comparison control group limited the attribution of the increased SOC attendances to the intervention programme.33 For this current study, the post-enrolment SOC attendances had increased among patients enroled in the NUHS TCP and their matched controls too.

The availability of matched controls allows comparisons to take into account the regression to the mean phenomenon that could occur in both groups even without the intervention.22 34 Comparison between the intervention and matched control groups allow for the partitioning of the intervention programmes’ effects on utilisation frequencies and charges. Post-enrolment all-cause inpatient admissions, emergency admissions, emergency department attendances and LOS all regressed to a lower mean following an acute period of high hospital utilisation pre-enrolment, for patients in both programmes and their matched controls. These findings suggest the importance of a comparator group to assess whether changes before and after an intervention can in fact be attributed to the intervention programme.

Limitations

The traditional ‘gold standard’ for assessing an intervention programme would conventionally be to conduct a randomised control trial (RCT). However, as both the NICE programme and NUHS TCP were being rolled out hospital-wide, the operational demand and timelines for implementing these programmes at such a large scale could not allow for a RCT to be conducted. Additionally, the use of retrospective study design for the real-world evaluation of other similar interventions have been previously used32 35 36 and recommended.37–39 Although patients were not randomised into intervention and control groups, inherent bias from confounding factors was mitigated with matching. However, the matching of controls was limited to only information that was recorded and available in administrative databases. Ambulation status and availability of a caregiver at home were part of enrolment criteria for NUHS TCP, but this information was not available for potential controls to be matched on. Even though intervention patients were matched with control patients on comorbidities, our matching was limited in not matching on the diagnoses and reasons for admissions, as well as the specific post-discharge care needs. Additionally, there was a possibility that there were potentially other intervention programmes which some patients were exposed to but such scenarios were not possible to track and account for.

The NICE programme and NUHS TCP each catered to a variety of patient profiles and disease complexities. The variability in patient needs within the programmes could potentially lead to implementation inconsistencies with care provided by different teams, particularly when the programmes implemented in the real-world evolve organically with time. Based on the evaluation findings of other similar interventions, and earlier pilot or equivalent versions by other health systems in Singapore, NUHS TCP was expected to reduce the rates of overall all-cause inpatient admissions,13 14 16 40 emergency admissions41 and emergency department attendances,41 and also reduce all-cause inpatient admission charges,13 40 even if LOS was not expected to improve.41 However, these previous interventions were applied to selective patient populations of elderly above 65 years of age, with heart failure, and/or with chronic obstructive pulmonary disease, whereas NUHS TCP cared for a more heterogeneous group of patients.

Implications for clinicians and policymakers

The lack of reductions in all-cause inpatient admissions, emergency admissions, emergency department attendances, SOC attendances and LOS for patients in the NICE programme was not unexpected. Previous research on similar large-scale post-discharge care efforts targeting frequent hospital utilisers,42 43 have similarly found limited impact of the interventions within such timeframes. This could be due to the fact that frequent hospital utilisers are a unique group of patients experiencing an extreme acute period of high medical need. Previous research has also demonstrated the regression to the mean phenomenon, with hospital utilisation dropping in the following year after a year of frequent hospital admissions.22 34 Any intervention implemented would thus have to bring about additional benefits beyond this decrease from the regression to the mean. The intervention could thus be more effective if selectively applied to patients assessed to have better prognosis but at risk to continue as frequent hospital utilisers, such as patients who are persistent high utilisers.44–46 With close to 30% of these frequent hospital utilisers (and their matched controls) having less than 1 year of survival, there might be a limit to what can be done to intervene and reverse the course of illness. If these patients could be accurately identified, tailored programmes could instead focus on end-of-life palliative care and management.

The NICE programme and NUHS TCP might be considered large multi-disciplinary interventions, but at their cores, the healthcare services provided as part of these interventions were still centralised within hospital-based teams. The traditional hospital, as we know it today, could be considered to be insufficient in meeting the needs of patients with chronic diseases and multiple comorbidities in the community.47 Singular intervention-based adjustments to the existing system hence might not be as impactful in effecting change without an overhauling transformation and restructuring of the public hospitals and healthcare system. In Singapore, there are about 1700 private primary care physician clinics that cover about 80% of the total primary care demand48 but handle about 60% of the chronic disease-related attendances in primary care.49 The public hospitals can foster stronger partnerships with private primary care physicians to have integrative co-management of patients post-discharge, which would help facilitate shifting from care in hospital to care based in the community and even homes for patients.

Unanswered questions and future research

The higher costs incurred by the programmes could potentially be due to increased case-finding,50 51 with close monitoring by case managers in the programmes leading to uncovering of previously undetected problems and directing of patients for further assessment and treatment. Such case finding, and increased further assessments and treatments might also occur more with certain sociodemographical characteristics of the patient population, whereby the younger, female, and more well-off patients are more receptive to take follow-up action, and thus have higher healthcare utilisation. Increased case-finding could also explain the improved mortality, as patients may potentially have returned to the acute hospital in time and received additional medical procedures that contributed to the patients’ survival, but not avert higher costs. It should be noted that charges assessed in this study did not include the additional costs incurred to run the NICE programme and NUHS TCP. Future cost-benefit analysis could be conducted to determine if the decrease in mortality for patients enrolled in the NICE programme outweighs the cost to run the programme and the increased healthcare costs.

There is also a crucial need for qualitative implementation fidelity evaluation work set a prori in order to further interpret the findings from healthcare utilisation frequencies and charges outcomes research pertaining to specific programmes. Additionally, evaluation studies should shift beyond just healthcare utilisation frequencies and charges outcomes, and focus instead on outcomes of patients’ health confidence, quality of life and quality of care received.52 This is particularly important when evaluating programmes in their early stages, and determining if they have achieved their objectives. Future work should also explore longer follow-up duration and continue through more mature stages of a programme. Such evaluation studies are important to help inform hospital administrators and policymakers on how to refine these programmes and any other similar efforts towards more integrated care.