You are here

Article Text

Article menu

Download PDFPDF

XML
Health services research
Protocol
Realist evaluation of a complex integrated care programme: protocol for a mixed methods study
  1. Milawaty Nurjono1,
  2. Pami Shrestha2,
  3. Alice Lee2,
  4. Xin Ya Lim1,
  5. Farah Shiraz2,3,
  6. Shermin Tan2,
  7. Shing Hei Wong2,
  8. Kah Mun Foo2,
  9. Thomas Wee2,
  10. Sue-Anne Toh2,
  11. Joanne Yoong1,3,4,
  12. Hubertus Johannes Maria Vrijhoef5,6,7
  1. 1 Centre for Health Services Research and Policy Research, Saw Swee Hock School of Public Health, National University Health System, Singapore
  2. 2 Regional Health System Planning Office, National University Health System, Singapore
  3. 3 Saw Swee Hock School of Public Health, National University of Singapore, Singapore
  4. 4 Center for Economic and Social Research, University of Southern California, California, USA
  5. 5 Vrije Universiteit Brussels, Brussels, Belgium
  6. 6 Panaxea B.V., Amsterdam, The Netherlands
  7. 7 Department of Patient and Care, University Hospital Maastricht, Maastricht, The Netherlands
  1. Correspondence to Milawaty Nurjono; ephmn{at}nus.edu.sg

Abstract

Introduction The lack of understanding of how complex integrated care programmes achieve their outcomes due to the lack of acceptable methods leads to difficulties in the development, implementation, adaptation and scaling up of similar interventions. In this study, we evaluate an integrated care network, the National University Health System (NUHS) Regional Health System (RHS), consisting of acute hospitals, step down care, primary care providers, social services and community partners using a theory-driven realist evaluation approach. This study aims to examine how and for whom the NUHS-RHS works to improve healthcare utilisations, outcomes, care experiences and reduce healthcare costs. By using a realist approach that balances the needs of context-specific evaluation with international comparability, this study carries the potential to address current research gaps.

Methods and analysis This evaluation will be conducted in three research phases: (1) development of initial programme theory (IPT) underlying the NUHS-RHS; (2) testing of programme theory using empirical data; and (3) refinement of IPT. IPT was elicited and developed through reviews of programme documents, informal discussions and in-depth interviews with relevant stakeholders. Then, a convergent parallel mixed method study will be conducted to assess context (C), mechanisms (M) and outcomes (O) to test the IPT. Findings will then be analysed according to the realist evaluation formula of CMO in which findings on the context, mechanisms will be used to explain the outcomes. Finally, based on findings gathered, IPT will be refined to highlight how to improve the NUHS-RHS by detailing what works (outcome), as well as how (mechanisms) and under what conditions (context).

Ethics and dissemination The National Healthcare Group, Singapore, Domain Specific Review Board reviewed and approved this study protocol. Study results will be published in international peer-reviewed journals and presented at conferences and internally to NUHS-RHS and Ministry of Health, Singapore.

  • Realist Evaluation
  • Integrated Care
  • Mixed Methods Evaluation

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/

https://doi.org/10.1136/bmjopen-2017-017111

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Strengths and limitations of this study

    • This study is one of the first studies to holistically evaluate a new integrated care network in the context of Southeast Asia.

    • Using a theory-driven realist evaluation approach, findings from this study are expected to generate contextually relevant evidence for improving efficiency and effectiveness of integrated care as well as informing policy decisions in Singapore and similar health systems.

    • The use of mixed methods in this study allows us to draw on the strengths of both qualitative and quantitative methods, enhancing the credibility of the evaluation findings and allows for generation of in-depth insights.

    • Inclusion of relevant stakeholders in the design of the evaluation provides confidence that the evaluation efforts are relevant and findings are likely to be adopted for improvement in current models and for development of future initiatives.

    • Realist evaluation and mixed methods design are relatively new methods in comparison with the randomised controlled trials (RCTs) commonly used in the evaluation of medical interventions. Therefore, this study design may draw criticism for not being as rigorous as the typical RCT.

    Introduction   

    Across the world, rapidly ageing populations and increasing demand for healthcare services call for a paradigm shift from disease-centred healthcare provision to a more holistic, people-centred care.1 People-centred care involves placing people and communities at the focus of health systems, not diseases. It highlights the importance of empowering people to take charge of their health, takes into account holistic needs of a person and seeks to provide quality care timely and appropriately.2 Integrated care supports people-centred care by coordinating healthcare services in a way that ensures people receive a continuum of care at the different levels and sites of care within the health system, according to their needs. The WHO’s framework of people-centred integrated care describes the complexity of such interventions and the need to involve various stakeholders in their execution. Complex multicomponent delivery strategies are also typically recommended.1

    As characterised by Valentijn et al 3, care integration can take place on many dimensions: at the micro-level clinical level, the meso-level professional or organisational level and the macro or systemic level. The types of interventions that qualify under the wide umbrella of ‘integrated care’ vary tremendously, as do the settings in which they are introduced and their attendant benefits and costs. Moreover, introducing integrated care often calls for complex, multicomponent programmes that involve many stakeholders, sometimes with conflicting perspectives and interests. When implemented successfully, integrated people-centred health services can be more effective, cost less, improve patient engagement and better prepared to respond to increasing demands for healthcare services.4

    However, despite these growing needs and the availability of basic design principles related to integrated care, the development and implementation of such care models remains challenging. In their systematic review, Suter et al identified 10 universal principles for successful health systems integration including: (1) comprehensive services across the continuum of care, (2) person-centredness, (3) collaboration between organisations, (4) standardised care delivery through interprofessional teams, (5) performance management, (6) information systems, (7) organisational culture, (8) professional integration, (9) good governance and (10) financial management.5 Furthermore, it is acknowledged that successful implementation of integrated care requires an effective composition of interventions at the micro-level, meso-level and macro-level.6 7 However, the multifaceted interplay between the building blocks of integrated care and the influence of various contextual factors on outcomes render the development of a simple and standardised implementation model impossible.8 In every instance, there is a need for healthcare providers and organisations to understand which care models work, for whom and how they work in the unique setting in which integrated care is planned for a particular population so as to facilitate implementation and ensure longer term sustainability.9

    Current evaluations such as the ‘gold-standard’ randomised controlled trial (RCT) design rarely adequately or even explicitly address the context-specific drivers behind implementation outcomes and their relationship to the underlying programme theory, making it difficult to interpret their findings in light of other programmes in different settings. As a result, few evaluation strategies are widely accepted as appropriate, and to date, the net benefit of integrated care interventions and understanding of how variable outcomes are achieved remains empirically uncertain. It is therefore essential to develop comprehensive, rigorous and practical methods to evaluate people-centred integrated care programmes, to inform the selection of effective and efficient interventions and to facilitate improvement and scaling-up. In the evaluation of such complex interventions, the Medical Research Council (MRC) argues for the importance of process evaluation in conjunction with outcome evaluation, to account for variability in implementation.10 The MRC’s process evaluation framework guides evaluators to understand the implementation processes (what is implemented and how), mechanisms of intervention (how the delivery of the intervention produces change) and contextual factors that affect implementation and outcomes.11

    Singapore is typical of many other countries with a rapidly ageing population that is accompanied by an increasing prevalence of chronic and complex illnesses.12 Designed with an emphasis on providing episodic care within acute hospitals in a largely disease-specific manner and controlling infectious disease in a young population, Singapore’s health system now faces the challenge of ensuring appropriate care and long-term fiscal sustainability for a long-lived population at increasing risk of multiple chronic diseases. In addition, having already reached the highest levels of efficiency in the world,13 simple or easy fixes to increase performance while keeping costs low are increasingly rare. A radical change in vision and strategy is therefore needed, requiring complex systems-level interventions that bring about changes of organisational, policy, power and financing structures and break down existing siloes within the healthcare system to refocus on prevention, primary care and community-based management. Such interventions will need to be built on a strong foundation of integrated care.

    In 2012, the Ministry of Health (MOH) Singapore launched a major initiative to reorganise healthcare at the national level into six clusters or regional health systems (RHSs) to foster care integration.14 This was recently reorganised in 2017 into three integrated clusters to better meet future healthcare needs.15 Every RHS comprises of a network each led by a major public hospital working in close partnership with healthcare providers (primary care providers, community hospitals, nursing homes, home care and day rehabilitation providers) and social care providers (including senior activity centres, grassroots organisations and social service offices) within the same geographical region. Each RHS has the mandate and funding support to design and implement integrated care programmes that leverage this network to provide healthcare beyond the hospital to the community, value-driven healthcare and holistic care across the entire care continuum in a cost-effective way.16 To support this common vision, every RHS was tasked to implement programmes identified by the MOH to be of priority and held accountable for same outcomes jointly agreed on between the MOH and the RHSs throughout the country. Following this lead, each RHS implemented the priority programmes tailored to the unique needs and demographic of the population in which it serves as well as the different strengths of community partners within its network. Other programmes unique to respective RHS were also developed and implemented accordingly.

    In this study, we propose to take advantage of a unique opportunity to document and evaluate the formation and implementation of a multipillared, regional integrated care network. This evaluation study began in June 2016 at the National University Hospital System (NUHS)-RHS, the RHS at the western part of Singapore. This study aims to examine how and for whom the NUHS-RHS, as an integrated care network, works from healthcare providers’ and healthcare users’ perspectives to improve healthcare utilisation, health outcomes and care experiences as well as to reduce healthcare costs.

    Methods and analysis

    Considering the developmental nature of the NUHS-RHS, this evaluation is designed primarily as a formative evaluation that incorporates outcome valuation principles.17 The evaluation findings will be used to facilitate modifications to existing NUHS-RHS programmes as well as to provide evidence to support the increased likelihood of future success.

    The evaluation will be conducted using a realist evaluation approach,18 which seeks to test and refine the programme theory while assessing whether and how the programme succeeds in the local setting, in order to generate important insights for Singapore and for the wider field of integrated care research. Realist evaluation is increasingly applied in the evaluation of complex healthcare interventions as it seeks to provide a more explicit and in-depth understanding of what works, for whom and in what circumstances and has been recommended for the evaluation of integrated care interventions.19 20 It is a theory-driven approach in which interventions are assumed to be based on theories but are also active, flexible to changes and embedded in a social reality that influences how the intervention is implemented and how various actors in that reality respond to it.18 A realist programme theory specifies which outcomes are linked to the intervention and what mechanisms generate the outcomes and what features of the context affect them. The context–mechanism–outcome (CMO) configuration is used as the main structure for analysis, with the goal of identifying both mechanisms (what and how components of interventions result in changes) and contextual factors (features of the conditions that influence the mechanisms of interventions) are associated with variation in outcomes.21 22 Pawson and Tilley21 argue that an intervention can only achieve successful outcomes if the appropriate ideas are applied to the right context with appropriate social and cultural conditions. A realist evaluation therefore includes a theory-driven formative evaluation,23 process evaluation10 as well as outcomes evaluation and avoids the rigorously successionist format of experimental design.

    This evaluation will be conducted in three research phases according to the framework for realist evaluation outlined by Pawson and Tilley (figure 1)21: (1) development of initial programme theory (IPT) underlying the NUHS-RHS; (2) testing of programme theory using empirical data; and (3) refinement of IPT.

    Figure 1
    Figure 1

    Realist evaluation processes and research phases according to Pawson an d Tilley. CMO, context, mechanism and outcome; NUHS-RHS, National University Hospital System-Regional Health System.

    Phase 1: development of IPT

    This evaluation began with first eliciting the IPT which subsequently formed the basis of the evaluation, focus the evaluation activities and determined the appropriate study design, data collection and analysis methods. First, program-related documents of the NUHS-RHS that describe the rationales, components of interventions and programme protocols were reviewed to identify the underlying assumptions about how the respective programmes are expected to work to achieve their intended outcomes. Ethnographic observations were then conducted to verify and obtain additional information to comprehensively describe the theory of change underlying the NUHS-RHS programmes.

    Subsequently, key stakeholders involved in the planning, development and implementation of the NUHS-RHS programmes were engaged through group discussions to identify: (1) factors that influence the implementation of the programmes, (2) initial observations on how and for whom the programmes work, (3) objectives of evaluation and (4) outcomes that are relevant to the stakeholders. In collaboration with the RHS strategic planning office, a logic model (figure 2), a tool that describes logical linkages among programme resources, activities and intended outputs, audiences and short-term, intermediate-term and long-term outcomes24 related to the establishment of NUHS-RHS was drafted to depict IPT underlying the NUHS-RHS. To ensure the accuracy of the IPT, the logic model was circulated to stakeholders who were previously engaged for further inputs. After which, refinement to the IPT was made.

    Figure 2
    Figure 2

    Logic model of NUHS-RHS. NUHS-RHS, National University Hospital System-Regional Health System.

    As illustrated under activities on figure 2, programmes (P1–6) describes the MOH priority programmes implemented nationally. The NUHS-RHS Integrated Interventions and Care Extension (P1) programme aims to help patients with complex conditions who were admitted at least three or more times a year through a holistic case management. NUHS Transition Care Programme (P2) was implemented to enable patients to transit smoothly from hospital to homes through a multidisciplinary team to ensure continuity of care. Appropriate Sitting of Care (P3 and P4) aims to ensure appropriate sitting of patients who are medically stable and deemed to not required specialist care from acute hospital to the community through partnership with primary and community care providers. Health Promotion and Disease Management (P5) focuses on health education, early detection of chronic diseases and timely interventions to prevent development of chronic illnesses as well as slow down the deterioration of diseases. Strengthening of primary and community care within the NUHS-RHS (P6) is achieved through cross deployment of skilled manpower from the acute hospital to the partnering primary and community care providers so as to improve the capability and confidence of the partners of the NUHS-RHS. Leveraging on telehealth and IT enablers, the Carehub aims to provide a single point of contact through individual’s continuum of care posthospital discharge. The Primary Care Network brings together a group of family physicians and community partners to create a support network for patients with chronic medical conditions in the community.25

    Phase 2: testing of programme theory using empirical data

    Prioritisation of NUHS-RHS programmes for testing of programme theory

    As the number of programmes undertaken under the purview of the NUHS-RHS is large, programmes were selected for the testing of the IPT in consultation with stakeholders including healthcare providers and administrators, based on: (1) maturity of programme, (2) urgency for evaluation, (3) complexity of programmes and (4) scientific importance.

    Programme maturity was assessed based on the duration of programme implementation. A programme that had been implemented for a longer duration was considered to more mature and given higher priority than one which was recently implemented. Programmes nearing the end of their funding period were also given higher priority for evaluation, as evaluation was essential to inform the future development of the programme. Complexity was determined by assessing the number of dimensions of integrated care as defined previously (clinical, professional, organisational, systematic, normative and functional integration).3 A programme that set out to integrate care in more dimensions was considered to be more complex than one which integrated care in fewer dimensions.

    Finally, scientific importance was examined by a knowledge gap analysis conducted by reviewing available published articles relevant to these programmes on PubMed. A programme with the least number of relevant articles was considered to be of greatest scientific importance. According to these criteria, MN and HJMV ranked various NUHS-RHS programmes as low, medium and high priority for evaluation. Through the priority-setting process, four programmes (P1, P2, P3 and P4) were identified to be the best candidate programmes for evaluation.

    Study participants

    Two distinct groups of study participants—healthcare providers/managers and healthcare users—are selected based on defined criteria in this study. Healthcare providers/managers who are a part of the governance, planning and implementation of the programmes are invited to take part in structured interviews and to complete the online Rainbow Model of Integrated Care (RMIC) survey,26 while healthcare users, including patients or proxies (caregivers), are invited for structured interviews or prospective surveys over three time points. Interactions between healthcare providers and healthcare users will be observed.

    Study design

    Given the complexity of integrated care programmes being evaluated and a range of perspectives in which this study tries to capture, a convergent parallel mixed methods study will be undertaken to test the IPT developed in phase 1. Using the convergent parallel strategy, both quantitative and qualitative data will be collected concurrently. Components will be given equal weight, and two datasets will be analysed, compared and merged through iterative cycles of validation and confirmation of findings.27

    Evaluation of implementation, context and mechanisms

    Implementation fidelity, context and mechanisms that underpin the NUHS-RHS as a whole will be assessed according to the modified version of the Conceptual Framework of Implementation Fidelity (CFIF).28 The CFIF provides a framework to assess content, frequency and dose of programme implementation and the moderating factors affecting implementation fidelity of complex interventions. Moderating factors including participants’ responsiveness, intervention complexity, comprehensiveness of policy description, strategies to facilitate implementation, quality of delivery, recruitment and context will be assessed. Data will be gathered through reviews of programme documents, medical records, structured interviews healthcare providers and observations of actual delivery of interventions as described in table 1.

    View this table:
    Table 1

    Evaluation of implementation fidelity, context and mechanism underlying NUHS-RHS programmes

    Programme documents including guidelines and databases will be reviewed by evaluators. Programme guidelines will be examined to provide an understanding of the planned activities for each programme and the proposed mechanisms of achieving targeted outcomes. The number of eligible patients referred, response rate, reasons for non-enrolment into the programmes, frequency of programme delivery as well as duration of enrolment will be retrieved from the existing programme databases to determine the coverage, frequency and duration of respective programmes. Furthermore, medical records of patients enrolled into the programmes will be reviewed to provide a comprehensive picture of interactions between healthcare providers and users throughout a patient’s healthcare journey from enrolment to discharge. Medical records will be proportionately sampled based on the number of healthcare providers that are able to provide referrals within each programme and the time point at which patients are enrolled into the programmes.

    Healthcare providers (physicians, nurses and other allied health professionals) and healthcare managers (care coordinators and healthcare administrators) who are involved in the planning, development and implementation of the four programmes are invited to participate in face-to-face structured interviews. Written informed consent and demographic information including their age, gender, duration of involvement and roles will be collected prior to the interviews. Guided by the CFIF, content of interventions and moderating factors, which may have contributed or hampered the implementation of the interventions, will be assessed during interviews. All interviews will be audio-recorded, transcribed verbatim and coded thematically.

    Two sessions of ethnographic observations (induction and follow-up) for each study participant will be randomly conducted at patients’ homes and/or through phone calls depending on where and how the interventions are provided. We expect to observe two sessions each for approximately 30 participants from each programme and informed consent will be obtained from participants. Content of care delivery, interactions between healthcare professionals and users together with patients’ responses to the service will be observed. For each session, field notes will be taken and findings will be written in narratives and analysed thematically.

    Data analysis

    Data from the various sources will be given equal weightage and will subsequently be integrated at data analysis stage guided by the modified version of the CFIF using the triangulation protocol methodology.29 The extent of which the intervention components were implemented will be rated on a 5-point Likert scale ranging from never to always by two independent evaluators. Potential moderating factors that affect the implementation of the programmes will be assessed by examining participant responsiveness, comprehensiveness of protocols/guidelines, support available to facilitate recruitment and implementation, quality of service delivery and context (patient, professionals, organisational, political and economic factors). Analysis will also take into account other emergent themes not defined by the CFIF.

    Evaluation of outcomes

    Healthcare utilisation and health outcomes

    Healthcare utilisation, health outcomes and patient-reported experiences and outcomes associated with the NUHS-RHS programmes will be assessed through review of clinical indicators, structured interviews and administration of surveys as described below to identify alignment with the Chronic Care Model (CCM).30 The CCM identifies the community, the health system, self-management support, delivery system design, decision support and clinical information systems as key elements of a health system that are essential to provide good quality chronic care.30

    The effect of the programme on healthcare utilisation will be measured by analysing the number of admissions, emergency department (ED) attendance, number of specialist outpatient clinic (SOC) attendances and average bed days for patients enrolled from the start of the programmes. To capture the potentially evolving nature of the programmes and their effects, data will be collected over various time points with respect to their respective date of enrolment programme: 12 months before (T-12), 9 months before (T-9), 6 months before (T-6), 3 months before (T-3), date of enrolment (T0), 3 months after (T3), 6 months after (T6), 9 months after (T9), 12 months after (T12) and 15 months after (T15). Demographic data (gender, age, race, comorbidity and socioeconomic status) will also be retrieved for all patients.

    Clinical indicators at various time points for all patients who are enrolled and who declined will be extracted from the existing NUHS-RHS databases. In reference to the point of patients’ enrolment into the programmes, clinical indicators at T-12, T-9, T-6, T-3, T0, T3, T6, T9, T12 and T15 will be extracted.

    The effect of the programmes on patient-reported outcomes, patients’ quality of life, care experience and self-efficacy will be assessed using validated survey instruments over three time points: at the starting point, midway and the end of the programme. Quality of life of patients will be assessed using the EuroQol-5 Dimension (EQ-5D) instrument. The EQ-5D consists of five items, which recognises the fundamental importance of independent physical, emotional and social functioning, as part of a more holistic view of health. It has been validated in Singapore and is a common measure used to assess quality of life as the result of healthcare interventions.31 Patients’ perception of the quality of care received will be examined using the Patient Assessment of Care for Chronic Conditions (PACIC).32 The PACIC seeks to understand the frequency with which various aspects of care patients with chronic conditions received from their healthcare providers aligned with components of the CCM. The healthcare providers include regular doctors, nurse, care coordinators, allied health providers and others. PACIC has been widely validated and used at various healthcare settings across the world.33–36

    Patients’ experience of care continuity will be assessed using the CCAENA (continuity of care between care levels) questionnaire. The CCAENA consists of 29 items and assesses continuity of care across different levels from the patients’ perspectives.37 The components covered by the CCAENA include the relationships between patient and primary care providers and between specialists and primary care providers. Finally, self-efficacy is measured using the Patient Activation Measure, which consists of 13 items, and will be used to measure patient knowledge, skill and confidence for self-management.38

    Complementary to the quantitative analysis of outcomes, a more in-depth understanding of healthcare users’ perception of healthcare quality, including why and how it has worked, or did not work, will be explored qualitatively through interviews with healthcare users. Healthcare users including patients and their family members who have had experiences with the programmes for at least 3 months are invited to take part in face-to-face interviews. Interviews are structured according to the programme logic model and key components of the CCM.30 Perception of the level of care integration as well as recommendations for improvement to the programmes will also be elicited during the interviews. Potential participants will be recruited through care coordinators and will be screened against the selection criteria by the study team prior to the interviews. Only those who are eligible and are willing to be audio recorded will be interviewed. Written informed consent and demographic information will be attained from the participants before proceeding with the interviews. Interviews will be conducted in English, Chinese or Malay at the patients’ homes or at NUHS. Notes will be taken during observations, and all interviews are audio recorded, transcribed verbatim and will be imported to Atlas.ti V.7.0 for analysis.

    Data analysis

    An interrupted time series analysis (ITSA) will be conducted for all programmes to examine the effects of the NUHS-RHS programmes on healthcare utilisation, health outcomes and care experiences. In addition, difference in difference (DID) analysis will be conducted for those with comparison groups.39 40 Proposed to be more flexible compared with the traditional RCTs analysis, ITSA is conducted to estimate the changes associated with the introduction of interventions over time. To estimate changes in the above outcomes, time series regression that includes data from patients enrolled into the programmes 1 year prior to and after referral to the programmes will also be run. The model will have three main parameters: (1) estimation of the annual trend in healthcare utilisation (ie, hospital admission, SOC attendances, ED attendances and average bed days) and clinical indicators 1 year before until the introduction of the programmes; (2) estimation of changes in all outcomes associated with introduction of the programmes; and (3) the trend change in outcomes associated with each year. For each outcome, we will apply a generalised estimating equation model clustering by the combination of age, sex and comorbidity score with robust standard errors and an autoregressive correlation matrix. We will report rate ratios calculated from the parameter estimations, which represent population-averaged rates. DID analysis will subsequently be performed to estimate the impact of the programmes by comparing differential effects in the outcomes between intervention and comparison groups. Sensitivity analyses will be performed, and confounding effects will be accounted for.

    Observation notes and interview transcripts will be thematically coded by two independent researchers using a two-step approach according to the integrated approach as described by Bradley et al.41 We have selected this approach as this study adopts an existing CCM framework, but we would like to ensure completeness of findings by assuring other emergent themes not previously described in the CCM are also considered. The first step consists of a deductive analysis, where units of data are coded according to the CCM. This will be followed by an inductive analysis, where new themes or unexpected findings are elicited through coding and categorising. If the interviews were conducted in other languages beside English, these will be transcribed, and the analysis will be conducted by a researcher that is fluent in that language. The relevant quotes representative of the analysis will then be translated into English. Subsequently, data collected through prospective surveys and interviews with healthcare users will be merged using the CCM as a framework.

    Care integration

    The extent of care integration within the NUHS-RHS will be assessed through the administration of the RMIC measurement tool (MT) adapted to Singapore.26 42 Tested and validated in Singapore, the RMIC-MT was developed to measure the extent of care integration from the healthcare providers’ perspectives. It consists of 62 items grouped into eight factors of integrated care related to person-focused and population-based care, clinical integration, professional, organisational, systematic, functional and normative integration. The RMIC-MT uses a 4-point Likert scale ranging from ‘never’ to ‘all the time’ and an additional option of ‘not sure/don’t know’.26

    Data analysis

    Incomplete RMIC-MT will be excluded from analysis. The ‘not sure/don’t know’ option will be considered as missing data and will be imputed with median score of each item. The RMIC-MT will be scored as described previously by Nurjono et al in which the average score for respective dimensions of RMIC and overall care integration will be computed. A higher score on the RMIC-MT is considered to reflect a greater extent of care integration. The RMIC-MT was developed through a systematic literature review and validated by international Delphi panels.3 It describes six dimensions of care integration (clinical, professional, organisational, functional and normative integration) across different levels within a health system needed to provide a continuous, comprehensive and coordinated delivery of services to the individual and population. Furthermore, the degree of integrated care within NUHS will be compared with the degree of integrated care in other health systems (Netherlands and Australia) in which the RMIC-MT has been administered.

    Healthcare cost

    To examine the effect of NUHS-RHS programmes on healthcare costs, six categories of costs including (1) development costs, (2) programme implementation costs, (iii) healthcare utilisation costs (inpatient, outpatient and community), (4) productivity costs incurred to patients and/or caregivers, (5) travel costs borne by patients for travelling to receive care and (6) caregiving cost will be collected.43 Cost categories 1–3 are considered related to the healthcare system whereas categories 4–6 take into account costs from the societal perspective.

    Development costs include costs incurred during the development of the NUHS-RHS and their respective programmes. The programme implementation costs take into account costs related to the implementation of the programmes including the manpower costs, travel costs, costs of equipment and costs of materials used for patient and caregiver training and costs associated with multidisciplinary team meetings. The development and implementation costs will be systematically collected using WHO’s CostIt instrument44 and will be divided by the number of clients included into the programmes to obtain per patient cost.

    The cost of healthcare utilisations (hospital inpatient, outpatient services, primary and community care services), productivity costs (estimated based on absence from paid employment due to illness or providing care), travel costs (distance to healthcare providers and expenses incurred related to providing care to enrolled patients) and caregiving costs (costs related to the hire of domestic helpers, costs related to improvement in home environment and costs of caregiver training) will be collected using routinely collected hospitals data and a cost questionnaire administered to participants at T0 and T12 where each participant will be asked to recall his/her expenses 3 months prior to the time of the interview.

    Data analysis

    Differences in costs by category before and after enrolment will be analysed using paired Wilcoxon tests. The difference will be considered statistically significant if the p value is <0.05. Paired analysis will be adjusted by age, gender and number of comorbidities at baseline. To estimate the cost-effectiveness of the right-site care programmes, an incremental cost-effectiveness ratio (ICER) will be computed using changes in quality-adjusted life-years (QALYs) attributable to the programme as the primary measure of programme effectiveness, divided by changes in costs measured both from the health system and societal perspectives. The programme will be considered cost-effective if the ICER is lower than Singapore-specific thresholds. Currently, in the absence of a formal nationally accepted threshold for societal costs, we will adapt a default threshold consistent with WHO Choosing Interventions that are Cost-Effective (CHOICE) benchmarks of approximately $60–75 000 per QALY.45 The appropriate cost-effectiveness thresholds will be reviewed and updated to be consistent with national best practice at the time of completion of the analysis.

    Phase 3: refinement of IPT

    Finally, to provide an overall evaluation of the NUHS-RHS taking into consideration various perspectives, qualitative and quantitative data obtained from various stakeholders including the programme team members and patients/proxies will be integrated through the process of triangulation at the data interpretation stage when both quantitative and qualitative data have been analysed separately.29 The triangulation protocol will be adopted to guide data integration by first producing a convergence coding matrix according to the guiding conceptual frameworks to display findings emerging from each component followed by consideration of where there is agreement, partial agreement, silence or dissonance between findings from different data sources. Assessment of the fit of data integration will be conducted by examining the coherence of findings from various methods used, as suggested by Fetters et al.46 Data on CMO will be gathered and analysed guided by respective frameworks as described in phase 2. These will then be linked according to the realist evaluation CMO formula in which findings on context and mechanisms will used to explain outcomes observed. A few potential CMO configurations will then be proposed and discussed (for validation purposes) through 2–3 focus group discussions comprising of 8–10 different stakeholders each. After which, the IPT will be refined to highlight how to improve the NUHS-RHS by detailing what works (outcome), as well as how (mechanisms) and under what conditions (context).

    Ethics and dissemination

    Data collection for this study was started in June 2016 and is expected to continue until June 2019. The National Healthcare Group, Singapore, Domain Specific Review Board reviewed and approved this study protocol (DSRB Ref: 2016/00410 and 2016/00914). Written informed consent forms are obtained from participants (healthcare providers and users) included for interviews and surveys. To maintain confidentiality of the research participants, identifiable information obtained from research participants are kept securely under password protection. A unique respondent identification is assigned to each study participant so that data can be processed anonymously. Waiver of informed consent was obtained for analysis of retrospective data collected as part of the hospital and MOH routine data collection, and no identifiable data are known by the researchers. Study results will be written up and published in international peer-reviewed journals and will be presented at national and international conferences and internally at NUHS-RHS and the MOH, Singapore.

    Mitigation of potential risks and limitations

    While realist evaluation approach and mixed methods design have been advocated for the evaluation of complex healthcare interventions, they are relatively new in comparison with the RCTs commonly used in the evaluation of medical interventions. Due to the lack of familiarity on the use of realist evaluation and mixed methods in the field of health and healthcare research, this study design may draw criticism for not being as rigorous as the typical RCT. Furthermore, we acknowledge that the use of mixed methods design also increases complexity. Nonetheless, we strive to reach optimal integration of data at multiple levels—study design, methods, interpretation and reporting—using the convergent parallel mixed methods approach that connects and merges methods and findings.

    Furthermore, as participation in the evaluation efforts is voluntary and only individuals who are able or have proxies who are able to provide informed consent are included in this evaluation effort, we also recognise that selection bias may be introduced in this study. To account for selection bias, demographic information including age, gender, ethnicity and role (for healthcare providers) are collected and will be compared between responders and non-responders. Given the prospective nature of some of our data collection methods, difficulties in following up with respondents are also anticipated. To minimise the numbers lost to follow-up, research appointments are scheduled at participants’ time and place of convenience in collaboration with participants’ care coordinators. In the event when loss to follow-up is inevitable due to death, survival analysis will be conducted to account for the missing data.

    In addition, it must be acknowledged that as much as it was intended for the delivery of care to be consistent, variability in the extent of how care was provided exists, especially across different disease-specific subprogrammes and also within individual subprogramme. To account for such variability, subgroup analysis will be conducted, and adjustments based on the number of care providers within a subprogramme will also be made.

    Discussion

    Singapore is in the early phase of developing new integrated care models. Notwithstanding the scope of more general design principles of integrated care, development and implementation of integrated care require adaptation to the context, needs of the local population and the capacity and capability of local healthcare providers. To the best of our knowledge, this study provides the first opportunity to holistically evaluate the NUHS-RHS, and one of the first to conduct a comprehensive evaluation of a new integrated care network in the context of Southeast Asia using a realist evaluation approach.

    This study carries significant implications for the health system and society in Singapore as well as for the field of integrated care internationally. By ensuring the rigour of this evaluation and adopting a realist approach, findings from this study are expected to generate contextually relevant evidence for improving efficiency and effectiveness of the NUHS-RHS as it develops and also to inform future initiatives of the NUHS. It is also expected to yield policy-relevant insights for national-level decision makers as they continue to monitor and refine the RHS model for Singapore over the coming years. In designing this evaluation, stakeholders including programme team members, healthcare managers and policymakers were engaged to help evaluators understand the context in which the programmes were implemented, how interventions have been delivered, to define objectives, evaluation questions and outcomes that are relevant to the stakeholders and to identify constraints to the feasibility of implementation of the evaluation itself. Various potential pitfalls in evaluation design and implementation have been collectively identified, and risk management strategies were put into place wherever possible. This provides confidence that the evaluation efforts are rigorous, relevant and has the potential to provide insights for programme improvement as well as provide evidence to support policy decisions to scale-up/down activities within NUHS. Engagement of stakeholders early in the evaluation efforts is also expected to increase the likelihood that recommendations will be adopted to improve the relevant components of the existing care models and define future new care models. However, the involvement of stakeholders may exert pressure to act on preliminary findings. Understanding these dynamics will also be of interest as part of the evaluation itself.

    Like NUHS, other integrated care networks within Singapore and abroad are experimenting with new integrated care models. However, the findings from existing research of the (cost-)effectiveness of integrated care are to a major extent inconsistent because of the variations in the strategic outcomes, methods of implementation, contexts (ie, system and policy) and/or applied evaluation measures. Examples include next-generation accountable care organisations in the USA,47 integrated care pioneers and vanguard sites in England,48 49 population health management pilots in the Netherlands,50 integrated care pilots in Belgium51 and the integrated care demonstrators in Australia.52 This study provides an opportunity to fill some of the gaps in current research by evaluating the NUHS-RHS and its constituent programmes using a rigorous and comprehensive design that balances the needs of context-specific evaluation with international comparability. Wherever possible, steps such as using internationally validated instruments for evaluation of chronic disease are taken, to allow for meaningful international comparisons that can increase collective knowledge about the restructuring of CCMs towards advancing integrated care.

    Acknowledgments

    We would like to thank Gerald Koh, Lim Yee Wei and Wee Hwee Lin for their advice for this study.

    References

    1. 1.↵
      World Health Organization. WHO global strategy on people-centred and integrated health services- interim report. 2015.
    2. 2.↵
      2. Starfield B
      . Is patient-centered care the same as person-focused care? Perm J 2011;15:63–9.doi:10.7812/TPP/10-148
      OpenUrlPubMed
    3. 3.↵
      2. Valentijn PP ,
      3. Schepman SM ,
      4. Opheij W , et al
      . Understanding integrated care: a comprehensive conceptual framework based on the integrative functions of primary care. Int J Integr Care 2013;13:e010.doi:10.5334/ijic.886
      OpenUrlPubMed
    4. 4.↵
      World Health Organization. Integrated Health Services - What and Why? World Health Organization Report 2008.
    5. 5.↵
      2. Suter E ,
      3. Oelke ND ,
      4. Adair CE , et al
      . Ten key principles for successful health systems integration. Healthc Q 2009;13 Spec No:16–23.doi:10.12927/hcq.2009.21092
      OpenUrlCrossRefPubMed
    6. 6.↵
      2. Minkman MMN
      . Care dintegrated towards a development model for integrated care. Int J Integr Care 2012;12:e197.
    7. 7.↵
      2. van der Klauw D ,
      3. Molema H ,
      4. Grooten L , et al
      . Identification of mechanisms enabling integrated care for patients with chronic diseases: a literature review. Int J Integr Care 2014;14 http://doi.org/.doi:10.5334/ijic.1127
    8. 8.↵
      2. Goodwin N
      . Understanding and evaluating the implementation of integrated care: A ’Three Pipe' Problem. Int J Integr Care 2016;16:19.doi:10.5334/ijic.2609
      OpenUrl
    9. 9.↵
      2. Greenhalgh T ,
      3. Robert G ,
      4. Macfarlane F , et al
      . Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q 2004;82:581–629.doi:10.1111/j.0887-378X.2004.00325.x
      OpenUrlCrossRefPubMedWeb of Science
    10. 10.↵
      2. Craig P ,
      3. Dieppe P ,
      4. Macintyre S , et al
      . Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ 2008;337:a1655.doi:10.1136/bmj.a1655
      OpenUrlFREE Full Text
    11. 11.↵
      2. Moore GF ,
      3. Audrey S ,
      4. Barker M , et al
      . Process evaluation of complex interventions: Medical Research Council guidance. BMJ 2015;350:h1258.doi:10.1136/bmj.h1258
      OpenUrlFREE Full Text
    12. 12.↵
      Ministry of Health Singapore. STATE OF HEALTH Report of the Director of Medical Services 2003-2012. Singapore.
    13. 13.↵
      Bloomberg Best (And Worst). Most Efficient Healthcare 2014: Countries. https://www.bloomberg.com/graphics/best-and-worst/#most-efficient-health-care-2014-countries
    14. 14.↵
      Ministry of Health. Healthcare 2020: Improving Accessibility, Quality & Affordability, 2012.
    15. 15.↵
      Ministry of Health S. Reorganisation of healthcare system into three integrated clusters to better meet future healthcare needs 2017 https://www.moh.gov.sg/content/moh_web/home/pressRoom/pressRoomItemRelease/2017/reorganisation-of-healthcare-system-into-three-integrated-cluste.html (accessed 01 Mar 2017).
    16. 16.↵
      Ministry of Health Singapore. Speech by Minister for Health, Mr Gan Kim Yong, at the MOH Committee of Supply Debate 2016 https://www.moh.gov.sg/content/moh_web/home/pressRoom/speeches_d/2016/speech-by-minister-fo-health-mr-gan-kim-yong-at-the-moh-commit.html2017.
    17. 17.↵
      CDC. Types of Evaluation, 2017. Available from. https://www.cdc.gov/std/Program/pupestd/Types%20of%20Evaluation.pdf2017.
    18. 18.↵
      2. Pawson R ,
      3. Tilley N
      . Realistic Evaluation. London: SAGE, 1997.
    19. 19.↵
      2. Salter KL ,
      3. Kothari A
      . Using realist evaluation to open the black box of knowledge translation: a state-of-the-art review. Implement Sci 2014;9:115.doi:10.1186/s13012-014-0115-y
      OpenUrlCrossRefPubMed
    20. 20.↵
      2. Byng R ,
      3. Norman I ,
      4. Redfern S
      . Using realistic evaluation to evaluate a practice-level intervention to improve primary healthcare for patients with long-term mental illness. Evaluation 2005;11:69–93.doi:10.1177/1356389005053198
      OpenUrlCrossRef
    21. 21.↵
      2. Pawson R ,
      3. Tilley N
      . Realist Evaluation, 2004.
    22. 22.↵
      2. Berwick DM
      . The science of improvement. JAMA 2008;299:1182–4.doi:10.1001/jama.299.10.1182
      OpenUrlCrossRefPubMedWeb of Science
    23. 23.↵
      2. Parry GJ ,
      3. Carson-Stevens A ,
      4. Luff DF , et al
      . Recommendations for evaluation of health care improvement initiatives. Acad Pediatr 2013;13:S23–S30.doi:10.1016/j.acap.2013.04.007
      OpenUrlCrossRefPubMed
    24. 24.↵
      2. McCawley P
      . The Logic Model for Program Planning and Evaluation University of Idaho Extension. https://www.cdc.gov/std/Program/pupestd/Components%20of%20a%20Logic%20Model.pdf.
    25. 25.↵
      System NUH. Regional Health System - Our Programmes. http://www.nuhs.edu.sg/regional-health-system/programmes.html2016.
    26. 26.↵
      2. Nurjono M ,
      3. Valentijn PP ,
      4. Bautista MAC , et al
      . Psychometric validation of Rainbow Model of Integrated Care Measurement Tool (RMIC-MT) within Singapore’s Regional Health Systems [submitted for publication]. 2016
    27. 27.↵
      2. Curry LA ,
      3. Krumholz HM ,
      4. O’Cathain A , et al
      . Mixed methods in biomedical and health services research. Circ Cardiovasc Qual Outcomes 2013;6:119–23.doi:10.1161/CIRCOUTCOMES.112.967885
      OpenUrlFREE Full Text
    28. 28.↵
      2. Hasson H
      . Systematic evaluation of implementation fidelity of complex interventions in health and social care. Implement Sci 2010;5:67.doi:10.1186/1748-5908-5-67
      OpenUrlCrossRefPubMed
    29. 29.↵
      2. O’Cathain A ,
      3. Murphy E ,
      4. Nicholl J
      . Three techniques for integrating data in mixed methods studies. BMJ 2010;341:c4587.
      OpenUrlFREE Full Text
    30. 30.↵
      2. Wagner EH ,
      3. Austin BT ,
      4. Davis C , et al
      . Improving chronic illness care: translating evidence into action. Health Aff 2001;20:64–78.doi:10.1377/hlthaff.20.6.64
      OpenUrlAbstract/FREE Full Text
    31. 31.↵
      2. Wee HL ,
      3. Loke WC ,
      4. Li SC , et al
      . Cross-cultural adaptation and validation of Singapore Malay and Tamil versions of the EQ-5D. Ann Acad Med Singapore 2007;36:403–8.
      OpenUrlPubMed
    32. 32.↵
      2. Glasgow RE ,
      3. Wagner EH ,
      4. Schaefer J , et al
      . Development and validation of the Patient Assessment of Chronic Illness Care (PACIC). Med Care 2005;43:436–44.doi:10.1097/01.mlr.0000160375.47920.8c
      OpenUrlCrossRefPubMedWeb of Science
    33. 33.↵
      2. Krucien N ,
      3. Le Vaillant M ,
      4. Pelletier-Fleury N
      . Adaptation and validation of the patient assessment of chronic illness care in the French context. BMC Health Serv Res 2014;14:269–69.doi:10.1186/1472-6963-14-269
      OpenUrl
    34. 34.↵
      2. Cramm JM ,
      3. Nieboer AP
      . Factorial validation of the Patient Assessment of Chronic Illness Care (PACIC) and PACIC short version (PACIC-S) among cardiovascular disease patients in the Netherlands. Health Qual Life Outcomes 2012;10:104–04.doi:10.1186/1477-7525-10-104
      OpenUrlPubMed
    35. 35.↵
      2. Aragones A ,
      3. Schaefer EW ,
      4. Stevens D , et al
      . Validation of the Spanish translation of the Patient Assessment of Chronic Illness Care (PACIC) survey. Prev Chronic Dis 2008;5:A113.
      OpenUrlPubMed
    36. 36.↵
      2. Liu LJ ,
      3. Li Y ,
      4. Sha K , et al
      . Patient assessment of chronic illness care, glycemic control and the utilization of community health care among the patients with type 2 diabetes in Shanghai, China. PLoS One 2013;8:e73010.doi:10.1371/journal.pone.0073010
    37. 37.↵
      2. Aller MB ,
      3. Vargas I ,
      4. Garcia-Subirats I , et al
      . A tool for assessing continuity of care across care levels: an extended psychometric validation of the CCAENA questionnaire. Int J Integr Care 2013;13:e050.doi:10.5334/ijic.1160
      OpenUrl
    38. 38.↵
      2. Hibbard JH ,
      3. Mahoney ER ,
      4. Stockard J , et al
      . Development and testing of a short form of the patient activation measure. Health Serv Res 2005;40(6 Pt 1):1918–30.doi:10.1111/j.1475-6773.2005.00438.x
      OpenUrlCrossRefPubMedWeb of Science
    39. 39.↵
      2. Linden A ,
      3. Adams JL ,
      4. Roberts N
      . Evaluating disease management program effectiveness: an introduction to time-series analysis. Dis Manag 2003;6:243–55.doi:10.1089/109350703322682559
      OpenUrlPubMed
    40. 40.↵
      2. Kontopantelis E ,
      3. Doran T ,
      4. Springate DA , et al
      . Regression based quasi-experimental approach when randomisation is not an option: interrupted time series analysis. BMJ 2015;350:h2750.doi:10.1136/bmj.h2750
      OpenUrlFREE Full Text
    41. 41.↵
      2. Bradley EH ,
      3. Curry LA ,
      4. Devers KJ
      . Qualitative data analysis for health services research: developing taxonomy, themes, and theory. Health Serv Res 2007;42:1758–72.doi:10.1111/j.1475-6773.2006.00684.x
      OpenUrlCrossRefPubMedWeb of Science
    42. 42.↵
      2. Nurjono M ,
      3. Valentijn PP ,
      4. Bautista MA , et al
      . A Prospective Validation Study of a Rainbow Model of Integrated Care Measurement Tool in Singapore. Int J Integr Care 2016;16:16(1):1.doi:10.5334/ijic.2021
      OpenUrl
    43. 43.↵
      2. Tsiachristas A
      . Payment and economic evaluation of integrated care. Int J Integr Care 2015;15 http://doi.org/.doi:10.5334/ijic.2009
    44. 44.↵
      World Health Organization. CostIt. http://www.who.int/choice/toolkit/cost_it/en/.
    45. 45.↵
      World Health Organization. Cost effectiveness and stratgeic planning (WHO-CHOICE). 2016;2017 http://www.who.int/choice/costs/CER_levels/en/.
    46. 46.↵
      2. Fetters MD ,
      3. Curry LA ,
      4. Creswell JW
      . Achieving integration in mixed methods designs-principles and practices. Health Serv Res 2013;48(6 Pt 2):2134–56.doi:10.1111/1475-6773.12117
      OpenUrlCrossRefPubMedWeb of Science
    47. 47.↵
      Centres for Medicare & Medicaid Services. Next Generation ACO Model. 2016 https://innovation.cms.gov/initiatives/Next-Generation-ACO-Model/ (accessed 15 June 2016).
    48. 48.↵
      2. England N
      . Integrated Care Pioneers. 2016 https://www.england.nhs.uk/pioneers/ (accessed 15 June 2016).
    49. 49.↵
      2. England N
      . Health and high quality care for all. now and for future generations 2016 https://www.england.nhs.uk/ourwork/futurenhs/new-care-models/ (accessed 15 June 2016).
    50. 50.↵
      2. Kodner DL
      . All together now: a conceptual exploration of integrated care. Healthc Q 2009;13:6–15.doi:10.12927/hcq.2009.21091
      OpenUrlCrossRefPubMed
    51. 51.↵
    52. 52.↵
      NSW Government. Integrated Care Demonstrators 2016. http://www.health.nsw.gov.au/wohp/Pages/demonstrators.aspx (accessed 15 June 2016).

    Footnotes

    • Contributors S-AT, TW and HJMV initiated the study and HJMV conceptualised the study. HJMV and MN designed and constructed the first draft of study protocol. S-AT, SHW, KMF, S-AT, TW and JY reviewed study protocol and provided feedback for the refinement of the protocol. MN, PS, AL, XYL, SHW and KMF were involved in data collection, and MN, PS, XYL and FS were involved in data analysis related to this study. MN wrote the first draft and the final version of the manuscript. All authors read, provided suggestions for revision and approved the final manuscript.

    • Funding This study is funded by NUHS Research Office (Grant No: NUHSRO/2016/008/RO5+5/FY16CF/LOA) awarded to HJM Vrijhoef and taken over by Thomas Wee after HJM Vrijhoef changed positions.

    • Competing interests ST, SHW, KMF, S-AT and TW are a part of the NUHS-RHS strategic planning office, which is involved in the planning and development of programmes under the purview of the NUHS-RHS.

    • Patient consent Obtained.

    • Ethics approval National Healthcare Group -Domain Specific Review Board, Singapore.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Data sharing statement For access to the data set, a formal request should be sent to the corresponding author. The request will only be considered when the primary results of the study have been published.