Article Text
PDF
Abstract
Objectives Programmes to address chronic disease are a focus of governments worldwide. Despite growth in ‘implementation science’, there is a paucity of knowledge regarding the best means to measure sustainability. The aim of this review was to summarise current practice for measuring sustainability outcomes of chronic disease health programmes, providing guidance for programme planners and future directions for the academic field.
Settings A scoping review of the literature spanning 1985–2015 was conducted using MEDLINE, CINAHL, PsychINFO and The Cochrane Library limited to English language and adults. Main search terms included chronic disease, acute care, sustainability, institutionalisation and health planning. A descriptive synthesis was required. Settings included primary care, hospitals, mental health centres and community health.
Participants Programmes included preventing or managing chronic conditions including diabetes, heart disease, depression, respiratory disease, cancer, obesity, dental hygiene and multiple chronic diseases.
Primary and secondary outcome measures Outcome measures included clarifying a sustainability definition, types of methodologies used, timelines for assessment, criteria levels to determine outcomes and how methodology varies between intervention types.
Results Among 153 abstracts retrieved, 87 were retained for full article review and 42 included in the qualitative synthesis. Five definitions for sustainability outcome were identified with ‘maintenance of programme activities’ most frequent. Achieving sustainability was dependent on inter-relationships between various organisational and social contexts supporting a broad scale approach to evaluation. An increasing trend in use of mixed methods designs over multiple time points to determine sustainability outcomes was found.
Conclusions Despite the importance and investment in chronic disease programmes, few studies are undertaken to measure sustainability. Methods to evaluate sustainability are diverse with some emerging patterns in measurement found. Use of mixed methods approaches over multiple time points may serve to better guide measurement of sustainability. Consensus on aspects of standardised measurement would promote the future possibility of meta-analytic syntheses.
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Strengths and limitations of this study
The systematic identification and abstraction of eligible publications assessed by more than one author and the use of manual reference searching and forward citation tracking as well as the use of an appropriate quality assessment tool for classifying the eligible publications.
The scoping method of this review has allowed us to map the heterogeneous body of literature in this field.
The scoping method has also allowed us to include a greater range of study designs and methodologies, currently used in the chronic disease field, to allow for rich and in-depth discussion.
The heterogeneous nature of the literature precludes a meta-analytic review making the results less generalisable.
The systematic method applied in this scoping review means we may have missed some relevant reports if these have been published in the grey literature.
Background
The continuation of an effective health programme, beyond the initial implementation phase, is paramount to maintaining better outcomes for patients with chronic disease. There exists a paucity of knowledge regarding the best means to ensure sustainability of chronic disease health programmes.1 The over-riding challenge is to sustain health programmes after initial programme support has been removed or has expired. However, research providing evidence for effective sustainability strategies for health programmes is underdeveloped,2 and clear recommendations to promote sustainability are limited.3
Measurement of sustainability in the chronic disease field presents a challenge due to the scope of assessments utilised, interventions provided and the heterogeneity between and within illnesses.4 In addition, comorbidities are frequent for people with chronic disease and the subsequent interactions between diseases and treatments provide further challenges for measurement.5 Thus, the interventions developed and implemented in the chronic disease field are complex.
As such, in recent times, it has been suggested that a multifaceted approach to measuring sustainability is required to determine outcomes.6 ,7 The outcomes are determined by the aim of individual researchers and may vary from sustained health outcomes to continuation of programme activities. For example, Rowley et al8 were interested in evaluating the effectiveness of a new programme to prevent obesity, diabetes and heart disease in a remote indigenous community. Thus, the outcomes of interest to determine sustainability of an effective health programme were specific health measures (body mass index and impaired glucose tolerance).8 In contrast, Brand et al9 were interested in evaluating adherence to clinical practice guidelines for chronic obstructive pulmonary disease. The guidelines were evidence based and thus had previously proven to improve health outcomes. As such, the authors aimed to evaluate effectiveness of adherence to the guidelines since adherence had previously been proven to improve health outcomes.9 A different approach has also been to determine the level of community ownership of the programme. Koskan et al10 found that participants in the evaluation of an obesity prevention programme viewed sustainability as increased community ownership of the programme with less support from outside organisations. Indeed, these examples illustrate that sustainability outcomes often move beyond longitudinal programme outcomes and are reframed to include spread of the programme and community ownership. In this paper, sustainability of chronic disease health programmes focuses on programme processes as opposed to health outcomes.
Clearly, sustainability is a multidimensional concept encompassing a diversity of forms along a continuation process with indicators of success that fall into distinct categories.7 These include: (1) maintenance of the health benefits achieved through a programme; (2) maintenance of the core activities central to the programme; and (3) continued capacity of the community to build and deliver health programmes (the extent to which community members are educated and can access programme resources).7 Scheirer and Dearing6 have added to the list including: (1) programme diffusion (when the underlying concepts or innovations spread to new locations);6 (2) maintaining new organisational practices, procedures and policies;6 and (3) maintaining attention to the issues addressed by the programme.11 Clearly, many variables need to be considered and clearly defined in studies investigating sustainability, and planning for, or measuring, sustainability must be included in the programme development stages.
It is acknowledged that it is not always necessary to sustain all original programme activities.1 ,7 ,12 Therefore, Greenhalgh et al13 contend that the more complex approach of drawing narratives from multiple interacting processes can offer unique and in-depth insights into measuring sustainability outcomes as a move away from simply measuring relationships between a set of dependent and independent variables.14 As such, in sustainability research, differing approaches based on either being intervention focused or making an assessment of sustainability as a part of complex systems have implications for the way research in this field is conducted and is influenced by the health discipline.15 Several authors have proposed frameworks for programme sustainability assessment to enable some standardisation and guidance within different disciplines.16–19
A recent review of empirical studies used to investigate health programme sustainability revealed that 40–60% of programmes continued in some form; however, the studies’ designs were weak.20 For example, key variables and definitions were not clearly defined.20 Clearly, more evidence and clearer recommendations are needed for current sustainability research. A current understanding of the research base is needed to provide evidence-based recommendations and facilitate well-informed decision-making on programme sustainability methods in the health field. Without clear evidence, successfully sustaining health programmes may continue to be a challenge.14
Identifying the research question
Chronic diseases, in particular those related to cardiovascular disease and diabetes, are a growing concern to governments in developed and developing countries since they are a major source of health loss in society.21 Until now, the information on how to best sustain and measure the sustainability outcomes of chronic disease prevention and management programmes is rare. The primary aim of this paper was to conduct a review of the current literature describing the sustainability of chronic disease programmes. The second aim was to summarise the empirical methods used to measure sustainability of chronic disease health programmes. Health programmes in the field of chronic disease management within hospital, primary care and community health settings were the primary focus. For this review, outcomes measured for determining if a programme has been sustained are referred to as the ‘sustainability outcome’. The research questions for this scoping review included:
How are sustainability outcomes of health programmes defined for measurement in the field of chronic disease?
What methodologies are used to measure the defined sustainability outcomes including the types of study designs?
What is the typical timeline for assessing the sustainability of a programme? This is designed to enable a better understanding of when the sustainability phase generally begins.
What criteria levels are set to determine if sustainability has been achieved based on the sustainability outcome and are these predefined?
How does the methodology vary between intervention types? This question was designed to explore the relationship between the nature of the programme itself and type of study methodology used.
Methods
Ethics and dissemination
This is a scoping review of the literature, thus formal ethics was not required as patient data or meta-analysis was not conducted.
We conducted a scoping literature review. The aim of a scoping review is to map existing literature in a given field when the topic is of a heterogeneous nature and has not been extensively reviewed.22 Our decision to adopt this approach for our review was based on allowing us to provide an overview of the diverse body of sustainability literature, including the greater range of study designs and methodologies, currently being used in the chronic disease field.
In order to clarify the focus of this scoping review, the outcomes of interest need to be specified.23 Therefore, in this review, a sustainability outcome was defined as the long-term survival of programme activities: health benefits or continued capacity of organisations to deliver and adapt programme activities.7
Identifying relevant studies
An electronic database search was conducted for studies, using MEDLINE, CINAHL, PsychINFO and The Cochrane Library, searching dates between 1 January 1985 and 1 June 2015 using a combination of MeSH and free text by Boolean operators (table 1). The combination of MeSH and free text operators included one of the key MeSH headings together with each of the key words (eg, Programme evaluation with chronic disease, acute care, continuation, institutionalisation, maintenance and programme development) for each of the databases. The reference lists of included papers were also manually searched to identify any potentially relevant studies not found in the electronic search. Forward citation tracking24 was also used to identify additional papers.
Electronic database key word terms and search engines
Study selection
Each potentially eligible publication was independently assessed for inclusion and quality. LF performed the initial review of publications with the following criteria set for inclusion of studies: (1) studies with a focus on health programme sustainability, including the various measures used to determine the sustainability outcome, as well as the reported factors, facilitators and barriers to sustainability, (2) no pre-selected criteria for assessment time periods were used, (3) settings included primary care, hospitals, mental health centres and community health, (4) studies that were randomised controlled, controlled or descriptive and had retrospective or prospectively collected data were included, and finally, (5) multicase and single-case studies without original data were included.
The following criteria were set for exclusion of the studies: (1) studies limited to implementation processes only, (2) programmes that were not related to chronic illness, (3) studies that did not specify clear measures for assessing programme sustainability, (4) studies were not included if the stated measures did not match the main study findings, and finally, (5) opinion pieces and conceptual studies were not included. The co-authors assessed the filtered abstracts or full articles following the above criteria.
After initial review, methodological quality of the studies was assessed by evaluating the design, methods, baseline data, interventions, assignment methods, statistical methods and explanation of results using the Transparent Reporting of Evaluations with Non-randomised Designs (TREND) guidelines.25 Additional items from the Consolidated Standards of Reporting Trials (CONSORT) guidelines were included for the one RCT study included.26 For example, studies were excluded if there was insufficient information on study sample, statement of objectives, methods of recruitment, unit of delivery, methods used to collect data, statement of results for stated primary and secondary outcomes, and clear discussion.
Charting the available evidence
An Excel-based (Microsoft Office version14.2.1) extraction checklist was designed and used to check data using the following descriptors: information on target population, eligibility criteria and methods of participant recruitment, details of intervention delivery, statement of specific objectives and hypotheses, methods used to collect data, sample size, assignment methods, unit of analysis, relevant results, theoretical relevance and generalisability. Since the broad nature of the included studies would preclude meta-analysis, a descriptive synthesis is provided. One study included in the analysis was a randomised controlled trial (RCT). This study was evaluated using the CONSORT guidelines.27 Additional items were added to the excel extraction checklist for these studies including protocol and registration, eligibility criteria, risk of bias, study selection and summary measures.26
Defined sustainability outcomes were extracted from the studies and categorised according to six intervention-based indicators from the literature: (1) maintenance of the health benefits achieved through a programme; (2) maintenance of the core activities central to the programme; (3) continued capacity of the community to build and deliver health programmes,7 (4) programme diffusion;6 (5) maintaining new organisational practices, procedures and policies;6 and (6) maintaining attention to the issues addressed by the programme.11 The indicators were specifically selected since they cover a diverse range of possible outcomes that fit with the broad definition of sustainability outcomes chosen for this review. The indicators were adapted from authors Shediac-Rizkallah and Bone7 and Scheirer and Dearing.6 These authors have shaped different understandings of the concepts of sustainability and have emphasised that a multifaceted approach to measuring sustainability is now required to determine outcomes. Therefore, these indicators were used to provide the framework for describing the sustainability outcomes reported from the included studies. Each study included in this paper was also categorised for intervention type according to Scheirer's19 definitions of intervention type. Six intervention type categories were used: (1) interventions implemented by individual providers, (2) interventions requiring coordination by multiple staff, (3) new policies, procedures and technologies, (4) capacity or infrastructure building, (5) collaborative partnerships and (6) broad scale system change.19
Collating, summarising and reporting the results
Database searching yielded 130 studies (figure 1) for screening and 32 additional articles were identified through manual searching and forward citation tracking. After duplicates were removed, 153 articles were screened. Sixty-six were eliminated based on the eligibility criteria. A further 45 papers were excluded after full-text review because they did not conform to the criteria for chronic disease programme settings (eg, specialist infectious disease centres); did not meet methodological criteria because they included non-health-based programmes (were based on the education system and settings in public safety—housing, crime, etc), or did not state clear sustainability outcomes or related factors or determinants of sustainability. This left 42 articles meeting the review criteria.
Literature search flow diagram.
The 42 eligible studies were focused on programmes for preventing or managing chronic conditions including diabetes (n=5), heart disease (n=4), depression (n=8), respiratory disease (n=3), cancer (n=3), obesity (n=5), dental hygiene (n=1), aged care (n=4) and multiple chronic diseases across primary care and/or remote settings (n=9). The qualitative synthesis of the included studies is presented according to each of the specific research questions as outlined below.
Defined sustainability outcomes
The defined outcomes are listed in table 2. The maintenance of programme activities as a defined outcome for measurement was used in the majority of studies (n=37). Specific programme components were identified as the measure for sustainability outcome in these studies. Specific components included: continued adherence to evidence-based recommendations,9 ,13 ,28–37 use of programme-specific tools,38 patient referrals,31 ,39 regular measures of clinical indicators,40 ,41 and direct outcomes from involvement in programme activities such as the percentage of people attempting dietary change from the Rowley et al8 study.
Defined outcomes, intervention type, unit of analysis and methodology for all studies included in the final analysis
Most often, the system was the unit of analysis in these studies to determine outcome. For example, Meredith et al,32 in their study of depression among 17 primary care centres, found that even though only 11 of the sites continued programme activities, 15 reported spread of the activities to other providers and patient groups, suggesting that the programme had reach into new areas. This provides a holistic overview of the continued performance of activities within the context of organisational culture and programme growth. In addition, Greenhalgh et al,13 in their follow-up study of three services focused on the prevention of stroke, kidney and sexual health in primary and secondary care centres, found that most programme activities continued at 7 years follow-up. In addition, they found that significant cultural changes had occurred within these organisations and that services had spread into new areas.
Summary of study design and methodologies
Varying study designs and methodologies were used in the studies eligible for this review. Designs included case study reports (n=21), through to randomised controlled trials (n=1), and participant numbers varied from 20 to 2000. Table 3 summarises the study designs for all included studies.
Study designs used to measure defined outcomes
Nearly half of all the studies (n=19) were designed using a mixed methods approach to measure outcomes. The combination of methods included quantitative measures (audit data, document review and surveys) combined with qualitative methods (in-depth interviews and focus groups with organisational staff members). Table 4 illustrates the relationship between the various outcomes evaluated and the most common methods used to collect these data. The research methodology varied between studies according to the type of defined outcomes (see table 4). Data collection for studies designed to measure health outcomes mostly used quantitative methods (50%) such as document review and audit.9 ,13 ,29–31 ,35 ,38–40 ,42 Mixed methods were most commonly used to measure all other indicators.
Methods of data collection used across defined sustainability outcomes
Timing of data collection
The timing and duration of data collection varied between the studies with assessment periods ranging from 1 to 10 years. Just over half of the studies (n=24) had a single ‘snapshot’ time point to measure sustainability outcome indicators against baseline data.9 ,13 ,28 ,30 ,31 ,33–37 ,40 ,41 ,43–55 The other half used longitudinal data14 ,42 ,56 or multiple time points8 ,29 ,32 ,38 ,39 ,57–65 such as the Ramsey et al39 study. Timing of data collection appeared to vary according to the way sustainability was defined and the indicators selected to measure outcomes. For example, Ramsay et al39 were specifically interested in observing any variability in intervention effects over time. They used monthly quantitative audit data over a 12-month period from baseline to assess adherence to clinical guidelines.39 In contrast, for O'Loughlin et al,34 sustainability was defined according to the level of ‘institutionalisation’ a programme reached within an organisation. Institutionalisation refers to the integration of a programme into the normal routines and everyday practice of an organisation.34 They focused on one single follow-up time point to assess the degree of ‘institutionalisation’ through specific survey questions relating to organisational characteristics and activities.34
In just over half of the studies (n=25), the assessment of the sustainability outcome started immediately post the implementation period. In a quarter of the studies (n=10), data were collected from the time of withdrawal of programme support, while in several studies (n=6) the assessment of sustainability started from baseline. Baseline was commonly defined as the period before active implementation of interventions prior to the start of the programme. The time to the start of that assessment is relevant information to address the first aim of this review—how are sustainability outcomes defined for measurement in the field of chronic disease? We seek to understand perspectives around when sustainability is achieved, as opposed to ongoing programme activity, and when does this start? The results reveal that in just over half of the studies in the review researchers view sustainability as beginning from the implementation phase and beyond.
Criteria for judging sustainability outcome
None of the studies in this review had published preset or post hoc numerical criteria for judging if sustainability had been achieved. Programmes were defined as being sustained if there was evidence of continued improvements from baseline measures or if outcomes were maintained following the implementation phase or cessation of funding, regardless of the magnitude achieved.
Intervention type and study methodology
The most commonly used intervention types were broad scale system change (n=18), interventions requiring coordination of multiple staff (n=26) and new policies, procedures and technologies (n=12) (see table 2). Interventions requiring coordination of multiple staff predominantly used quantitative measures (n=22) and approximately two-thirds qualified the findings with qualitative methods.
The studies designed to assess broad scale system change interventions had longer assessment periods (3–7 years) and more commonly used longitudinal study designs with multiple time points to measure outcomes throughout the programme cycle. For example, Bailie et al29 repeated an audit on five occasions over a 3-year period. Likewise, McDermott et al40 evaluated retrospective patient data over time from implementation.40
Discussion
The scope of care provided to people with chronic illness is heterogeneous, spanning multiple disease types and management interventions. This heterogeneity of care is reflected in the methods used for sustainability research in the chronic disease field. The results of this review reveal that indicators for sustainability, research methodology and timing and duration of data collection methods varied according to the defined outcomes and intervention type.
In most of the studies, the sustainability outcome was defined as the maintenance of programme activities. This fits well with the implementation cycle: the problem area is identified, and goals and activities are designed and implemented and evaluated over time to assess ongoing maintenance.66 ,67 They are intervention-focused with activity-based outcomes. However, this narrow perspective does not provide evidence on whether health benefits are being achieved based on this ongoing activity, and is often an indirect causal assumption that is implied.
Given that improved outcomes are what most programmes aim for, it was surprising that only about one-quarter of studies included the measurement of patient-level health outcomes as a primary outcome of sustainability. Nevertheless, most of the programme activity outcomes measured in the included studies could be considered reasonable surrogates for achieving health outcomes. For example, adherence to COPD evidence-based guidelines in the Brand et al9 study is likely to result in improved outcomes since the relationship between patient interventions delivered through a programme has an evidence base associated with achieving positive health benefits.
A trend towards defining the primary outcome as the maintenance of programme activities, as evidenced in this review, could be due to a large focus on self-management interventions to manage chronic disease.4 Self-management strategies are typically measured by patient self-report outcomes.4 It has previously been suggested that the effectiveness of self-management interventions are mostly derived from variable evaluation-based measures that carry a substantial amount of measurement error.4 This may contribute to a shift in focus towards activity-based outcomes focusing on the quality and continuation of delivered services rather than health-based outcomes. Clinical indices provide clear reference points for precomparison and postcomparison allowing for concise summaries and recommendations for health providers.
The majority of studies were characterised as broad scale system change interventions.19 These studies used clinical markers as a primary outcome in addition to programme activities, community capacity, programme diffusion and/or policy outcomes. The various outcomes together gave a bigger picture of change and transformation with the whole organisation itself. This method of using health outcome data with other outcomes may provide more robust evidence about the whole broad scale change. Consideration of intervention type may assist in the complex process of planning for sustainability research in the chronic disease field.
Our results reveal that the study methodology varied according to intervention type and this may go some way in explaining the variation that exists in research between programs. For example, evaluations of broad scale system change interventions often employ multiple time points to measure outcomes, demonstrating an interest in researching trends over time. These studies often evaluated multiple outcomes as well. A growing number of studies (one half from this review) are using multiple time points over time to measure indicators. This finding supports Scheirer et al's19 hypothesis that such interventions require ongoing evaluations.
This is an important consideration for future sustainability research on chronic disease management. The feasibility of continuous measurement should be explored through clinical registries or document review from the outset of programme planning.68 ,69 Longitudinal testing over multiple time points may serve to distinguish residual improvements from implementation to sustained persistence of improvement.70
Measurement needs to encapsulate how programs evolve over time, adapt to changing contexts and transform to suit changing health system needs. Evaluations of such studies should also capture spread into new areas or any difficulties as a result of potential environmental changes. Importantly, none of the studies in this review defined predetermined or post hoc numerical criteria as an empirical measure of sustainability being achieved or what would be considered acceptable degrees of regression/attrition. Decision rules made around defining outcomes for dependent variables appeared to be a subjective process based on the objectives and theoretical underpinnings unique to each study. Moreover, since these programs are situated within complex systems with evolving contexts, it may be that setting rigid criteria for judging outcomes is not realistic or desirable.
Our review supports the conceptualisation of sustainability as a broad scale approach.14 ,2 ,13 The results reveal that most studies took a complex approach with mixed methods and defined outcomes that extend beyond continuation of programme activity. For example, Meredith et al32 quantitatively evaluated improved delivery of services and qualified their findings through in-depth interviews to determine the spread of programme into other areas. Consistent with the work of Gruen et al14 and Greenhalgh et al,2 our findings reveal that the sustainability outcome is dependent on a complex set of inter-relationships between various organisational and social systems impacting broad scale change. It extends the work of Gruen et al's14 and Greenhalgh et al's2 by providing a synthesis of the specific research methods used to measure sustainability in the area of chronic disease. It also extends Wiltsey-Stirman et al's15 review of methods used to measure sustainability outcomes by providing a description of intervention types in the chronic disease field. Finally, it supports and extends Scheirer et al's19 framework for evaluating interventions by reviewing intervention types and relating these to the study methodology currently used.
The strengths of our review include the systematic identification and abstraction of eligible publications assessed by more than one author and the use of manual reference searching and forward citation tracking as well as the use of an appropriate quality assessment tool for classifying the eligible publications. The limitations of our review include the fact that we may have missed relevant reports if these have been published in the grey literature. The lack of standardisation and reporting of the sustainability outcomes precluded our ability to perform meta-analytic synthesis. Consensus on aspects of standardised measurement and reporting would promote the future possibility of meta-analytic syntheses.
Conclusion
Overall, despite chronic diseases being a major focus of attention worldwide, owing to the growing financial impacts on health systems and associated disease burden imposed on society, publications on the sustainability of programmes within this field are limited and cover only a small range of conditions, such as diabetes. This review provides an important contribution to understanding the way sustainability is currently measured in the chronic disease field. Currently, research methods to evaluate health programme sustainability in the area of chronic disease management vary widely and very little work in this area has been published. Given the increasing burden of chronic disease in our society, it is also a matter of concern that there are no clear guidelines on the best way to measure sustainability in this field.
However, our review shows some emerging patterns in the research methods used to measure sustainability. First, there is a clear trend in the use of assessment through multiple time points from baseline to gain valuable information about the ongoing effectiveness of given programmes. Second, a clear relationship between intervention type and use of research methods is shown, with broad scale system change interventions using longitudinal designs with multiple time points. Finally, our results support recent recommendations that evaluation of sustainability requires a holistic approach capturing all elements with the use of mixed methods approaches.
Further development of current empirical methods used to measure sustainability, in the area of chronic disease, is needed to provide direction for programme planners and evaluators in designing, implementing and evaluating health programmes. More research into methodology and measurement is needed to provide a more rigorous science behind sustainability research and clearer guidelines for future programme planners and evaluators.
Acknowledgments
The authors thank the Melbourne Health Centre for Excellence in Neuroscience.
References
Footnotes
Contributors LF is the first author and performed the initial review of publications. DD and DAC assessed abstracts or full articles to confirm eligibility for full review. DD and DAC contributed to the writing of the draft and revisions of the manuscript for intellectual content, and supervised the research process of the review.
Funding This work is supported by the NHMRC Centre for Research Excellence Grant 1001216. DAC was supported by a fellowship from the National Health and Medical Research Council (NHMRC; 1063761 co-funded by National Heart Foundation).
Competing interests None declared.
Ethics approval This review is part of a larger PhD. Ethics has been submitted and approved by the Melbourne Human Research Ethics Committee. Ethics ID-1137091.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement No additional data are available.