Introduction Diabetes represents a major public health and health system burden. As part of the Alberta's Caring for Diabetes (ABCD) Project, two quality-improvement interventions are being piloted in four Primary Care Networks in Alberta. Gaps between health research, policy and practice have been documented and the need to evaluate the impact of public health interventions in real-world settings to inform decision-making and clinical practice is paramount. In this article, we describe the application of the RE-AIM framework to evaluate the interventions beyond effectiveness.
Methods and analysis Two quality-improvement interventions were implemented, based on previously proven effective models of care and are directed at improving the physical and mental health of patients with type-2 diabetes. Our goal is to adapt and apply the RE-AIM framework, using a mixed-methods approach, to understand the impact of the interventions to inform policy and clinical decision-making. We present the proposed measures, data sources and data management and analysis strategies used to evaluate the interventions by RE-AIM dimension.
Ethics and dissemination Ethics approval for the ABCD Project has been granted from the Health Research Ethics Board (HREB #PRO00012663) at the University of Alberta. The RE-AIM framework will be used to structure our dissemination activities by dimension.
Results It will be presented at relevant conferences and prepared for publication in peer-reviewed journals. Various products, such as presentations, briefing reports and webinars, will be developed to inform key stakeholders of the findings. Presentation of findings by RE-AIM dimension will facilitate discussion regarding the public health impact of the two interventions within the primary care context of Alberta and lessons learned to be used in programme planning and care delivery for patients with type-2 diabetes. It will also promote the application of evaluation models to better assess the impact of community-based primary healthcare interventions through our dissemination activities.
- Primary Care
Statistics from Altmetric.com
Diabetes represents a public health and health system burden; primary care requires collaborative team interventions and self-management support.
Study protocol using a mixed-method approach for a comprehensive evaluation.
Evidence on effectiveness of public health interventions in real-world settings is needed to better inform decisions about practice and policy.
RE-AIM model provides a framework to elicit contextual information to better interpret the effectiveness of interventions.
Dissemination activities can be structured using the RE-AIM dimensions to identify target audiences, key messages and appropriate knowledge products.
Strengths and limitations of this study
Application of the RE-AIM framework and a mixed-methods approach allows for a comprehensive evaluation of the ABCD Project interventions beyond clinical effectiveness.
This represents the evaluation of a pilot study in four Primary Care Networks (PCNs), which may not be representative of other primary care models.
Diabetes represents a major public health and health system burden. The Canadian National Diabetes Surveillance System has estimated that 6.2% of the population has diabetes.1 In Alberta, 206 000 people were living with diabetes in 2009, representing over 5.5% of the population.2 This signifies a doubling of affected individuals within the past decade. The majority (ie, >90%) of these individuals have type 2 diabetes. As the number of people with diabetes increases, the number of resulting complications and co-morbidities increases, creating a greater demand on healthcare resources.2 ,3
The Alberta's Caring for Diabetes (ABCD) Project, funded by the Alberta Health ministry as part of the provincial diabetes strategy, was developed to improve the quality and efficiency of care for diabetes in Alberta, Canada, with a focus on supporting Primary Care Networks (PCNs) in non-metro areas of Alberta. PCNs consist of a voluntary network of family physicians (hereby referred to as ‘member physicians’) and allied health professionals, who identify priorities and coordinate health services for patient populations.4 ,5 The PCN model is akin to the ‘patient-centered medical home’ model emerging in the USA.6 ,7
The ABCD team has worked with participating PCNs to implement a number of quality improvement interventions. This includes an ongoing, survey-based cohort study that seeks to understand why some people with type 2 diabetes develop complications while others do not. This study involves an annual survey of individuals with type 2 diabetes over 5 years, to collect data on lifestyle behaviours, self-management and patient-reported outcomes and linkage with administrative databases to assess healthcare utilisation and longer-term clinical outcomes. In addition, participating PCNs will implement pilot interventions including: (1) Healthy Eating and Active Living in Diabetes (HEALD-PCN), a pedometer-based walking programme;8 and (2) TeamCare-PCN, a collaborative team-based, depression case management intervention.9 Key features of HEALD-PCN include the provision of information in a group setting by an exercise specialist on increasing the amount and intensity of physical activity (ie, walking), the glycemic index and individual goal setting. The HEALD-PCN programme also provides opportunities for participants to implement lessons learned (ie, walking group sessions) through partnerships with community recreational facilities.10 Key features of TeamCare-PCN include coordinated care by a nurse care manager to direct active patient follow-up, treat-to-target principles and specialist (ie, psychiatrists and internists/endocrinologists) consultation.11
The efficacy of both pilot interventions has been proven in other settings,10–12 and the study protocols to determine the effectiveness of HEALD-PCN and TeamCare-PCN in the PCN environment in Alberta have been published.8 ,9 Our goal is to also assess the impact of the entire ABCD project activities, including how these different interventions were simultaneously implemented, in Alberta's PCN environment. The purpose of this paper is to describe the design of the evaluation for the different elements of the ABCD project, using the RE-AIM framework.13
Evaluating the ABCD pilot interventions using RE-AIM
The gaps between health research, policy and practice have been well documented.13–16 Evaluations of health interventions are often limited to efficacy studies rather than assessment of potential public health impact.17 Efficacy studies tend to focus on the internal validity of high-intensity health interventions with motivated and homogeneous populations in controlled settings.13 This narrow focus hinders the translation of research into practice and reduces the ability to generalise findings to similar settings.13 Evidence on the external validity of less-intensive interventions in real-world settings is needed to better inform decisions about practice.13
In this context, assessment of clinical effectiveness alone is not enough to inform decisions about a programme's broader public health impact. The RE-AIM evaluation framework was designed to assess health interventions beyond effectiveness to include multiple criteria to better identify effect and transferability.18 The framework consists of five dimensions: Reach into the target population; Effectiveness of the intervention; Adoption by target settings, institutions and staff; Implementation, including consistency and cost of delivery; and Maintenance of intervention effects over time.18
The RE-AIM model addresses two levels of assessment: individual (Reach, Effectiveness); organisation (Adoption and Implementation) or both (Maintenance).13 To fit our evaluation goals, we expanded the assessment level of ‘Reach’13 beyond the individual assessment level (ie, absolute number, proportion and representativeness of individuals willing to participate in an intervention) to include an organisation assessment level (ie, an organisation's ability to identify the entire target population) (table 1). An example of an organisational strategy to identify a population is the development and use of a patient registry.
In table 1, we provide the original definitions for each RE-AIM dimension. Italicised words or phrases indicate modifications made by the ABCD Project team to the original ‘Reach’ definition and assessment level.13 This table was compiled and adapted from several sources.13 ,17 ,18
While there are other evaluation frameworks, such as Procede-Proceed19 and Health Impact Assessment,20 we assert the RE-AIM model is well suited to evaluate the ABCD pilot interventions for two reasons. First, RE-AIM is considered more appropriate for evaluation of behavioural change interventions21 than other models. Second, the dimensions of the RE-AIM model are well matched to inform the specific needs of our audiences and interested parties including healthcare providers, PCN management, policy makers and funders.
Methods and analysis
We will employ a mixed-methods approach22 for our comprehensive evaluation of the ABCD pilot interventions. Using the RE-AIM model, our research team developed logic models and data matrices for both interventions in consultation with advisory committees (see online supplementary Appendix 1; web only file). The overarching questions guiding the evaluation for each intervention are: (1) Is the service delivery model effective in the context of Alberta's primary care setting and (2) What factors contribute to the effectiveness (or ineffectiveness) of the intervention? The more specific evaluation questions related to the RE-AIM framework that will direct the collection and analysis of data for both interventions include:
Reach: Is the intervention reaching the intended target population?
Adoption: Has the intervention been adopted by the PCNs and staff?
Implementation: Is the intervention being implemented as intended? Is it cost-effective?
Effectiveness: What are the immediate, intermediate and long-term impacts of the intervention?
Maintenance: Is the intervention sustainable in a cost-effective way?
Measurement by RE-AIM dimensions
In the following section, we outline the measures proposed for each dimension of RE-AIM to evaluate the ABCD project interventions. A detailed summary is provided in table 2.
Evaluation of reach will be done at the individual (patient) and organisation (PCN) assessment levels to determine if the ABCD pilot interventions are reaching those in most need. At the individual assessment level, we will examine total recruitment into the interventions and usual care groups and compare their characteristics with respect to eligibility criteria, demographic information and other measures. As possible, we will compare characteristics between participants (ie, intervention and usual care groups) and non-participants using aggregate demographic information accessed through PCN patient registries and Alberta Diabetes Surveillance System data.23 Facilitators and barriers to individual patient recruitment and suggestions for improvement will be identified through interviews with PCN staff.
At the organisation assessment level, we will document usual care in the PCNs, including the ability to estimate and identify target patient populations in the focus areas (ie, type 2 diabetes management, depression management and lifestyle counseling) through completion of a standardised checklist. We will examine processes related to registry development and identify facilitators and barriers related to development, use and maintenance through interviews with PCN staff. In addition, we will elicit recommendations related to the PCNs’ ability to identify patient populations to actively offer targeted health services.
Evaluation of effectiveness will be conducted at the individual assessment level to determine impact of the pilot interventions on important outcomes. The design and rationale for controlled evaluations of the effectiveness of the two ABCD pilot interventions have been described elsewhere.8 ,9 The primary outcome of HEALD-PCN is improvement in physical activity (ie, brisk walking), determined by step pedometers and self-report.8 For TeamCare-PCN, the primary outcome is improvement of depressive symptoms as measured by the Patient Health Questionnaire-9 (PHQ-9) items.9 We will also use a variety of measures to determine the effectiveness of both interventions on important outcomes at the individual assessment level including clinical measures (eg, improvements in glycemic control, blood pressure, lipid measurements and body mass index), self-reported health-related quality of life,24–26 self-efficacy,27 satisfaction with care,28 ,29 and process indicators. In addition, we will document unanticipated consequences (positive or negative), such as improved patient linkages with community health resources, to provide a richer understanding of effectiveness. Additional measures and data sources to assess effectiveness are provided in table 2.
We will assess the adoption of the ABCD pilot interventions at the organisation level, including documentation of the criteria for PCN selection and participation in the ABCD Project and PCN Board approval. Also, we will document and compare the characteristics of the participating PCNs (eg, number of family physicians, number of patients served and governance structure) as well as usual care in the focus areas. Dependent on availability of secondary data, we will consider the representativeness of participating PCNs compared with non-participating PCNs. This will be accomplished through document review (eg, ABCD project documents, PCN websites, business plans), use of a standardised usual care checklist and interviews with PCN staff. In addition, perceptions related to the extent to which the ABCD pilot interventions have been adopted by the PCNs and modified to suit their contexts will be elicited through interviews with PCN staff. Identified facilitators and barriers to adoption of the interventions along with creative solutions or modifications will also be documented.
Evaluation of implementation of the ABCD pilot interventions will be done at the individual and organisation assessment levels to determine patient adherence, consistency of implementation and costs of delivering the pilot interventions. To address implementation at an individual assessment level, participant adherence to the intervention models will be determined for both interventions. For HEALD-PCN, attendance at group sessions, participant step logs (ie, recording the number of steps over 3 days) and self-reported physical activity will be assessed. For TeamCare-PCN, adherence to treatment plans, including medication and behavioural modifications (eg, engaging in planned pleasant activities), will be assessed. These types of data will be derived from patient outcome tracking systems employed in each PCN and/or survey items.
At the organisation assessment level, consistency of implementation and the cost of delivering the ABCD pilot interventions will be evaluated to determine the practicality of the interventions. Actual versus intended implementation will be assessed through extensive documentation including development of project materials (eg, training and resource materials), presence of systems and processes (eg, patient registries), intervention staff recruited or hired by PCNs and provision and quality of training in the intervention models. Additional measures and data sources to assess consistent implementation are provided in table 2. Our implementation assessment will also include economic evaluations of the ABCD pilot interventions, which have been described in detail elsewhere.8 ,9
For both ABCD pilot interventions, maintenance will be evaluated at the individual and organisation assessment levels to measure continuation of intervention effects over time. We will use a previously developed conceptual framework that defines sustainability outcomes of health interventions.30 At the individual level, maintenance will be evaluated based on patient-reported health behaviours and self-care collected annually through the ABCD cohort study survey and interviews with a subsample of HEALD-PCN intervention group participants at 6-months post-intervention.
At the organisation assessment level, interviews with PCN staff will be conducted post-interventions to assess integration of intervention model components into practice (eg, continued use of patient registries or screening tools), enhanced organisational capacity (eg, maintaining partnerships) and continued focus on the interplay between diabetes, depression and lifestyle (eg, incorporation of the intervention models into future business plans). In addition, interviews with specialists participating in TeamCare-PCN will be conducted with a focus on sustainability of the model in the current primary care environment, including appropriate compensation and funding approaches and potential medicolegal liability issues.
Our comprehensive evaluation will involve the collection and management of a wide range and large volume of data. Primary data sources for the evaluation of the ABCD pilot interventions include: (1) clinical outcome measures; (2) patient-reported outcomes; (3) interviews (eg, with PCN staff, HEALD-PCN intervention group participants and specialists for TeamCare-PCN); (4) document review (eg, usual care checklists, project documents and field notes) and (5) administrative healthcare datasets.
Clinical outcomes and survey data captured in the patient outcome tracking systems or standardised case forms used in each PCN will be entered into centralised, web-accessible databases. These study databases will be housed on secure servers in the research offices at the University of Alberta. Once the pilot interventions are completed, all data will be exported and merged, based on individually assigned study ID numbers, to form an analysable dataset. Investigators, research assistants and analysts will be masked to allocation status at all times.
Semistructured interviews will take place at the PCN offices of the interviewees. Interviews with HEALD-PCN intervention group participants and TeamCare-PCN specialists will be conducted via telephone. Interviews will be facilitated through the use of interview guides. Interviews will be digitally recorded for subsequent analysis, transcribed verbatim by an independent transcriptionist and verified for accuracy.
Regarding document review, we will develop a standardised usual care checklist by adapting themes from the Change Process Capability Questionnaire31 and the Organisational Readiness to Change Scale32 to be validated by staff of the participating PCNs. Topic areas include: usual care for people with type-2 diabetes; existing PCN diabetes, depression and lifestyle programming and organisational factors and strategies related to PCN patient care. Also, we will document how the ABCD pilot interventions unfolded in each PCN through field notes, communications and meeting minutes. All qualitative data sources, including interview transcripts and documents, will be compiled and managed using Nvivo V. 9.0 software.
Patients enrolled in the pilot interventions and the ABCD Cohort study will be asked for permission to access their medical records by providing their personal health number, thus allowing linkage to provincial healthcare administrative data from Alberta Health for physician, hospital, and emergency department billing and pharmaceutical data (for patients 65 years and older). This linkage will allow healthcare utilisation and healthcare costs to be included in the evaluation.
We are undertaking a broad mixed-methods approach to analysis. In terms of quantitative data, the approach to power, sample size calculations, assessment and statistical modelling of clinical effectiveness have been previously detailed.8 ,9 In terms of qualitative data, we will take a general inductive approach33 with the evaluation questions related to the RE-AIM framework directing the analysis of data. Findings will be derived directly through a content analysis34 of the raw data without preconceived notions about specific findings.
Ethics and dissemination
Ethics approval for the entire ABCD Project and its associated interventions has been granted from the Health Research Ethics Board (HREB #PRO00012663) at the University of Alberta. However, the Board deemed this component of the ABCD Project as evaluation and not research; therefore, it did not require ethics review and approval. Regardless, the requirements outlined in the Canadian Tri-Council Policy Statement: Ethical Conduct of Research Involving Humans35 will be followed.
Discussion and dissemination
The ABCD Project was developed to improve the quality and efficiency of diabetes care in non-metro Alberta. In order to address the gap between research, policy and practice, we have adapted and expanded the RE-AIM model to conduct a comprehensive evaluation of the ABCD pilot interventions. This will contribute to our knowledge of the broader impact of the two interventions within the evolving primary care context of Alberta beyond effectiveness, as outlined in the study trial designs.8 ,9 The purpose of this article was to present the proposed measures and data sources to be used to evaluate the interventions by RE-AIM dimension. Using the RE-AIM evaluation framework will allow us to systematically identify facilitators, challenges, opportunities and lessons learned to be used in programme planning and care delivery for patients with type-2 diabetes. In addition, our application of the RE-AIM evaluation framework may encourage others to use similar models to determine the impact of community-based primary-healthcare interventions. The RE-AIM model will also be used to structure our dissemination activities. For example, each RE-AIM dimension will inform the development of products (such as academic manuscripts for peer-review publication, presentations at relevant conferences and workshops, and briefing reports) and identification of relevant target audiences.
JAJ is a Senior Scholar with Alberta Innovates-Health Solutions and a Centennial Professor at the University of Alberta. SRM is a Health Scholar funded by Alberta Innovates-Health Solutions and holds the Endowed Chair in Patient Health Management funded by the Faculties of Medicine and Dentistry and Pharmacy and Pharmaceutical Sciences of the University of Alberta. This work was supported in part by a contract from Alberta Health, and a CIHR Team Grant to the Alliance for Canadian Health Outcomes Research in Diabetes (reference #: OTG- 88588), sponsored by the CIHR Institute of Nutrition, Metabolism and Diabetes (INMD).
Review history and Supplementary material
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Files in this Data Supplement:
- Data supplement 1 - Online appendix
Contributors LW developed the evaluation framework for the Alberta's Caring for Diabetes (ABCD) Project and drafted the manuscript. SR, AS, STJ actively contributed to the development of the evaluation framework and critically reviewed and revised the manuscript. FA provided feedback on the manuscript. SRM provided expert feedback on the study design and critically reviewed the manuscript. JAJ conceived of the study, participated in its design and helped in drafting the manuscript. All authors read and approved the final manuscript.
Funding This work was supported in part by a contract from the provincial government (ie, Alberta Health), a grant from the Lawson Foundation (reference #: GRT 2010-028) (www.lawson.ca) and a Canadian Institutes for Health Research (CIHR) Team Grant (reference #: OTG- 88588) to the Alliance for Canadian Health Outcomes Research in Diabetes36 sponsored by the CIHR Institute of Nutrition, Metabolism and Diabetes. The funding sources had no role in the design of the studies or evaluation and will have no role in the conduct, analysis or reporting of the studies or evaluation, nor access to the data.
Competing interests None.
Ethics approval Ethics approval for the entire ABCD Project and its associated interventions has been granted from the Health Research Ethics Board (HREB #PRO00012663) at the University of Alberta. However, the Board deemed this component of the ABCD Project as evaluation and not research; therefore, it did not require ethics review and approval. Regardless, the requirements outlined in the Canadian Tri-Council Policy Statement: Ethical Conduct of Research Involving Humans will be followed.
Provenance and peer review Not commissioned; internally peer reviewed.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.