Article Text
PDF
Abstract
Objectives The Choice and Partnership Approach (CAPA) was developed to create an accessible, child-centred and family-centred model of child and adolescent mental health service delivery that is adaptable to different settings. We sought to describe the state of evidence regarding the extent, outcomes and contextual considerations of CAPA implementation in community mental health services.
Design Scoping review.
Data sources Published and grey literature were searched using MEDLINE, Embase, CINAHL, PsycINFO, Scopus and Google to 13 and 20 July 2022, respectively.
Eligibility criteria We included reports focused on the implementation, outcomes (clinical, programme or system) or a discussion of contextual factors that may impact CAPA implementation in either child and adolescent or adult mental health services.
Data extraction and synthesis Data were extracted using a codebook that reflected the five domains of the Consolidated Framework for Implementation Research (CFIR) and reviewed for agreement and accuracy. Data were synthesised according to the five CFIR domains.
Results Forty-eight reports describing 36 unique evaluations were included. Evaluations were observational in nature; 10 employed pre–post designs. CAPA implementation, regardless of setting, was largely motivated by long wait times. Characteristics of individuals (eg, staff buy-in or skills) were not reported. Processes of implementation included facilitative leadership, data-informed planning and monitoring and CAPA training. Fidelity to CAPA was infrequently measured (n=9/36) despite available tools. Health system outcomes were most frequently reported (n=28/36); few evaluations (n=7/36) reported clinical outcomes, with only three reporting pre/post CAPA changes.
Conclusions Gaps in evidence preclude a systematic review and meta-analysis of CAPA implementation. Measurement of clinical outcomes represents an area for significant improvement in evaluation. Consistent measurement of model fidelity is essential for ensuring the accuracy of outcomes attributed to its implementation. An understanding of the change processes necessary to support implementation would be strengthened by more comprehensive consideration of contextual factors.
- MENTAL HEALTH
- Child & adolescent psychiatry
- Organisation of health services
Data availability statement
Data sharing not applicable as no datasets generated and/or analysed for this study. The reports included in this scoping review are available via the citations.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
A comprehensive search was employed to capture sources not found in the formal literature to provide a broad picture of the implementation of Choice and Partnership Approach in mental health services.
The Consolidated Framework for Implementation Research was followed to ensure thorough capture of relevant contextual constructs and to provide consistent terminology in our review.
We recognise the possibility of publication bias introduced through the potential over-representation of positive experiences.
While we did not assess the quality of included reports in order to include all relevant literature, we recognise that this may have limited our ability to assess gaps in the literature.
Introduction
Mental health and addiction disorders are the most common sources of morbidity among children and youth in developed countries, affecting as many as one in five by age 15.1–5 However, long wait times or other challenges in access mean that many young people do not receive care when they need it.6–9 The delay or absence of appropriate care during childhood and adolescence is associated with poor outcomes, including increased severity of illness and the emergence of secondary disorders.10
The Choice and Partnership Approach (CAPA) was developed to create an accessible, child-centred and family-centred model of child and adolescent mental health service (CAMHS) delivery that better matches care to needs. CAPA incorporates several features that differentiate it from traditional models of mental health service delivery. The philosophy underlying CAPA reflects a shift in clinician stance from ‘expert with power’ to ‘facilitator or partner with expertise’ and values the expertise the client and caregivers offer.11 In response to a family-oriented recovery focus philosophy of mental healthcare in recent years, CAPA emphasises a collaborative approach to mental healthcare where young people, family or caregivers (a member of a young person’s support network) and clinicians jointly develop treatment goals. The model also incorporates continuous quality improvement practices and data-informed decision-making to improve efficiency and effectiveness.12
CAPA consists of 11 key components: leadership, language, handle demand, Choice framework, full booking to Partnership, selecting clinician, core and specific work, job plans, goal setting, peer group discussion and team away days. The creators of the model posit that the totality of the 11 components is greater than the sum of the parts and implementation of all components is required to successfully transform services.11 The implication is that implementation of only select components, or a ‘CAPA-lite’ version of the model, is likely to lead to poor results, reflecting a failure in implementation rather than a failure of the model.11
The model is intended to work ‘in any setting, culture, health organisational system and language’.11 To date, CAPA has been implemented in community-based (or ‘outpatient’) mental health and addictions services in the United Kingdom (UK), Australia, New Zealand, Norway, Belgium, Ireland and Canada.11 Despite being grounded in evidence-informed elements such as demand and capacity theory, elimination of waste, shared decision-making and outcome measurement, there has been little evidence of formal evaluations of CAPA implementation in the published literature.12–17 As mental health systems face significant pressures to provide timely access to effective services, there is a need to better understand the current scope of evidence and to identify any implications of context on successful implementation and expected outcomes.
The aims of this scoping review are, therefore, twofold: (1) to gain an understanding of the extent and outcomes of the implementation of CAPA in community mental health and addictions services and (2) to identify how context influences the implementation of CAPA and resulting measurement of client and system outcomes.
Methods
Overview
A scoping review approach was selected after initial searches of academic journals revealed much heterogeneity, indicating that the evaluation of the implementation and efficacy of CAPA is an emerging field of study.18 The protocol for this scoping review was published a priori (https://bmjopen.bmj.com/content/9/12/e033247).19
Review steps
This scoping review follows the steps proposed by Arksey and O’Malley20 and revised by Levac and colleagues.21 We recognise that the evaluation of formative outcomes in addition to traditionally reported summative outcomes is necessary to establish the success of implementation of healthcare interventions such as CAPA as well as support sustainability and dissemination in other contexts.22 Our overarching programme of research has adopted a realist paradigm developed by Pawson and colleagues to aid in understanding the role of context in the implementation of CAPA—specifically, how mechanisms (the implementation and individual reactions to the key components of CAPA) are influenced by context to produce expected (or unexpected) outcomes.23 24 The Consolidated Framework for Implementation Research (CFIR) offers a comprehensive framework for capturing information about context in that it encompasses many implementation theories while including important constructs not included in individual theories.22 We employed the CFIR in the analysis to ensure thorough capture of relevant constructs related to contextual barriers or facilitators of implementation and to provide structure as well as to use consistent terminology in our review that would also allow comparison with other studies employing the CFIR.22 The CFIR is organised by five contextual domains: intervention characteristics, outer setting, inner setting, characteristics of the individuals involved and the process of implementation.22 This review adheres to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews25 and the Realist And Meta-narrative Evidence Syntheses: Evolving Standards publication standards.26
Identifying the research question
Within our programme of research, our overarching research question is, ‘To what degree does CAPA work, for whom, and under what circumstances?’ (https://www.healthyyoungminds.ca). This scoping review, therefore, serves both to (1) describe the extent and measurement of the outcomes of the implementation of CAPA in community mental health and addictions services; and (2) identify the role of context in implementation.
Identifying relevant records
We developed the search strategy in consultation with a medical librarian.18 21 Sources included both published and grey literature. We conducted an initial search to familiarise ourselves with relevant terminology, which we incorporated into the search queries in multiple databases representing research from healthcare, social work and social sciences (MEDLINE, Embase, CINAHL, PsycINFO and Scopus). Records from 1 January 2005 to 13 July 2022 were considered for inclusion. Please see online supplemental file 1 for our full search strategies.
Supplemental material
Our database search was augmented by hand searching the reference lists of all included records, soliciting records from professional contacts and by reviewing the first 100 most relevant results of Google searches for ‘choice and partnership approach’ and for ‘CAPA’, updated to 20 July 2022.
Given the iterative nature of scoping reviews, we revised several aspects of the search strategy after publishing the study protocol. Deviation from the search methods outlined in the protocol includes the databases that we searched and the search terms used. The following databases were excluded due to their lack of unique or relevant content: Academic Search Premier, ERIC, Cochrane, Dissertations Abstracts, NCBI Bookshelf, PubMed Central and the Canadian Health Research Collection. Web of Science was excluded due to subscription cancellation at the researchers’ institutional library and replaced with Scopus. The database and grey literature searches were also expanded to include the term ‘CAPA’, in an effort to capture literature referring to the programme by acronym only. As the term ‘CAPA’ is not specific to the CAPA, the grey literature (Google) search added the terms ‘approach’ or ‘model’ or ‘program’ or ‘programme’ to improve the relevance of the search using the acronym.
Selecting records
After identifying potentially relevant literature, two members of the research team (LAC and SEC) independently screened records based on title and abstract. Records that could not confidently be excluded were carried forward to full-text screening. The reviewers met at the beginning, midpoint and end of the process to discuss challenges and resolve any ambiguity with the inclusion criteria. Any discrepancies were resolved via discussion, reaching consensus on each.
We used the following criteria to determine eligibility of records for inclusion:
Focused on CAPA, including its implementation, outcomes or a discussion of contextual factors that may impact its implementation.
Outcomes may include clinical, programme or system outcomes.
Study population included child and adolescent or adult population in a community mental health and addictions setting.
Context or setting was not limited.
Examined CAPA in its entirely, not just a component(s) of the model.
We did not exclude records based on methodology, quality of evidence, outcomes, the stage of CAPA implementation, record type, language or country of publication.
Charting the data
Team members LAC and SEC independently extracted data from the eligible records using a codebook developed in consultation with team members that reflected the five domains of the CFIR22 and included categories such as document identification, objectives, methods, contexts, implementation and outcomes. (Please see online supplemental file 2 for the detailed codebook.) The CFIR guided both data extraction and summation/interpretation, as we explicitly and systematically considered how context(s) were described in included records relative to the implementation and function of CAPA.22
Supplemental material
After independently coding three records, LAC and SEC compared data extraction to address any discrepancies and refine the codebook. Once completed, data extraction was reviewed for agreement and accuracy. Any discrepancies were minimal and were resolved by consensus.
Collating, summarising and reporting the results
We followed Levac and colleagues’21 extension of Arksey and O’Malley’s approach,20 by analysing the data using both numerical summary and thematic analysis to create a narrative synthesis and identify knowledge gaps. Data were first summarised as frequencies and ranges. Contextual and process-oriented data were then analysed using thematic analysis, mapped to the five CFIR domains.22 Finally, the resulting themes were reviewed by content experts on the team (SEC, JC, DE, JM) to verify and frame findings.
Consulting with stakeholders
Following the recommendation of Levac and colleagues,21 we included Arksey and O’Malley’s optional sixth step: consultation with stakeholders to increase methodological rigour and assist in framing our findings.20 Our research team includes researchers, clinicians, health system administrators, and policymakers. Several members of our team (SC, JC, DE and JM) are practising psychologists with direct experience in the implementation of CAPA. Furthermore, we reviewed our results more broadly with staff, multidisciplinary clinicians and administrators working in mental health and addiction services in different contexts (eg, general and specialised mental health and addictions services, urban and rural settings) during a day-long research workshop to assist in framing our findings and developing the recommendations presented in the Discussion section.
Patient and public involvement
While our overarching programme of research into the implementation of CAPA includes the involvement of clients and families or caregivers (see https://www.healthyyoungminds.ca), our review did not include direct involvement of clients (patients), families or the public. However, its undertaking was motivated by the observed need to better understand the barriers to and facilitators of the successful implementation of a client-centred and family-centred model of mental health and addiction services. It is anticipated that the results of this review will inform implementation and evaluation efforts, ultimately supporting improved outcomes for young clients and their families.
Results
Our database searches yielded 953 records (664 unique). The Google searches produced another 134 records. We obtained one record by soliciting our professional contacts and 16 via hand search of the reference list of previously included records. During the full-text screening phase, 170 records were assessed for eligibility and 48 records (corresponding to 36 unique evaluations) met our inclusion criteria and were included for data extraction. A detailed description of search results, along with reasons for exclusion, is presented in figure 1.27
Preferred Reporting Items for Systematic Reviews and Meta-Analyses for Scoping Reviews (PRISMA-ScR) flow diagram of the Choice and Partnership Approach to community mental health and addictions services.27
In some instances, individual evaluations were presented in multiple formats (eg, report, journal article and presentation), which we refer individually to as ‘reports’. We included all reports to ensure capture of contextual information, but for the purposes of synthesis of findings, we considered reports at the level of the evaluation to avoid double counting. Two reports28 29 represented ad hoc summaries of CAPA evaluation efforts prior to 2010, so included several of the evaluations (n=17).
Characteristics of included reports
Characteristics of included reports are listed in online supplemental file 3. Publication dates ranged from 200630–32 to 2022.33 All reports were written in English. Reports were limited to four regions: the UK,15 17 28–32 34–57 Canada,13 16 33 58–63 New Zealand28 29 64–67 and Australia.14 28 29 68 69 Despite CAPA having also been implemented in Norway, Belgium and Ireland, we did not identify any reports from these settings in our searches.
Supplemental material
Most evaluations (n=31/36) were local or regional in scope and situated in urban centres or mixed urban, suburban or rural settings13 16 30–32 34–37 39–54 56 58–62 66 67 69; one described a rural context.14 68 Two represented national evaluations of CAMHS that had implemented CAPA across England15 17 and New Zealand.64 65 Only three evaluations included services that provide care to adult and/or geriatric populations.16 28 29 62
The evaluations did not include any experimental designs, and few (n=10/36) reported pre–post comparisons.13 14 31 36 43 45–47 54 58 60 61 63 68 Sampling strategies, when described (n=7/36 evaluations), were largely of convenience13 15 17 34 36 39 43 58 68; none employed random selection.
Context—the intervention
Many (n=20/36) evaluations reported the motivation for the implementation of CAPA, including to reduce wait times or waiting lists,13 14 33 35 37 39 43 55 57–59 62 62–64 68 improve efficiency,14 34 35 38 40 42 69 improve care quality, service user experience or accessibility,13 17 40 56 69 choice in service,39 41 43 60 68 meet service demands or client needs or values,14 15 17 39 42 58 60 68 provide client-focused service,15 17 41 43 58 60 support staff,40 43 provide transparency39 40 and provide meaningful data.40 Few evaluation (n=10/36) cited theories supporting how CAPA or its components ‘work’; those that did most often reported that CAPA’s strength as a service delivery model is in its efficiency in managing demand/capacity.13–17 33 35 36 43 55 58 63 68 69 Fewer evaluations (n=5/36) mention that CAPA ‘works’ because it provides client-centred services.17 35 41 43 62
Context—inner setting
No evaluations reported the CFIR inner setting constructs of ‘structural characteristics’ (eg, the social architecture, age, maturity and size of an organisation or service), ‘networks and communications within the organisation’, ‘culture’ (ie, norms and values) or ‘readiness for implementation’.22 An evaluation of CAPA implementation in a specialist setting reported that clinicians felt stressed and overwhelmed by workloads prior to implementing CAPA.40 Other sources reported organisational challenges such as staffing issues (clinical staff69 and psychiatry59 understaffing, mismatch of clinician skills for client population60 and procedural problems (eg, complex assessment process,68 poor throughput68 and arbitrary intake process)).69
Context—outer setting
Some (n=14/36) evaluations referred to constructs within the CFIR outer setting domain, including ‘client needs and resources’, ‘community characteristics’ and ‘pressures, policies or incentives that implicate the service’.22 Services described caring for complex, severely ill or special client populations68 dispersed populations57 68 or populations with a wide range of needs,36 and two served specific care populations (clients with mood and anxiety disorders16 and learning disabilities34 57). One reported redesigning their centralised referral system and creating specific care clinics for severely ill clients, or those requiring specialised skill sets, to support the implementation of CAPA.13 Other considerations included culturally relevant care for Māori and Pacific clients in New Zealand,64 65 and Indigenous and racialised communities in Nova Scotia, Canada.62
CAPA implementation often occurred within contexts of low resources,36 58 pressure to meet or maintain the ability to meet demand,17 42 58 61 65 69 lack of second-tier services,68 increasing expectations from the public,65 challenges in access to care (due to long wait times,15 lack of second tier services,68 siloed or fragmented services59 60 or poor coordination of services60 68) or inequitable access,17 and the need to provide high-quality, evidence-based care.15 17 36 69 The UK and New Zealand governments influenced implementation through directives aimed to ameliorate challenges in mental health service delivery by setting goals for mental healthcare, including wait time benchmarks.36 43 Some services cited UK government endorsement of CAPA as a means of improving service efficiency, adding value, eliminating waste, and reducing wait times.34 35
Context—characteristics of individuals
Evaluations did not report the characteristics of individuals in the service who were implementing CAPA, such as their knowledge and beliefs about CAPA, self-efficacy, individual stage of change, identification with the service or organisation or other personal attributes that may affect implementation.22 One evaluation reported that major concerns for clinicians prior to adopting CAPA were that the quality of care would be negatively affected by increased client throughput, leading to poorer outcomes and that there would be difficulties in handing over families between clinicians from Choice to Partnership.36 However, these did not emerge as major themes in their findings postimplementation.36 Another evaluation suggested that considering individuals’ readiness to change would be important for employing appropriate change strategies, such as support networks.43
Context—process of implementation
Efforts to support adaptation and planning for the implementation of CAPA were varied and included the development of implementation teams,68 formal43 55 68 or informal60 planning meetings or team away days to discuss CAPA,17 55 60 65 66 68 and the collection of data regarding client needs or clinical presentations69 or the service capacity (eg, determining the number of available Choice and Partnership appointments or the skills within the service).40 60 69 Services conducted waitlist blitzes (periods of time during which waitlists are reviewed for determination of individuals’ eligibility for entrance to the service and match with capacity)13 17 43 45 48 60 63 68 70 articulated eligibility and redirection criteria60 or staggered implementation across teams to facilitate implementation.15 68 Some adapted their services by redistributing clinicians from specialist to multidisciplinary teams,13 creating emergency Choice appointment tiers to ensure wait time targets for both children in crisis or not in crisis,68 creating care bundles40 or enhancing supports for less-experienced clinicians to conduct Choice appointments, such as by pairing with more experienced clinicians or providing training.15 17 In two instances, adaptations of CAPA such as those requiring all clients to be seen by a psychiatrist16 or limiting the number of sessions with clients62 were incompatible with the CAPA model.
Key themes related to CAPA implementation observed across the evaluations emerged, including facilitative or engaged leadership, data-informed planning and monitoring and training in CAPA. Facilitative leadership was identified as a key contributor to successful implementation.15 17 30 40 43 60 65 While full commitment from senior leadership was identified to be important,40 60 65 the need for consistent, clinically informed leadership was deemed critical to successful implementation.15 17 65 Clinical leads and managers with clinical backgrounds offered credibility and the ability to liaise effectively with all team members15 17 as a starting point, but alignment of the services’ senior leadership was critical for consistent messaging and ongoing support during system transformation.65 Champions or change leaders in management were noted to be influential by promoting staff buy-in,17 43 65 but they needed to be well respected, knowledgeable about CAPA, responsive to staff concerns17 and represent all parties involved43 to be effective.
Engagement of leadership was operationalized in various ways, including through the collaboration of clinical leads or senior clinicians and service managers,17 60 68 weekly meetings of clinical team leaders,13 and regular email updates and weekly drop-in sessions discussion of general CAPA issues to bridge the gap between once monthly meetings.43 Pressure to implement CAPA from senior management outside the team could lead to inadequate preparation (in terms of lack of time and/or resources, or adequately prepared team management) for implementation, which in turn may have caused resistance from the teams themselves.17 65
Team away days, one of the 11 core components of CAPA, were noted to provide opportunities for implementation planning, reflection and evaluation of CAPA to improve the service,17 60 65 66 68 while monitoring and feedback to teams were deemed essential for identifying ‘teething problems’17 or ‘drift’65 during implementation, very few evaluations (n=2) reported ongoing quality monitoring activities, such as robust information and data collection systems within teams or processes for review.42 65 Monitoring was supported by the development of process goals and metrics13 65 but was noted to be done largely manually by teams17 or as individual audits.15 35 36 40 68 Lack of feedback was identified as a barrier to implementation.17
CAPA training was reported to be important for supporting successful implementation13 17 36 43 58 60 66 but was noted to be variable in intensity between services.17 Importantly, training was identified as a means of providing opportunities to address misconceptions of the model,17 which included the common misunderstanding that CAPA limits the number of sessions per client,17 33 62 and the assertion that the model is based on averages without means for adjustment.16
Mechanisms—CAPA components and fidelity to the CAPA model
From a realist lens, Pawson and Tilley conceptualised mechanisms as a combination of both resources and stakeholders’ reasoning in response.23 Accordingly, we sought to capture the reporting of the 11 key components of CAPA both in terms of resources and responses.
Of the 11 key components of CAPA, the Choice components, Choice framework13–17 30 34 35 38 40 43 60 61 65 66 68 and ‘handle demand’15 17 28 36 38 40 41 43 60 68 were most often cited, while the Choice component ‘language’ was cited less frequently.15 17 38 60 68 One evaluation noted that while a change in language was met with mixed views by clinicians, particularly more experienced clinicians who held on to the traditional language of assessment and treatment, inclusive language was identified as a core theme related to successfully moving from a model of diagnostic assessment to one of joint formulations and goal development.68
Few evaluations reported fidelity to the CAPA model (n=9/36); those that did either counted the number of the 11 key components implemented,15 17 28 29 65 71 or scored the CAPA Component Rating Scale (CAPA-CRS).13 45 48 53–55 72 None reported full fidelity. In the evaluation of CAPA across England, of 53 CAMHS teams who implemented CAPA and responded to follow-up questionnaires, 28 were self-reported ‘medium implementers’ (implemented 5–7 of the 11 components) and 18 were ‘high implementers’ (implemented 8+components).15
Context and mechanisms of implementation
Figure 2 consists of a heat map that depicts the frequency by which the key components and fidelity to CAPA were reported by the five CFIR domains at the level of evaluations (to avoid upweighting cells by ‘double counting’ reports). Cells with higher intensity shading represent larger numbers of evaluations in that cell. Outer setting factors were most often described, and characteristics of individuals were least often described in evaluations of the implementation of CAPA.
Heat map depicting frequencies of evaluation reporting key components of CAPA by CFIR domains. CAPA, Choice and Partnership Approach; CFIR, Consolidated Framework for Implementation Research.
Services that reported outer setting pressures (eg, needing to reduce wait times or increase throughput) often reported implementation of the process-related components of CAPA, such as handle demand, Choice framework, care planning and job planning. Fewer reported the more philosophically oriented components such as language, peer group discussion or Team away days.
Each of the five CFIR domains consists of several constructs. To further examine each of the CFIR domains, we also mapped the frequency by which the key CAPA components and fidelity to CAPA were reported by individual CFIR constructs (see figure 3; heat map). With respect to intervention characteristics, the efficiency of CAPA (eg, clear procedures, lean thinking, queuing theory or flow through service) was the most commonly reported construct, followed by client-centred care or client choice. Those evaluations citing the efficiency of CAPA most often implemented the Choice framework, job planning and goal planning/care planning components. In terms of outer setting constructs, government endorsement, external targets or external review and long wait times were the most frequently cited and most often reported the implementation of the leadership and management, Choice framework, job planning and goal setting/care planning components. Within inner contexts, staff pressure, shortage or morale were most often cited.
Heat map depicting frequencies of evaluations reporting key components of CAPA by CFIR constructs. CAPA, Choice and Partnership Approach; CFIR, Consolidated Framework for Implementation Research.
Reporting of the characteristics of individuals within teams or services (eg, knowledge and beliefs about the model, staff buy-in, personal stage of change, staff skills) was notably absent; however, staff skills were reported in one evaluation that implemented all CAPA key components.68
Several constructs associated with the process of implementation were reported, most commonly relating to leadership, formal training of team members, dedicated implementation teams and regular meetings. Less frequently, teams reported service-specific adaptations such as the addition of emergency Choice streams or single access points to the service. As with other CFIR constructs, these were most often reported with the implementation of Choice frameworks.
Outcomes of implementation
The most frequently reported outcomes were related to the health system (n=28/36 evaluations)13–17 28 29 31 34–36 38 40–43 45–48 50–55 57–61 63–69 71 and workforce (n=19/36 evaluations).13 15–17 28–32 36 38 40 43 46 48 51 53 54 58–61 65 66 68 69 Health system outcomes included numbers of clients seen by the team or service (including numbers of accepted referrals and first visits or Choice appointments),13 14 16 17 28 29 36 42 46 61 65 proportions of clients going on to attend second visits or referred elsewhere,14 17 35 36 41 43 45 46 60 66 wait times to first appointment (Choice appointment),13–17 28 29 31 36 40–43 45–48 50 54–61 63–66 68 71 wait times from Choice to first Partnership appointment or between Partnership appointments,13 15–17 28 29 41 46 48 51 58 61 63 65 wait time targets,28 29 38 42 43 53 71 waiting lists17 37 56 58 68 69 and ‘no-show’ or ‘did not attend’ rates.13 17 28 29 31 41 43 46 58 60 66
Workforce outcomes included job satisfaction, staff stress, morale, confidence or engagement,15 17 28–30 32 38 40 43 46 48 53 54 65 66 69 efficiency or provider productivity,13 46 59 61 66 collaborative teamwork or team cohesion,15 17 28–30 32 38 59 60 66 caseloads46 60 66 68 and transparency or accountability between clinicians or with clients.17 36 59 65
Acceptability of CAPA was captured (n=16/36 evaluations) from the perspectives of children, young people and families.13 17 28–31 36 39 44 45 47–50 58 60 64 66 67 Measures of client and family experience included the Experience of Service Questionnaire13 30 31 36 39 47 63 73 and the locally developed CAPA or Choice Experience Questionnaires.30 36 47 Clinical outcomes were infrequently captured (n=7/36 evaluations)14 30 31 36 41 64 66 and reported only in child and adolescent services, with teams measuring service effectiveness through the capture of treatment goals,36 41 64 66 the Health of the Nation Outcome Scales for Children and Adolescents,14 74 the Child Behaviour Checklist,66 75 the Strengths and Difficulties Questionnaire31 76 and an adapted clinician-rated Clinical Global Impressions Scale.36 77 Only three evaluations reported pre-CAPA/post-CAPA changes in clinical outcomes.14 31 36
Follow-up periods were short, with the longest follow-up of health system outcomes reported to be 18 months following implementation,68 and clinical outcomes to the point of closure or transfer or from the service.14 36 The national evaluation of CAPA implementation in England included perspectives of clinicians in services with an average of 18 months following implementation (range 7–30 months).15 17
Discussion
In this comprehensive scoping review, we identified 48 reports stemming from 36 unique evaluations of the implementation of CAPA. CAPA has been implemented in countries with differing health systems and opportunities for private/public health insurance. However, regardless of country of implementation, the transformation of mental health services through the implementation of CAPA is often undertaken by small teams without the resources to conduct formal evaluations or research. As such, we recognised the need for an inclusive search strategy to accurately capture the scope of implementation and to identify important considerations regarding context that may not appear in the formal literature. Accordingly, we did not restrict our search by methodology, quality of evidence, outcomes, the stage of CAPA implementation, report type, language or country of publication. While we did not assess the quality of included reports in order to include all relevant literature and provide a comprehensive overview of the scope of implementation, we recognise that this may have limited our ability to assess gaps in the literature.78 We are also aware of the possibility of publication bias introduced through over-representation of positive experiences.
Evaluations of CAPA implementation were exclusively observational in design, with some (n=10/36 evaluations) considering baseline or pre-CAPA data for comparison.13 14 31 36 43 45–47 54 58 60 61 63 68 At a minimum, the routine inclusion of both pre-CAPA and post-CAPA implementation data would strengthen the evidence base. As CAPA is a highly complex intervention intended to be adapted to meet the needs of individual services that function in different contexts and health systems, often with limited research and/or data resources, randomised controlled trials are likely infeasible and may not capture noteworthy contextual considerations necessary for successful generalisability and implementation.79 More pragmatic designs that capture important sources of heterogeneity, such as well-designed controlled before-and-after, interrupted time series, or stepped wedge cluster trial designs—provided resources are available to support the latter’s complex conduct and analysis—are likely more useful for informing policy recommendations.80 81 Mixed methods approaches would offer the opportunity for triangulation of theory, data and previous evaluations.
Demand and capacity concepts,82 83 lean thinking principles84 and queuing theory85 all suggest that tracking demand and service capacity within a quality improvement framework to support review allow for better planning and more efficient use of resources. However, the lack of data for providing feedback to clinicians and staff and for monitoring ongoing service performance and client outcomes was identified as a common barrier. Meaningful and timely data collection is noted to be a considerable gap for many mental health services, and particularly so for small, often under-resourced teams with little access to administrative or database infrastructure or supports. A recent systematic review revealed that despite the benefits of data-driven learning health systems, there remain significant challenges in uptake in healthcare more broadly due to barriers related to governance and regulatory systems, and technical, quality and interoperability problems.86
Meaningful evaluation of CAPA implementation would also be strengthened by consideration of fidelity to the model. Our review captured inaccuracies in the interpretation and application of CAPA, which likely contribute to unsuccessful implementation.16 17 33 62 The architects of CAPA strongly encourage implementation of all 11 key components; noting that ‘using CAPA principles’ or implementing ‘CAPA-lite’ is unlikely to lead to meaningful system transformation or may reflect reluctance to change.11 However, few evaluations reported fidelity to the model. This may reflect the state of change at the time of measurement or incomplete implementation. Future evaluations should include measurement of fidelity to CAPA to ensure that the accuracy of outcomes attributed to the model (both positive and negative) and to support ongoing monitoring to help prevent falling into previous ways of working. Without measuring and reporting on the fidelity to the CAPA components, it is impossible to know what in the implementation of ‘CAPA’ was changed in the way the service was organised and what the client and family may have experienced in their care. Incomplete or unsuccessful implementation that results in poor outcomes may be incorrectly reported as CAPA ‘doesn’t work’. Measurement tools designed to assess fidelity to the CAPA model include the CAPA-CRS,72 CAPA Pragmatics Rating Scale87 and the CAPA FACE: The Fidelity Assessment and Component Evaluation.88
Our review offers important insights into considerations of context in implementation efforts. Most evaluations reported CFIR constructs falling under the ‘intervention characteristics’, ‘outer setting’ and ‘implementation process’ domains. There was limited information available regarding the inner context (eg, team composition or service milieu) or the characteristics of individuals (eg, staff buy-in or skills). This is notable, as the implementation of CAPA often requires significant service transformation at the heart of which clinicians and staff are required to change, which may include shifting from a known way of working (typically introduced in training) and embracing new identities and new tasks in an unfamiliar system and often while a system is under stress.89 90 Future evaluations would be strengthened by attention to and measurement of constructs associated with the characteristics of the service team and individual members, and leadership and change processes to support the analysis of their impact on successful implementation.
The stance of CAPA, while centred on the client and family experience of care, was not reported to be the primary motivator for teams in selecting this approach to care as system accessibility problems are typically the focus for initiating change of this magnitude. While services in the UK and New Zealand cited government directives intended to improve service delivery, our review demonstrates that CAPA implementation efforts, regardless of setting, were largely motivated by needs to reduce wait times and to improve efficiency of services. As it is possible to improve initial waits to Choice (first) appointments at the expense of waits to or between Partnership appointments, it is essential to consider all wait times throughout the client experience of care.15 17 Because improved wait times are often an outcome of CAPA implementation, further exploration of teams’ understanding of the client experience as a motivator may provide additional valuable implementation guidance.
Perhaps stemming from the motivation for implementation, or the relative ease of capture, the most commonly reported outcomes were those related to the health system (eg, wait times, percentage of clients seen within target time periods, or attended visits) and workforce outcomes (eg, staff experiences). While some evaluations benefited from pre-CAPA and post-CAPA implementation audits supported by service data, an important limitation of administrative data is that attended sessions do not necessarily represent those required to meet client needs, so may underestimate need.34 A critical problem in the evaluation of CAPA, and of mental health services in general, is in the lack of measurement of client outcomes. In our review, only seven evaluations reported clinical outcomes, all reporting positive findings.14 30 31 36 41 64 66 However, only three measured changes from baseline.14 31 36 Patient-reported outcome measures, such as the Revised Children’s Anxiety and Depression Scales or Goal Based Outcome Tool, while recommended by healthcare systems internationally and demonstrated to benefit shared decision-making,91 were not often reported. Of the four evaluations that reported whether goals were set,36 41 64 66 only one captured post-treatment ratings, for which only half of the clients with baseline goals had post-treatment ratings for analysis.36 The paucity in measurement and reporting of client outcomes is commensurate with existing literature,92 and within mental healthcare has been a particular challenge as there is no standard for outcome measurement in clinical practice and recent standardisation of measures for research have the potential to introduce unintended consequences, including lack of transferability and narrowness of scope.93 Importantly, meaningful outcome measurement requires an understanding of the nuances or potential differences between measuring what matters to clients and families and what is often required for reporting to governments or other payers.
Similarly, the views of families or caregivers were under-represented among our findings. In the national evaluation of CAPA implementation in England, a key challenge identified was that of accessing the views of families.17 The authors noted that few attended the focus groups, and among those who did, none had heard of CAPA.17 They posit that the topic area may not be of relevance to families, or recruitment may have been hampered by Research Ethics Committee restrictions on direct recruitment by research teams. It would also be reasonable to consider that families may not know what ‘CAPA’ is as for them, it may just be the way a team works, which may be a function of how we talk with families about the way that services work both in their delivery and evaluation.
Conclusions
The transformation of mental health services to those that place clients and families at the centre of care can measure client-centred outcomes, tailor care and actively engage clients and families in the care process as aligned with the CAPA model, often requires major philosophical and organisational shifts in the way services are delivered and evaluated. Evaluations of implementation of CAPA in the face of complex system change would benefit from the consideration and capture of contextual factors to support its adaptation to different settings, measurement of fidelity to the model to ensure the validity and reliability of findings and to provide feedback during ongoing implementation, consideration of constructs related to the inner contexts of services (eg, team composition, staff pressures) and characteristics of the individuals involved in or affected by implementation (eg, staff buy-in for the model, skills and readiness for change) and the consistent capture of outcomes of importance to clients and families. Equally important are avenues for sharing experiences between teams, identifying facilitators and barriers to successful implementation, creating reliable evaluation and research metrics and sharing practice challenges that appear to be common during mental health service transformation within western healthcare systems.
Data availability statement
Data sharing not applicable as no datasets generated and/or analysed for this study. The reports included in this scoping review are available via the citations.
Ethics statements
Patient consent for publication
Ethics approval
The IWK Health Research Ethics Board approved the overarching research project, including this review (Title: Transforming Care in Nova Scotia: Implementation of Health System Change in Child and Adolescent Mental Health and Addictions, Project #: 1024356).
Acknowledgments
We are grateful for the funding from Research Nova Scotia (formerly Nova Scotia Health Research Foundation) Establishment grant number PSO-EST-2018-1564 to support this work and to Ms. Kathleen MacNabb for assistance with data management and copy editing and Ms. Kirstie Smith for reference management. The literature search was supported by Evidence Synthesis staff with the Maritime SPOR SUPPORT Unit. We thank them for their support.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Twitter @sharon13clark, @debbie_emberly, @GraceWarner_dal
Contributors All authors meet the ICMJE criteria for authorship. LAC, SEC, JC, DE, JM, AM, GW and LW collaborated on the proposal and methodology. LAC and SEC jointly extracted data. LAC and SEC conducted the data analysis and initial interpretation. LAC wrote the original draft and all authors reviewed and contributed to the revision of the manuscript. LAC is the guarantor and accepts responsibility for the work and decision to publish.
Funding This work was supported by Research Nova Scotia (formerly Nova Scotia Health Research Foundation) Establishment grant number PSO-EST-2018-1564.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design or conduct of this research; please see the Methods section for details.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.