Article Text
PDF
Abstract
Introduction Implementation evaluations provide insight into how interventions are delivered across varying contexts and why interventions work in some contexts and not in others. This manuscript outlines a detailed protocol of an implementation evaluation embedded in a stepped-wedge cluster randomised controlled trial of a model of care, Strengthening Care for Children (SC4C), that integrates paediatric care in general practice. The purpose of this manuscript is to describe the pragmatic methods that will be used to capture implementation evaluation process and outcome data within this trial.
Methods and analysis Our implementation evaluation will use a mixed methods design, with data collected in the intervention arm of the SC4C trial guided by a logic model developed using the Consolidated Framework for Implementation Research (CFIR) and Proctor and colleague’s taxonomy of implementation outcomes. Data collection will be via questionnaires and semistructured interviews with general practitioners, paediatricians, general practice administrative staff and children and families. Each of the 21 general practices recruited into the study will be described in terms of staffing, patient throughput and location, in addition to the nuanced inner and outer contexts, use of the intervention and its acceptability. We will quantify implementation effectiveness in each general practice clinic using the CFIR validated scoring system. Importantly, we have embedded data collection post intervention to enable assessment of the sustainable adoption of the intervention. An inductive approach to the analysis of qualitative data will identify additional emerging themes that may not be covered by the formal frameworks underpinning our analysis.
Ethics and dissemination Ethical approval was granted by the Royal Children’s Hospital Ethics Committee in August 2020 (HREC: 65955). Results will be submitted for publication in peer-reviewed journals and presented at relevant conferences.
Trial registration number Australia New Zealand Clinical Trials Registry 12620001299998 on 1 December 2020.
- PAEDIATRICS
- PRIMARY CARE
- Community child health
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
A strength of the methodology is the use of mixed methods to examine the barriers, facilitators and lessons learnt for a paediatric and primary integrated model of care.
A strength of the design of this implementation evaluation is the logic modelling process used to map the implementation context and intervention components to guide data collection methods.
This study is limited by a lack of direct observation of general practitioners and paediatricians in their role in the model. While this would provide objective fidelity data, it is not representative of how the model of care will be sustained following the conclusion of the trial.
As the qualitative interviewer is a member of the research team, there remains a risk that demand effects may impact the information that is shared. Efforts will be made to compartmentalise the implementation evaluation from the main trial by ensuring the implementation evaluation team have limited contact with general practices prior to the interview to minimise bias.
Background and rationale
Implementation research represents ‘the scientific study of methods to promote the systematic uptake of evidence-based interventions into practice and policy and hence improve health… it includes the study of influences on professional, patient and organisational behaviour in healthcare, community or population contexts’.1 Evaluating the implementation of complex interventions helps to identify the reasons why interventions work in some contexts and not in others. By accounting for the ‘real world’ factors of implementation, these evaluations can provide necessary insight to improve how interventions are delivered and adapted for specific contexts and systems.2 Considering the features of the organisation or broader environment that influence the delivery of the intervention (eg, leadership, engagement, culture, political landscape) is necessary to aid interpretation of trial outcomes, maximise the knowledge gained from trials, identify optimal delivery processes across different settings and inform broader dissemination efforts. These evaluations are highly valued, as evidenced by the UK’s Medical Research Council framework that emphasises the importance of capturing both contextual and implementation factors associated with complex interventions.3
The successful implementation of integrated care requires ‘an effective composition of a complex set of interventions at the microlevels, mesolevels and macrolevels’.4 The evaluation of the implementation of integrated care interventions in primary care for adults is a burgeoning field; however, there has been far less attention to integrated paediatric and primary care.5 6 Platt et al’s scoping review characterised the literature about implementation of colocated integrated interventions in primary care for children.7 In their review, they noted a clear need for research in integrated care for child health to better understand the specific opportunities and challenges for integrated paediatric and primary care. Our evaluation will contribute to this field by evaluating the implementation of Strengthening Care for Children (SC4C), a general practitioner (GP) and paediatrician integrated model of care.
Strengthening Care for Children (SC4C) Trial
In line with the best practice recommendations, study protocols that prespecify methods and approaches should be published to maintain research integrity.8 We describe a protocol for a mixed methods implementation evaluation embedded within the SC4C-stepped wedge cluster randomised controlled trial (RCT). Based on a trialled model in the UK,9 piloted within the Australia context,10 the SC4C RCT trial aims to reduce GP referrals to hospital emergency departments and outpatient clinics. The model consists of regular, shared GP paediatrician co-consulting sessions and case discussions held at the general practice clinic, with email and telephone support provided by paediatricians to GPs during weekdays. Following a successful pilot study with five Victorian general practices, the SC4C model reduced GP referrals to hospital services, improved family trust in the GP and improved GP confidence in providing paediatric care.9 The specific features of this model can be found in our trial protocol companion paper.11 Sequentially, one practice from each site per month will switch from control to intervention until all general practices receive the intervention from 2021 to 2023. GP practice recruitment commenced in January 2021 with study paediatricians commencing in the first practices in June 2021. Data collection will continue until September 2023.
Objectives
We aim to undertake the implementation evaluation of the SC4C model concurrently with the trial to understand how SC4C is implemented and delivered in general practice, and to identify the local contextual differences and approaches to delivering the intervention. Through the implementation evaluation, we aim to capture important information at the general practice, practitioner and patient/parent level, which may ultimately impact the adoption of the intervention and its effectiveness to reduce referral rates and to improve outcomes. We also aim to provide insight into factors which support and/or hinder the implementation of the SC4C model in primary care settings to inform strategies for optimising implementation at scale. Additionally, our evaluation of the implementation of SC4C will need to factor in the impact of the COVID-19 pandemic on primary care and general health service delivery. Purposefully considering the elements of implementation affected by pandemic conditions may prove to be an important contribution to the implementation of integrated care interventions.
Methods and analysis
Study design
Our mixed methods implementation evaluation will identify and assess the impact of barriers and facilitators to implementation of the SC4C model as guided by the Consolidated Framework for Implementation Research (CFIR).12 The CFIR is a comprehensive framework designed to ‘offer an overarching typology to promote implementation theory development and verification about what works where and why across multiple contexts’.12 The CFIR is a determinant theoretical framework that has been widely used in diverse healthcare contexts, including primary care settings.13 Determinant frameworks such as the CFIR describe general domains of barriers and facilitators that influence implementation.14 The CFIR identifies five major domains (intervention characteristics, outer context, inner context, characteristics of individuals and process) and provides a framework to guide the consideration and assessment of factors which might impact intervention implementation and effectiveness. In addition to mapping the barriers and facilitators to implementation of the SC4C model, the study team will evaluate specific implementation outcomes, as guided by the taxonomy proposed by Proctor et al.15 These outcomes include intervention acceptability, appropriateness, fidelity to the implementation strategy, coverage and sustainability (table 1). An economic evaluation running in parallel to the trial and implementation evaluation will analyse cost-effectiveness.
Proctor and colleague’s implementation outcomes mapped to SC4C evaluation
Logic model
The integrated clinical academic research team developed a logic model to inform the trial and the SC4C process evaluation (figure 1) in a series of workshops. We used a modified version of existing logic model frameworks16 17 to ensure the inclusion of the specific contextual factors the implementation evaluation would need to consider both within each of the sites (inner context representing individual factors and organisational settings) and external to the sites (outer context, including area demographics, policy climate and relevant geographically adjacent clinical services). These organisational contextual factors were incorporated within the logic modelling based on their relevance for enabling implementation researchers to more fully describe the determinants of successful implementation in clinical practice.18 For example, implementation research in primary care settings in Australia has identified the importance of interventions working within existing workflows and organisational logics.19
Strengthening Care for Children process evaluation logic model. ED, emergency department; GP, general practitioner; OP, outpatient; PHN, Primary Health Network.
In addition to the contextual factors, we included a detailed description of measurable intervention characteristics such as the number of scheduled co-consultations and case discussions between GPs and paediatricians (formal monthly discussions and one-on-one discussions), use of the phone and email support services monitored by paediatricians during business hours, extent of administrative support including regular meetings with the project team, booking management and use of project resources, and the nomination of a ‘champion’ of SC4C within each practice. The practical elements of the intervention are underpinned by theoretical principles including Wenger’s concept of communities of practice and elements of the behaviour change wheel.20 21 Consideration of the theoretical principles of perceived mechanisms of change within the logic model was a novel development, which will allow exploration via lines of questioning and probes in qualitative interviews. Finally, we will draw connections from these underlying theories of change to the specific intermediate and long-term outcomes we hypothesised the model would produce.
Importantly, our logic model helped us to identify the key CFIR constructs that our evaluation would focus on as part of data collection and analysis, which were similarly identified by the previous literature as being feasible and appropriate to measure within general practice contexts.19 22 23 Our methods and analysis were developed by visually mapping these CFIR constructs, along with their subconstructs, to represent how we hypothesise they will impact implementation and clinical effectiveness outcomes. The logic model has guided the implementation evaluation mixed methods approach to data collection including surveys, interviews and focus groups.
Study setting
The implementation evaluation sample includes all 22 general practice clinics (11 in Victoria, 11 in New South Wales) participating in the SC4C trial within the North Western Melbourne Primary Health Network and the Central and Eastern Sydney Primary Health Network catchment areas.
Study participants and recruitment
Participants will include GPs, administrative staff, patients<18 years and their families, and project team members after consent is given. In the SC4C pilot study, the study team recruited a total of 49 GPs within 5 practices. Therefore, across 22 practices, we have the potential to recruit between 100 and 200 GPs and practice staff. We will attempt to engage in the implementation evaluation with all participating GPs at each practice who see paediatric patients (<18 years) and work at the practice at least two sessions (1 day) per week. Additional participants will include SC4C paediatricians delivering the model of care, the practice manager and administrative staff for each of the practices in the trial (where they are not a GP already recruited), patients and their families, and project team members. Patients and their families who have participated in a co-consultation will be recruited during the family survey intervention period, which will include an item seeking permission to contact them about the opportunity to participate in a qualitative interview. The interview can be conducted either in person, online via video or via telephone. We aim to conduct 20–30 interviews with parents/caregivers across both sites with or without their children (as decided by the parents/caregivers).
Data collection
Description of local context and practice
Table 2 provides a summary of the data collection methods. To understand the characteristics of general practice outer context, we will collate a profile of the catchment area including child population served, and socioeconomic indexes by postcode as provided by the Australian Bureau of Statistics and patient profile of each practice recruited to the trial. We will also describe any policies or programmes relevant to paediatric care implemented by the participating general practice clinics.
Methods and measures and outcomes by participant type
Fidelity to the model
To determine how the intervention is being carried out in each site as compared with the intended roll-out, we will draw on trial data, which will record the specific details of model delivery. This will include recording:
Number of co-consultations attended by participating GPs.
Number of formal monthly case discussions and attendance of GPs at each practice.
The frequency of one-on-one discussions between GPs and SC4C paediatricians.
The frequency of the use of phone and email support.
Acceptability, appropriateness, feasibility and GP buy in
We will supplement the GP surveys collected concurrently with implementation with additional items measuring GP buy in to the model and appropriateness. To determine GP buy in to the model of care, we will ask them at the completion of the data collection section of the trial to complete an adapted version of the Normalisation MeAsure Development (NoMAD) tool based on the Normalisation Process Theory.24 The NoMAD tool will assess how the intervention was incorporated into standard work responsibilities. These 16 items are grouped into three categories: coherence (ie, making sense of an intervention), cognitive participation (ie, working with others to support an intervention) and collective action (ie, the type of work that people do to support an intervention). Items are rated on a 5-point Likert scale from 1 (strongly agree) to 5 (strongly disagree). We will also include the intervention appropriateness measure (IAM). The IAM is a pragmatic 4-item measure of the perceived fit, relevance or compatibility of an evidence-based practice for a context, person or problem. Items will be adapted for this study and are rated on a 5-point Likert scale from 1 (strongly agree) to 5 (strongly disagree). Scores will be re-coded such that higher scores indicate higher levels of perceived appropriateness.
Barriers and facilitators to running SC4C
To determine individual’s knowledge and beliefs about the model of care; relative advantages of the model of care; GP and paediatrician self-efficacy; barriers and facilitators affecting the delivery of the intervention both from an individual and organisational perspective; the appropriateness and acceptability of the intervention; and recommendations for future implementation, we will conduct semistructured interviews and focus groups with all participant groups at various stages during the trial. The study team have derived interview guides based on the logic model and CFIR online resources (https://cfirguide.org/evaluation-design/qualitative-data/). The theoretical underpinning of the model and the use of the behaviour change wheel constructs will allow a deeper understanding of what has changed in the practices and how this has affected participants.21 Practice managers will specifically be asked about how the model of care affected the normal operation of general practices. GPs and paediatricians will specifically be asked about features of the working relationship, for example, the collaborative nature of the relationship. Interviews with patients and their families, recruited via parents indicating their consent to be contacted to arrange an interview as part of the family survey, will determine their perceptions of the acceptability of the SC4C model and any potential adaptations to the model to make it more acceptable for children and their families presenting to GP clinics. This type of interview is known as a dyad and has been used previously to allow a parent to expand a child’s cognitive abilities, creating more meaningful data.25 26 Dyadic interviews involve two participants, and are often structured to allow discussion between participants, in this instance a parent and child. Dyadic interviews will help us to ascertain more detailed information about a child’s experience attending a co-consultation with a paediatrician and a GP. A dyad approach provides children with access to their parent as an important ethical and social support and resource for understanding and responding to questions. Interviews with project team members will provide detail about the process of implementing the model from an implementer perspective.
Data analysis
In this study, reliability, validity and confidence will be maximised through cross verification and exploration of differences between the findings from interviews with results from the questionnaires, exploring and accounting for differences and mapping the perspectives of different stakeholders across the study. Quantitative questionnaire data will be exported into SPSS 28/STATA for analysis. Descriptive statistics will be calculated for each of the 22 practices recruited in the study including information about inner and outer context and the intervention use and its acceptability. Any open-ended questions will be analysed and, where possible, a coding scheme will be developed to enable descriptive analysis, and where this is not possible, open-ended questions will undergo inductive thematic analysis.
The study team will audio-record and transcribe interviews verbatim and thematically analyse the transcripts to identify, interpret and report on the repeated patterns of meaning within the data, influenced by the CFIR constructs and drawing from Braun and Clark’s thematic analysis approach.27 Where appropriate, NVivo software will aid in the coding and organisation of themes. Implementation effectiveness will be evaluated using the validated scoring system of −2 to +2 with score descriptions as follows: −2: the construct had a negative influence in the practice, explicit examples of negative manifestations are described; −1: the construct had a negative influence in the practice, general statements of negative manifestations are made; 0: the construct had neutral influence in the practice; +1: the construct had a positive influence in the practice, general statements of positive manifestations are made; +2: the construct had a positive influence in the practice, explicit examples of positive manifestations are described.28 Using this scoring system, the overall construct scores could range from a low of −80 to a high of +80 based on the number of constructs, demonstrating the key barriers and facilitators to uptake and sustainability of the SC4C model. This method of quantifying implementation effectiveness will be supplemented with an inductive analysis of qualitative data to ensure openness to emerging themes not readily captured by the CFIR and Proctor et al’s outcome measures.
Ethics and dissemination
Research ethics approval
This study is approved by the human research ethics committees of: The Royal Children’s Hospital (HREC 65955) and The Sydney Children’s Hospital Network (STE03927), New South Wales, Australia.
Consent
Once general practices clinics have been recruited and the trial is underway, participants (GPs, study paediatricians, practice managers, administrative staff practices, families who have participated in co-consultations and project team members) will be contacted by the lead researcher. Participants will be contacted via telephone, email or met with in person and invited to participate in a semistructured interview or focus group at a preferred time and location (including online). Prior to conducting an interview or focus group, the researcher will describe the study to participants and the reasons for conducting an interview or focus group, providing a participant information sheet and a consent form. The researcher will invite and respond to any questions or concerns from participants and invite participants to sign a consent form. Prior to commencing the interview, the researcher will inform participants that they are able to stop the interview at any time and revoke their consent to participate during or after the interview. In this event, interview recordings and transcripts will be removed from the study and destroyed. The withdrawal of parent, caregiver or child participants at any point of the study will not affect their care in any way. When contacting parents or caregivers to schedule an interview, the researcher will describe the reasons for conducting a follow-up interview and will also invite their child to be present during the interview. Prior to conducting the interview with parents or caregivers, and their children, the researcher will describe the study, provide a participant information sheet for parents or caregivers, a child friendly version of the participant information sheet tailored to their child’s age or skills in comprehension, and a consent form. For the purposes of our consent process with children at varying ages, consent will be determined via a discussion with the parents or caregivers and the child to determine whether they are both comfortable for the child to be present, whether the parent and the researcher feel the child understands why they are being asked questions about their care, and continually checking that the child or young person wishes to continue participating during the interview. The researcher will invite and respond to any questions or concerns for parents or caregivers and their children and invite parents or caregivers and their children to sign a consent form. Prior to commencing the interview, the researcher will inform participants that they are able to stop the interview at any time and revoke their consent to participate during or after the interview. In this event, interview recordings and transcripts will be removed from the study and destroyed.
Confidentiality
Participant confidentiality is strictly held in trust by the investigators, research staff and the sponsoring institutions and their agents, and is extended to cover clinical information relating to participants. The study protocol, documentation, data and all other information generated is held in strict confidence and in password protected electronic files. No information concerning the study, or the data is released to any unauthorised third party, without prior written approval of the sponsoring institutions. Investigators and authorised representatives of the sponsoring institutions have access to the final dataset via permissions maintained by the data managers.
Dissemination
Principal investigator HH holds the primary responsibility for publication of the results of the study in accordance with the study publication and dissemination plan. The findings from this trial will be reported according to the Consolidated Standards of Reporting Trials statement guidelines.29
Ethics statements
Patient consent for publication
Acknowledgments
Strengthening Care for Children is a research collaboration between the Health Services group, Murdoch Children’s Research Institute; The Royal Children’s Hospital (Melbourne, Victoria); Population Health, The University of New South Wales; Sydney Children’s Hospitals Network; the University of Melbourne; North Western Primary Health Network and Central and Eastern Sydney Primary Health Network, Agency for Clinical Innovation New South Wales Health and the Sydney Partnership for Health, Education, Research & Enterprise. We thank all the general practice clinics, GPs, paediatricians, practice staff and families who participated in the trial.
References
Footnotes
Twitter @YvonneZurynski
Collaborators N/A.
Contributors The original implementation evaluation design was conceived by HH, RL YZ, PDH, SW and MH. MH is conducting the data collection and analyses. JB, RP, JL, SG, SK, TMM and KW have contributed to the data processes of the study and reviewed the paper. HH and RL are the overall guarantors.
Funding The study is funded through a 4-year National Health and Medical Research Council (NHMRC) Partnership Grant (APP1179176). This includes direct funding from the NHMRC, as well as cash and in-kind support from the following Partner Organisations: RCH, SCHN, NWMPHN, CESPHN, ACI NSW Health. The NHMRC has no direct role in study design; data collection, analysis, and interpretation; or writing of final reports, presentations, or publications. Murdoch Children’s Research Institute research is supported by the Victorian Government’s Operational Infrastructure Support Program. HH is supported by an NHMRC Practitioner Fellowship (1136222). SW is supported by an NHMRC Fellowship (1158954). Representatives from each Partner Organisation will form the Advisory Committee for the project; and will therefore have a role in the study and may influence the activities above.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.