Process evaluation objectives

FPE is a complex intervention that will be implemented into varied and multifaceted healthcare organisations. Following the recommendations of the Medical Research Council29 regarding complex interventions, our process evaluation aims to accomplish the following objectives (see figure 1 for key components of the process evaluation):

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

FPE process evaluation objectives: context, implementation and mechanisms of impact. FPE, Family Planning Elevated; FPE CAP, FPE Contraceptive Access Programme.

1. Understand both the state and local context in which FPE operates, and how that context impacts the intervention (as well as how the intervention impacts the context).

2. Document FPE as it is implemented. Explore how the actual implementation of FPE differs from the planned implementation, and identify where changes were an intentional adjustment to better meet programme needs (innovation) and where they were ‘unintentional drift’35 from the plan.

3. Identify the programme’s mechanisms of impact (how did clinicians and clients interact with FPE’s programming, were clients satisfied with care at FPE CAP clinics, what were the facilitators and barriers to implementation).

For the purpose of this process evaluation, we will consider policy-makers, FPE CAP clinic staff and the clients they serve to be recipients of the intervention. Our intervention is aimed at improving contraceptive access to the public; however, our programmatic approach to this improvement is mainly conducted through health systems, with clinics and providers as proximal recipients of FPE activities, while clients are distal beneficiaries of the programme. As such, we consider clinics, providers and policy-makers as recipients of the intervention, rather than intermediaries. FPE’s logic model (see figure 2) shows the theoretical path towards FPE’s ultimate goal of improving contraceptive access throughout the state. The process evaluation will document FPE’s adherence to the activities detailed in the logic model (implementation) as well as test the assumptions that those activities will lead to the documented outputs and outcomes (mechanisms of impact).

• Download figure
• Open in new tab
• Download powerpoint

Figure 2

Family Planning Elevated’s (FPE) logic model. FPE CAP, FPE Contraceptive Access Programme.

Design

FPE’s evaluation team consists of the director of evaluation who will oversee both the process and outcome evaluations; a data analyst researcher who will oversee the data analysis for the quantitative portion of the process evaluation and the outcome evaluation; and a process evaluator who will conduct and analyse the qualitative portions of the process evaluation, as well as integrate the quantitative portions for final analysis of the process evaluation. The implementation team is composed of a programme director, a project facilitator, an FPE CAP programme manager, a clinical training specialist and a medical director. FPE also has a research and policy advisor who is part of both the implementation and evaluation teams.

While the evaluation team is distinct from the implementation team, they will not be passive observers in the implementation process. The evaluation and implementation teams have opted to create feedback loops between the evaluation and implementation teams which will allow findings from the process evaluation to inform the ongoing implementation. The evaluation team will meet regularly with the implementation team. While the information will typically flow from the implementation team to the evaluation team, information regarding process indicators and intermediate clinic data will also be shared in the opposite direction.

Before beginning the process evaluation, FPE implementers and evaluators developed a foundational logic model to represent the projected activities (figure 2). This logic model charts how FPE aims to improve access to family planning by targeting efforts at each of the levels described in the Voluntary, Human Rights-based Approach to Family Planning Framework.8 At project midline, and again at end line, the team will revisit the logic model and make any required changes to represent how the project’s evolution. This will allow the team to visualise how implementation shifted, as well as present the most accurate logic model at project end.

Data collection and analysis

We will collect and analyse both qualitative and quantitative data for this process evaluation. Figure 1 diagrams the main components of the process evaluation, and how they fit into the larger programme evaluation. Table 1 details how our team will collect data on each of the primary components of the process evaluation.

Table 1 details the types of data that will be collected for each of the process evaluation objectives. Table 2 describes in detail how we will analyse each of the types of data.

Objective 1: context

Understand both the state and local context in which FPE operates, and how the context influences the intervention (as well as how the intervention influences the context).

FPE plans to support a wide variety of clinics including those in both rural and urban areas, clinics who primarily serve undocumented individuals, and clinics run by county health departments. To ensure our process evaluation captures each community’s unique context, the process evaluator will attend regular meetings of Utah’s RJ CAB. RJ CAB was formed in 2019 and its members are reproductive justice advocates and individuals from historically underrepresented communities. FPE selected the chairs and board members from a pool of interested applicants. Existing RJ CAB chairs will select board members for subsequent years of participation. RJ CAB members will provide community-level context data and highlight the needs of historically underserved groups. The process evaluator will take fieldnotes during the regular RJ CAB Meetings and conduct focus groups with the board members at project baseline, midline and end line.

To monitor the state-level sexual and reproductive health (SRH) context, our process evaluator will establish a close working relationship with FPE’s Research and Policy Advisor. The Research and Policy Advisor will meet frequently with Utah’s policy-makers and community advocates to consult and provide data for SRH policies that support reproductive autonomy, access to contraception, and education about the range of contraceptive methods. Additionally, all reports, white papers, internal analysis and notes from stakeholder meetings will be shared with the process evaluator. These policy documents will be stored in a secure cloud-based platform, and indexed with the date they were created, to create a timeline of changes in the political landscape.

The process evaluator will send a short list of questions to FPE stakeholders and partners identified by the FPE implementation staff. These questions will ask stakeholders and partners to identify barriers and potential solutions to SRH in Utah and their communities, as well as their opinions on FPE’s role. These questions will be sent at baseline, midline and end line of the intervention and responses will be indexed.

FPE will look to the National Family Planning and Reproductive Health Association (NFPRHA) and the Society of Family Planning (SFP) as guides to understanding the national context of SRH. NFPRHA works to improve the quality of family planning services throughout the country and supports administrators and providers to deliver quality care and advocate for improved healthcare policy. The SFP funds and disseminates high-quality research around best practices in contraceptive and abortion care.

The qualitative date for the context objective will include meeting notes, interview transcripts and email responses which will be uploaded into qualitative analysis software and coded deductively, according to predetermined process evaluation themes such as contextual barriers and facilitators to implementing FPE, how FPE research influenced policies around SRH, how other programmes working towards similar goals affect and are affected by FPE, and trends in the political landscape surrounding SRH. The results of the qualitative analysis will be combined with the literature reviews, policy briefs and resolution documents to create a comprehensive picture of the SRH contexts. Quantitatively, FPE will analyse findings from Utah’s Behavioral Risk Factor Surveillance System (BRFSS). In 2018, the Family Planning Division at the University of Utah submitted four family planning questions for inclusion in BRFSS. Those questions first appeared in the 2019 survey, and we hope to include them through 2024. Responses to the BRFSS questions will be analysed by FPE’s data analyst and included in the process evaluation report to understand the larger context in which FPE is implemented. For additional information on FPE’s use of BRFSS data, and information about the outcome evaluation more broadly, see FPE’s outcome evaluation protocol (Rebecca G. Simmons et al. Evaluating a longitudinal cohort of clinics engaging in a contraceptive access initiative in Utah: The Family Planning Elevated Contraceptive Access study protocol, under review).

Objective 2: implementation

Document how FPE is implemented. Explore how the actual implementation of FPE differed from the planned implementation.

The logic model (figure 2) outlines FPE’s primary activities. The activities are designed to address contraceptive access at the individual, community, clinic and policy level. Because of the complex nature of this intervention, and the need to tailor decisions to the individual needs of each clinic and community, the actual implementation will inevitably differ from the planned implementation. The goal of the implementation objective of the process evaluation is to understand what elements of the intervention are delivered (and to whom and how often), where and why changes are made to the implementation plan, and how the plan was adapted to fit the changing needs of the project.

To capture the programme implementation as it unfolds, we will collect both qualitative and quantitative data. The qualitative data on the programme implementation will include minutes from weekly FPE staff meetings, transcripts of interviews with FPE staff, summaries of monthly activities on FPE team calendar, monthly changes to project’s log frame, and narrative summaries from training reports. Quantitative data on the programme implementation will include monthly data reports from FPE CAP clinics showing the number of contraceptive services provided, quantitative responses to the applications submitted by FPE CAP members, progress indicators from the FPE CAP quarterly calls, quantitative aspects of training reports and media analytics reports.

The process evaluator will analyse the qualitative implementation on an ongoing basis. The fieldnotes from weekly team meetings will be coded during or immediately following the meetings, with any clarifying questions asked to the team members. The meeting notes will be organised into the codes derived from Finley et al’s article on periodic reflections as a tool for process evaluations.44 Monthly interviews with a member of the FPE staff will also follow the template of these periodic reflections. The clustered codes from the field notes and FPE staff interview for each month will be combined into a monthly process evaluation report along with the summaries of the month’s implementation activities and the revised logframe for that month. Portions of the implementation process will also be coded and analysed using constructs from the Consolidated Framework for Implementation Research (CFIR).45 The CFIR constructs will be modified to fit FPE’s programme (see the mechanisms of impact section for detail on analysis using CFIR).

The monthly reports will also incorporate the quantitative data collected on implementation. Where the qualitative data best answers the questions regarding fidelity and adaptation, the quantitative data will illuminate the dose and reach of FPE’s activities. The question of dose will be answered by information from the FPE CAP quarterly calls (questions such as: how many of your staff have been trained to insert IUDs or implants? Or, have you recently been out of stock of any methods?) which will be compared with the responses on the initial FPE CAP applications to track changes in those process indicators over item. The media analytics reports will also help us track how often and where the media campaigns have run. To measure how many people our intervention reaches we will track the number of individuals receiving services at FPE CAP clinics each month (from each’s clinic’s monthly data report), the number of views/interactions with the media campaign each month (from media analytics report), and the number of individuals who attended any FPE CAP trainings for that month (from training reports).

Objective 3: mechanisms of impact

Identify the programme’s mechanisms of impact (how did clinicians and client interact with FPE’s programming, were clients satisfied with care at FPE CAP clinics, what were the facilitators and barriers to implementation).

The goal of objective three is to understand what happens when FPE is implemented, to learn how clinicians and clients responded to the programme. To answer this question, the process evaluation will focus on the perceptions of those who received the intervention (FPE CAP staff, clients served at FPE CAP clinics, individuals who interacted with the media campaign and policy-makers) rather than the perspectives of those who implemented the intervention (which was the focus of objective two). Quantitatively we will collect responses from the client exit surveys, to measure how clients who visit FPE CAP clinics feel about the care they receive. These exit surveys will be administered for month-long periods several times during the intervention. The surveys will be offered, by clinic staff, to all clients who receive contraceptive care at FPE CAP clinics. We will also compile data from each clinic’s monthly data reports, tracking the number of patients served at each clinic.

Qualitatively, we will collect transcripts from interviews during implementation (individual interviews and focus groups) with FPE CAP clinic staff, as well as conduct exit interviews with FPE CAP members at the end of their clinic’s enrollment in the programme. All FPE CAP clinical and administrative staff will be invited to attend focus group interviews, including care providers (physicians, nurse practitioners and physician assistants), nurses, medical assistants, on-site pharmacists, clinic administration and management and front desk staff. While all will be invited, attendance is not mandatory, thus introducing the possibility of self-selection bias. Our documentation of the focus groups will include records of both who was present, and also who was missing, from each group to aid us in assessing the degree of potential bias. Additionally, we will use the same interviewer (JB) throughout the study, to increase trust from our clinics and thus, attempt to reduce social desirability bias throughout our interviews. We anticipate that the extent and impact of biases will be thoroughly assessed and addressed in our process evaluation outcome paper.

The transcripts from baseline and year 1 interviews with each clinic, as well as interviews with FPE staff members, will be coded using modified CFIR constructs.45 See online supplemental appendix A for a list of the CFIR constructs revised for FPE’s process evaluation and how those constructs align with the process evaluation. Once coded, the constructs from each clinic will be analysed using the process detailed by the CFIR website (https://cfirguide.org/evaluation-design/qualitative-data/), allowing us to view barriers and facilitators to implementing FPE across each implementation site.

We will analyse the quantitative data to determine trends in the number of contraceptive visits at FPE CAP clinics, as well as the experiences of contraceptive clients. By comparing these findings to the patterns regarding implementation, we will be able to identify which components of the intervention led to the greatest change, and which components did not create the expected impact. Additionally, by linking what was implemented with how the participants interacted and reacted, we can understand the barriers and facilitators to implementation and answer questions such as: why did clinic A observe an increase of patients after FPE was implemented, but the number at clinic B actually decreased after implementation?

Patient and public involvement

Family planning care is an area of healthcare that is particularly personal and incredibly individualised for each patient. The decision to conduct a process evaluation, and the methods we selected to conduct that evaluation, was largely motivated by a desire to understand the experience of patients receiving care in FPE CAP clinics. While patients were not involved in the design of the process evaluation, nor in the recruitment, we will involve patients as we collect and disseminate our results. Specifically, we will share results of the process evaluation with members of Utah’s RJ CAB. The board members will provide feedback on the results, help contextualise them from the viewpoint of some of Utah’s marginalised communities and help direct dissemination activities.