You are here

  • Home
  • Archive
  • Volume 12, Issue 1
  • Determining feasibility of incorporating consumer engagement into implementation activities: study protocol of a hybrid effectiveness-implementation type II pilot

Article Text

Article menu

Download PDFPDF

Mental health
Protocol
Determining feasibility of incorporating consumer engagement into implementation activities: study protocol of a hybrid effectiveness-implementation type II pilot
  1. Eva N Woodward1,2,
  2. Cathleen Willging3,
  3. Sara J Landes2,4,5,
  4. Leslie R M Hausmann6,7,
  5. Karen L Drummond1,2,
  6. Songthip Ounpraseuth8,
  7. Irenia A Ball1,
  8. JoAnn E Kirchner2,4
  1. 1VA Center for Mental Healthcare and Outcomes Research, Central Arkansas Veterans Healthcare System, North Little Rock, Arkansas, USA
  2. 2Department of Psychiatry, University of Arkansas for Medical Sciences, Little Rock, Arkansas, USA
  3. 3Behavioral Health Research Center of the Southwest, Pacific Institute for Research and Evaluation, Albuquerque, New Mexico, USA
  4. 4Behavioral Health Quality Enhancement Research Initiative (QUERI), Central Arkansas Veterans Healthcare System, North Little Rock, Arkansas, USA
  5. 5South Central Mental Illness Research Education and Clinical Center (MIRECC), Central Arkansas Veterans Healthcare System, North Little Rock, Arkansas, USA
  6. 6Department of Medicine, VA Pittsburgh Healthcare System, Pittsburgh, Pennsylvania, USA
  7. 7School of Medicine, University of Pittsburgh, Pittsburgh, Pennsylvania, USA
  8. 8Department of Biostatistics, University of Arkansas for Medical Sciences, Little Rock, Arkansas, USA
  1. Correspondence to Dr Eva N Woodward; eva.woodward{at}va.gov

Abstract

Introduction Implementation researchers could draw from participatory research to engage patients (consumers of healthcare) in implementation processes and possibly reduce healthcare disparities. There is a little consumer involvement in healthcare implementation, partially because no formal guidance exists. We will create and pilot a toolkit of methods to engage consumers from the US’ Veterans Health Administration (VHA) in selecting and tailoring implementation strategies. This toolkit, Consumer Voice, will provide guidance on what, when, where, how and why an implementer might engage consumers in implementing treatments. We will pilot the toolkit by implementing Safety Planning Intervention for suicide prevention with rural veterans, a population with suicide disparities. Safety Planning Intervention is effective for reducing suicidal behaviours.

Methods and analysis In Aim 1, we will use participatory approaches and user-centred design to develop Consumer Voice and its methods. In Aim 2, we will pilot Consumer Voice by implementing the Safety Planning Intervention in two clinics serving rural VHA patients. One site will receive a current implementation strategy (Implementation Facilitation) only; the second will receive Implementation Facilitation plus Consumer Voice. We will use mixed methods to assess feasibility and acceptability of Consumer Voice. We will compare sites on preliminary implementation (reach, adoption, fidelity) and clinical outcomes (depression severity, suicidal ideation, suicidal behaviour). In Aim 3, we will evaluate Aim 2 outcomes at 20 months to assess sustained impact. We will gather qualitative data on sustainability of the Safety Planning Intervention.

Ethics and dissemination These studies are overseen by the Institutional Review Board at the Central Arkansas Veterans Healthcare System. We plan to use traditional academic modalities of dissemination (eg, conferences, publications). We plan to disseminate findings through meetings with other trainers in implementation practice so they may adopt Consumer Voice. We plan to share results with local community boards.

  • mental health
  • public health
  • quality in health care
http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

https://doi.org/10.1136/bmjopen-2021-050107

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Strengths and limitations of this study

    • Rigorous, iterative process based on user-centred design.

    • Includes consumers/patients in several steps of toolkit development.

    • Consumers/patients are involved on research team who makes decisions about Consumer Voice.

    • Researchers will have difficulty detecting patient-level differences in outcomes due to relative infrequency of suicide.

    • Provides a toolkit on how to engage consumers/patients in implementation practice to generalise to other settings.

    Introduction

    Healthcare disparities are significant differences in receipt of, access to, quality of, or outcomes of healthcare between marginalised groups and reference groups.1 Healthcare disparities persist in the USA and in the Veterans Health Administration (VHA) for several marginalised groups who have experienced societal oppression.2 3 One reason disparities persist is that clinical interventions target patient factors only—patients’ individual attitudes, behaviours—and although these are necessary targets to reduce disparities, they are not sufficient.4 We must also intervene on broader structures—for example, cultures of change, policies, organisational climate. Implementation scientists can address these broad, organisational factors contributing to disparities by using implementation strategies.

    Implementation strategies are implementation interventions to address known barriers to uptake of a clinical intervention.5 Implementation strategies are commonly targeted at providers, clinics, hospitals or systems, such as provider training, performance data feedback or securing new funding streams.6 For example, to reduce racial disparities in guideline-concordant cardiovascular disease care, one possible implementation strategy is to plan for, act on and re-evaluate quality improvement efforts among patients by race. Typically, implementers—researchers, quality improvement personnel, facilitators—select and tailor implementation strategies. Tailoring a strategy involves refinements or tweaks so that it fits better with local context and more precisely targets implementation barriers.7 Although existing implementation strategies have improved care for the general population,5 they may not be sufficient to improve care for marginalised populations.8 One potential solution to reduce healthcare disparities is to engage marginalised patients (referred to as consumers) in selecting and tailoring implementation strategies to better fit their needs.

    Participatory approach to engage consumers in implementation

    Participatory research is an approach in which consumers are actively engaged in the research process. Consumers might be informants, discussants, or partners in research with varying degrees of decision-making power and trust with healthcare or academic staff. Among marginalised populations, participatory research has enhanced retention in health disparities research,9 improved fidelity to clinical care,10 better health outcomes,9 and reduced inequities in access to, satisfaction of, and quality of care.11 In fact, the Agency for Healthcare Research and Quality recommends participatory research as a ‘gold standard’ to reduce disparities.12

    Although implementers often engage healthcare staff, using participatory approaches to involve consumers throughout implementation does not often occur.13 In the most robust example of using a participatory approach to enhance implementation, quality improvement that included community members was more effective than technical assistance without a participatory approach for uptake of a depression intervention across diverse US healthcare settings.14

    A participatory approach to implementation shares principles with participatory research, such as work funded by Patient-Centered Outcomes Research Institute in which consumers inform research outcomes most important to them. Yet, engaging consumers in implementation is distinguishable from participatory research by its focus beyond intervention and outcomes to broader factors necessary to get patients, organisations and providers to be willing or able to implement the intervention. The benefit of this study is that we will engage consumers to focus on strategies to increase uptake of an intervention rather than more typical consumer engagement to determine components of an intervention or outcomes.

    Gap in implementation and purpose of the current study

    Implementation scientists need to reduce disparities in uptake and reach of interventions.15 Although consumer engagement in implementation has nascent evidence of improving healthcare among marginalised populations,16 methods for involving consumers in selecting and tailoring implementation strategies are not well synthesised or documented. Thus, participatory approaches to implementation are used less frequently than ideal, not well operationalised or reported, and not well studied as potential mechanisms for decreasing healthcare disparities. A Cochrane review of consumer engagement in healthcare called for greater specificity on how consumers are engaged and what resources are needed so processes and positive effects can be replicated.17

    The purpose of this study is to augment a conventional method of selecting and tailoring implementation strategies (Implementation Facilitation) with consumer engagement and assess feasibility, acceptability and preliminary impact of consumer engagement on implementation and clinical outcomes. We will systematically develop a toolkit, Consumer Voice, to guide processes for engaging consumers in selecting and tailoring implementation strategies. We will pilot it by implementing the Safety Planning Intervention to prevent suicide among rural VHA patients.

    Conventional strategy: Implementation Facilitation

    To evaluate Consumer Voice, we will pair it with a conventional implementation strategy, Implementation Facilitation.18–20 Implementation Facilitation is defined as ‘a process of interactive problem solving and support that occurs in the context of a recognized need for improvement and a supportive interpersonal relationship’.21 22 Implementation Facilitation involves methods to select and tailor implementation strategies and does not usually involve interfacing with consumers.

    Intervention to be implemented: Safety Planning Intervention among rural VHA patients

    Approximately six VHA patients die by suicide daily.23 Compared with urban dwelling US veterans, rural dwelling US veterans are more likely to consider suicide and less likely to access mental healthcare.24 Safety Planning Intervention is a suicide prevention intervention, effective at reducing suicidal ideation and suicidal behaviours, inside and outside VHA.25 26 Safety Planning Intervention is a one-session, clinical intervention for patients with suicidal thoughts or behaviours. Patients and providers collaboratively create a safety plan with prompts populated in the VHA electronic health record and a copy given to patients.25 A complete Safety Planning Intervention safety plan consists of six types of coping skills that patients can use when suicidal thoughts arise.

    Specific aims

    1. Using a participatory approach, we will develop a toolkit (Consumer Voice) containing methods to engage consumers in selecting and tailoring implementation strategies.

    2. Using a two-arm design, we will pilot feasibility and acceptability of Consumer Voice and its preliminary impact on implementation and clinical outcomes by implementing Safety Planning Intervention.

    3. We will compare Implementation Facilitation to Implementation Facilitation plus Consumer Voice on sustainability of Safety Planning Intervention and assess factors that enhance or hinder sustainability of Safety Planning Intervention.

    Methods and analysis

    Patient and public involvement

    The development of this research question was informed by patient and public opinion through our VHA centre’s Veterans Research Council. The lead author met with them as a group, presented research ideas, integrated some of their feedback while maintaining decision-making power, and returned to the council a second time to refine ideas before submitting this research for external funding. We also incorporated patient and public involvement in the design of Aim 1, especially recruitment strategies and locales suited for patients, by consulting with three patient representatives working in community organisations serving VHA patients in our US state.

    Although this is a protocol, we began early components of the study and added two community member consultants (Veterans) on our research team that makes decisions about the form and function of Consumer Voice. For dissemination, we plan to share results with our local community boards, such as the community service organisations serving VHA patients and our local Veterans Research Council. We also plan to create an infographic of key results and distribute on social media from our research centre.

    Theoretical approach

    We will use the Health Equity Implementation Framework27 (see figure 1) to inform this research. This framework posits domains that predict successful implementation and reductions in implementation disparities. Within each domain are several determinants or specific factors that are measurable and, together in constellation with other determinants, clarify barriers, facilitators, moderators or mediators to equitable and successful implementation. The framework also proposes a process—Implementation Facilitation—by which change in each domain would occur.28 29

    Figure 1
    Figure 1

    Health Equity Implementation Framework.

    Some examples of domains in the Health Equity Implementation Framework are described below. Innovation refers to the treatment, intervention, practice, or new ‘thing’ to be implemented (ie, the Safety Planning Intervention), adopted by providers and staff, and delivered to patients.30 Recipients are individuals who influence implementation and those who are affected by its outcomes (ie, rural VHA patients, VHA staff and providers), at the individual and collective team levels.29 Cultural factors of recipients are unique characteristics to a particular group in the implementation effort (eg, patients, staff, providers) based on their lived experience. Some examples are implicit bias, socioeconomic status, stress related to discrimination, health literacy, health beliefs, or trust in the healthcare staff or patient group.31 32 Economies include how innovations are marketed and acquired (ie, government controlled healthcare at low cost) and other market forces that change demand for the Safety Planning Intervention (eg, it becomes offered at local urgent care clinics outside of VHA). Physical structures are where people have to visit to get healthcare and what environmental elements people may be exposed to that exacerbate or minimise the health problem.33 One factor in rural areas can be lack of confidentiality for suicide screening in a small town with few providers where many residents know each other.

    We will use the Health Equity Implementation Framework to: (1) identify barriers/facilitators to using Consumer Voice (Aim 1), (2) identify barriers/facilitators for Safety Planning Intervention implementation among rural VHA patients that will guide Implementation Facilitation and Consumer Voice at local clinics (Aim 2) and its sustainability (Aim 3), and (3) interpret results from Aims 1, 2 and 3.

    Setting

    To reach a subset of rural VHA patients at risk for suicide, we will target rural VHA community-based outpatient clinics in Arkansas that house primary care and mental healthcare. One reason to implement suicide prevention in these primary care settings is because many veteran suicide deaths occur among those not engaged in mental healthcare who do seek primary care.23 Suicide prevention in primary care will reach more high-risk, rural veterans than in mental healthcare alone.

    Study design, processes and planned analyses by specific aim

    Aim 1: using a participatory approach, develop a toolkit (Consumer Voice) containing methods to engage consumers in selecting and tailoring implementation strategies

    We will build a toolkit for use in engaging consumers in selecting and tailoring implementation strategies. Consumer Voice will be a multimedia manual showcasing who, what, when, where, how and why implementers should engage veterans (as consumers of healthcare) in implementing new or improved healthcare services. Our team will build the first draft of Consumer Voice based on a complete environmental scan of existing examples of consumer engagement in implementation activities.34

    User-centered design

    We will build Consumer Voice to expand Implementation Facilitation by using a QUALITATIVE→quantitative→QUALITATIVE structure through three sequential steps in which qualitative data will be given more weight (figure 2).35 Drawing from user-centred design,36 we will use an iterative approach to engage end-users (implementers) and other stakeholders in initial prototype testing and then mini-pilot tests of Consumer Voice. Our three sequential steps are: (1) conduct individual qualitative interviews and cocreation sessions with diverse stakeholders, (2) ask implementers to pilot Consumer Voice briefly in their own work and reconvene through a Delphi process to achieve consensus on components, and (3) reconvene diverse stakeholders again in a nominal group technique process to clarify the most feasible and important components for the final prototype of Consumer Voice. Within each step, we will use a variety of user-centred design methods such as interviews about user perspectives, applying process maps to visualise system-level implementation activities needed for Consumer Voice, cocreation sessions in which stakeholders develop some aspects of Consumer Voice alongside our team, and experience sampling (ie, implementers briefly pilot using Consumer Voice in their work).36

    Figure 2
    Figure 2

    Steps to develop Consumer Voice toolkit.

    Step 1: stakeholder qualitative interviews

    We will conduct interviews with key stakeholders (see table 1) to refine operational definitions of consumer engagement in implementation methods, preferences or needs, potential barriers to and facilitators of using these methods, and technical resources needed for Consumer Voice. We expect to achieve saturation between 12 and 20 total interviews.37

    View this table:
    Table 1

    Planned stakeholder interview details

    We will reach out to existing contacts in each stakeholder group for potential participation to recruit stakeholders in a respondent-driven, non-probabilistic approach. These contacts will serve as referral agents who suggest other stakeholders in any group for recruitment. We have built connections and partnerships with two veteran community groups. Stakeholders will be offered financial payment as incentive.

    Interview guide

    The interview guide will be structured to assess preferred types of engagement and technical resources; see sample questions in table 2. Interviews will be audio recorded, approximately 45 min long, and interviewers will take notes during the interview.

    View this table:
    Table 2

    Sample questions for stakeholder interviews in developing Consumer Voice

    Qualitative analysis

    We will use a Rapid Assessment Process to analyse qualitative data from stakeholder interviews. The time required for this approach can range from 4 days to 6 weeks.38 This method is useful for an implementation study in which there is a time-sensitive demand for creation and modification of an implementation product (Consumer Voice), yet need for rigour in the analysis.39 The analysis will blend inductive and deductive approaches, using directed content analysis40 and allow a framework to guide analysis deductively while leaving room for emerging information. We will use the Health Equity Implementation Framework to create summary templates to categorise barriers and facilitators. We will present results to veteran community groups focused on suicide prevention to give feedback to inform the next iteration of Consumer Voice.

    Step 2: Delphi process with implementation experts

    We will ask implementers to use a beta version of Consumer Voice in their own work as an uncontrolled pilot. Then, using a modified Delphi process that will produce quantitative data from voting, we will generate consensus on Consumer Voice through rounds of discussion and voting with those implementers.41

    We will use respondent-driven sampling to identify up to 12 implementation experts by advertising on Twitter, and approaching professional implementation networks. We will ask these participants to reach out to one other potential participant through e-mail or social media.

    Experts will be engaged in 2–3, 60 min, virtual Delphi sessions using online polling and discussion to reach consensus through videoconferencing platforms, Microsoft PowerPoint and telephone calls. The two sessions will follow this cycle: (1) present the draft version of Consumer Voice, elicit discussion from participants based on experience, and vote on which components to include (70% agreement achieves consensus)42; and (2) present group results back to participants and elicit discussion to vote again on which components to include. Implementers will receive the final version of Consumer Voice and monetary payment as an incentive.

    We will also administer a one-time set of three questionnaires produced by Weiner et al,43 four questions each, assessing feasibility, acceptability and appropriateness of the beta version of Consumer Voice. Responses are on a Likert scale ranging from 1 (completely disagree) to 5 (completely agree). Example items include ‘Consumer Voice is appealing to me’ (acceptability), ‘Consumer Voice seems fitting’ (appropriateness) and ‘Consumer Voices seems doable’ (feasibility).

    Step 3: nominal group technique to finalise Consumer Voice

    Finally, we will use the nominal group technique with stakeholders to prioritise final components of Consumer Voice after step 2 (post-Delphi version) based on stakeholder rankings of importance and feasibility. The nominal group technique is a participatory research method in which exploratory questions about a topic are presented to small stakeholder groups to generate ideas, develop consensus and set priorities for guidelines, particularly for research areas that are underdeveloped.44

    We will host 2–3 2-hour meetings with subsets from the diverse stakeholder groups in table 1. Each stakeholder will attend only one meeting. We will offer very small groups (eg, 2–4 individuals), varying locations that can be private and confidential, and even individual interviews should a stakeholder prefer not to discuss these topics with others. Our sampling is consistent with recommendations for the nominal group technique: emphasis is on involving people from different roles/locations to ensure heterogeneity of viewpoints.45

    Participants will be provided an explanation of nominal group technique, key terms used in discussion, and a draft of the Consumer Voice toolkit. Participants will also be provided with preprinted forms that specify exploratory response questions. The exploratory questions will be honed through initial individual stakeholder interviews; they will likely resemble1: ‘What do you think are the most important and feasible ways to engage VHA consumers in implementing a healthcare intervention?’2; ‘What are other methods or ways to engage VHA consumers in implementation?’. Participants will be able to select, adapt and suggest new methods in their lists. Participants will be provided 15 min to brainstorm in silence followed by an oral round of listing ideas on flipcharts, serial discussion of each idea, group ranking of priorities, group discussion of rankings and re-ranking until consensus is reached.

    Analysis to finalise Consumer Voice

    The analytic plan is to use and connect data gathered after each of the three steps to form iterative prototypes of Consumer Voice.35 After each step, an analysis team (authors ENW, IAB, JEK, CW, veteran consultants) will meet to integrate data gathered from the prior step, using brainstorming and consensus, and decide how to integrate changes into the next Consumer Voice prototype. Data may take the form of suggested visual changes, stakeholder needs, suggested methods, activities or archival examples of consumer engagement in implementation. The analysis team will categorise the function of each consumer engagement method/activity on a continuum from least intensive to most intensive (eg, from informing consumers to partnering with them).46 One likely challenge we expect is for findings from stakeholders to diverge. The analysis team will work to resolve discrepancies during mixed-methods analysis between each step.47 In the final joint nominal group technique session, we will present remaining discrepancies to diverse stakeholders and elicit feedback on how to resolve, lending priority to different groups based on the function or form of the discrepancy (eg, clinical expert opinions will be given priority on components of clinical intervention delivery).

    Aim 2: using a two-arm design, we will pilot feasibility and acceptability of Consumer Voice and its preliminary impact on implementation and clinical outcomes by implementing Safety Planning Intervention

    We will use the Consumer Voice toolkit to conduct engagement meetings, events and interactions with rural veterans who have experienced suicidal thoughts or behaviour and their families in selecting and tailoring implementation strategies for Safety Planning Intervention. During and after these interactions, we will conduct a mixed-methods process evaluation of the Consumer Voice toolkit and process. We will conduct a pilot study using an effectiveness-implementation hybrid 2 design comparing Implementation Facilitation only with Implementation Facilitation plus Consumer Voice on implementation and clinical outcomes.48

    Sites

    Within our VHA regional system, one community-based outpatient clinic (referred to as ‘clinic’) will be the ‘standard care’ clinic at which Implementation Facilitation alone is used; the second clinic will be the ‘implementation site’ at which Implementation Facilitation plus Consumer Voice is used (table 3). We randomly assigned each site’s implementation condition. The sites are matched on clinic size and percentage of veterans defined as rural. One possible challenge is that sites might drop out of the study. If a site is unable to participate, mental health leaders at our VA facility identified alternate sites for this study.

    View this table:
    Table 3

    Timeline of Safety Planning Intervention implementation in 4-month phases with key data collection milestones

    Timeline for Safety Planning Intervention implementation and data collection

    Implementation will occur in four phases, each lasting 4 months: planning, pre-implementation, implementation and sustainability. Although time periods are short compared with larger trials, they will allow sufficient time to determine feasibility and acceptability of Consumer Voice (table 3).

    Implementation strategies across implementation phases

    There will be one facilitator who will use conventional Implementation Facilitation at the standard care clinic, and Implementation Facilitation plus Consumer Voice at the implementation clinic. Implementation Facilitation and Consumer Voice will occur on the same timeline, although we anticipate there to be additional or different activities at the Implementation Clinic in which Consumer Voice is used in conjunction with Implementation Facilitation. The facilitator will track their weekly time and activities related to Implementation Facilitation using pre-established tracking logs49 and key Implementation Facilitation events that occur using a pre-established checklist from our preliminary work.50 The facilitator will use these logs to document whether clinics receive the same amount and type of activities of Implementation Facilitation.51

    One anticipated challenge is that Consumer Voice participants may drop out over the course designing for implementation. In preparation for this, we will track retention as an outcome for the process evaluation (see table 4). If drop out occurs, we will apply similar techniques as in original recruitment, identify new participants and spend 1–2 hours orienting them to consumer engagement and implementation, the development of Consumer Voice, and study progress to date.

    View this table:
    Table 4

    Feasibility pilot outcomes and measures

    Mixed-methods process evaluation of Consumer Voice

    Because the end-user of Consumer Voice will be implementers as they will use Consumer Voice methods to engage consumers, we need to assess feasibility and acceptability of using the Consumer Voice toolkit and methods with implementers as well as consumers. We will conduct a mixed-methods process evaluation of Consumer Voice.52 We will use a qualitative+QUANTITATIVE design; data will be collected simultaneously and importance will be given to quantitative measures.35 53

    Procedures

    The function of these mixed-methods data will be convergence, which involves integrating them to answer the same question: is Consumer Voice feasible and acceptable to all stakeholders?.35 To assess feasibility, we will administer brief surveys at consumer engagement events to all consumers and healthcare professionals and use logs for tracking data in real-time during these interactions. To assess acceptability, we will use (1) the same surveys and logs used for feasibility data collection to assess retention and physical safety, and (2) brief qualitative interviews with consumers and healthcare professionals to assess burden and satisfaction. We will also attempt to interview consumers who responded to initial recruitment but did not attend or dropped out about reasons for non-attendance. This strategy allows for greater external validity by ensuring broader variability in the data.54

    Measures

    We will assess feasibility outcomes suggested by Orsmond and Cohn55 as seen in table 4. Acceptability outcomes were designed based on recommendations from Proctor and colleagues as seen in table 5.56 We will administer again Weiner’s three questionnaires,43 four questions each, assessing feasibility, acceptability and appropriateness of Consumer Voice that was used in Aim 1.

    View this table:
    Table 5

    Acceptability pilot outcomes and measures

    Analysis

    To integrate data, we will merge information from quantitative and qualitative datasets.35 We will use descriptive statistics to analyse quantitative data. Qualitative data from surveys and interviews will be extracted into summary templates aligned with the Health Equity Implementation Framework.27 The coding team will analyse data using a blend of inductive and deductive approaches through the Rapid Assessment Process described in Aim 1.38 39 As one way to triangulate data to answer questions about acceptability and feasibility of Consumer Voice, some qualitative categories will be able to be quantified (eg, 0=not satisfied, 1=somewhat satisfied) and converged with quantitative data. Another way to triangulate data will be for the mixed-methods analytic team to meet together to present, review, discuss and integrate findings from quantitative and qualitative data.

    Assessing preliminary impact of Consumer Voice

    As part of the pilot, we will also assess implementation outcomes of reach, adoption and fidelity to Safety Planning Intervention and clinical outcomes of patient depression, suicidal ideation and suicidal behaviour. This pilot will not have enough statistical power to detect a conclusive effect of Consumer Voice on implementation or clinical outcomes. The pilot study will allow us to obtain SD estimates of clinical outcomes for sample size determination of future trials.

    Measures

    To evaluate preliminary implementation and clinical outcomes, we will use Reach, Effectiveness, Adoption, Implementation, Maintenance (RE-AIM)57 as a framework (RE-AIM). We will collect these data from both clinics during month 13 after the implementation phase. Reach is defined by Safety Planning Intervention being used with the targeted patient population (ie, rural veterans with suicidal ideation or behaviour). Effectiveness is conceptualised as whether veteran depression symptoms and suicidal ideation and behaviour is different because of exposure to Safety Planning Intervention. Adoption is conceptualised as Safety Planning Intervention uptake by providers in primary care and specialty mental healthcare roles at each clinic. Implementation is conceptualised as high fidelity to the Safety Planning Intervention. We will randomly select 30% of rural veterans exposed to Safety Planning Intervention from both clinics to assess implementation fidelity of Safety Planning Intervention. Using this sample, we will conduct chart reviews of the safety plans created in the medical record to assess the quantity of Safety Planning Intervention steps completed; a complete safety plan involves six steps. Table 6 lists planned outcomes and sources from which we will collect data to evaluate these outcomes.

    View this table:
    Table 6

    Measures for Consumer Voice pilot

    One possible challenge with the adoption measure is that there may be very low adoption overall, and thus, we might need to increase the percentage of chart reviews to identify differences in adoption between sites. One limit to the Implementation measure is that it is a basic fidelity assessment that does not capture quality of completion of SPI safety plans. It is possible that a fidelity measurement focused on quality will be needed, and if so, I will use rating tools created by the VHA Safety Planning Intervention training group.

    Patient sample

    To assess clinical effectiveness outcomes, we will analyse a sample of rural VHA patients within both clinics. We will include patients who screen positive on a suicidal ideation question at primary care appointments (ie, Patient Health Questionnaire-2+Item 9 (suicidal ideation)). Because data on patient clinical effectiveness will be extracted directly from VHA administrative data, patients sampled in each clinic will represent a convenience sample (vs a random sample), therefore; there is no clear sample size.

    Analysis

    To assess preliminary impact of Consumer Voice on implementation outcomes of reach, adoption and implementation fidelity, we will calculate descriptive statistics. We will not conduct effect sizes from this pilot study due to concerns about inflation of type I and II errors in small samples sizes.55 58 We will describe variance in the outcomes detailed in table 6, including CIs about each point estimate (eg, mean, SD).58 59

    To evaluate the hypothesis that Consumer Voice will improve patient clinical outcomes, we will conduct inferential statistics. We will conduct an analysis of covariance and compare differences in patient outcomes between clinics that receive standard Implementation Facilitation and Implementation Facilitation plus Consumer Voice at the month 13 time period, while controlling for baseline level of depression at each site during the 4-month planning period. Independent variables will be implementation assignment and time. The dependent variables will be depression symptoms, suicidal ideation and self-directed violence.

    Aim 3: Evaluate sustainability of Safety Planning Intervention

    One metric of the impact of Consumer Voice is how well Safety Planning Intervention is sustained in a clinic that used Implementation Facilitation only compared with a clinic that received Implementation Facilitation plus Consumer Voice.

    Design

    Therefore, we will use mixed-methods (QUANTITATIVE+qualitative)35 to compare the two clinics on: (1) repeated implementation and clinical outcomes at months 19–22 (observation period) and (2) barriers and facilitators to Safety Planning Intervention sustainment. Quantitative data collection to assess implementation and clinical outcomes will precede qualitative data collection by 1 month to document the fidelity of Safety Planning Intervention at a later timepoint. Qualitative data will be used to assess stakeholder perceptions of Safety Planning Intervention sustainability barriers and facilitators.

    Measures and analysis of quantitative data

    For maintenance (sustainability), we will measure outcomes guided by the RE-AIM framework for reach, effectiveness, adoption and implementation again in Month 19, as reviewed in table 6. For analysis, we will repeat detailed analysis described in Aim 2.

    Sampling and recruitment for qualitative interviews

    We will interview again a subset of stakeholders listed in table 1. The purpose of the interviews will be to assess barriers to and facilitators of Safety Planning Intervention sustainment at each clinic to document differences between the contexts of the standard care clinic (Implementation Facilitation only), and the implementation clinic (Implementation Facilitation plus Consumer Voice). Participants will be sampled purposively, by selecting those that were most informative during stakeholder interviews and the nominal group technique in Aim 1 and implementation in Aim 2, and those that presented ‘negative cases’ in Aim 2 (ie, preliminary results that did not fit with majority of information used to implement Safety Planning Intervention).54 The interview guide will be semi-structured, with questions aligned to the Health Equity Implementation Framework.

    Analysis of qualitative data

    We will use the blended inductive-deductive analysis40 through a Rapid Assessment Process described in Aim 1.39 Initially, the analysis will be deductive and focused on these specific questions: How has Safety Planning Intervention been sustained? Which implementation strategies contributed to its sustainment? How has consumer involvement affected Safety Planning Intervention sustainment? We anticipate these interviews may elucidate potential mechanisms of change that we would investigate in a subsequent, fully powered trial of Consumer Voice.

    We will also compare Sustainability Action Plans completed for each site in the sustainability phase, including any updates to the plans. Specifically, we will code for three criteria within each Sustainability Action Plan: (1) communication between consumers and clinic, (2) consumers involved in developing or reviewing the plan and (3) consumers being sampled for some metric of sustainability (eg, consumer satisfaction, use of Safety Planning Intervention). These criteria were informed by a consumer partnering subscale of a reliable, quantitative sustainability measure.60

    Ethics and dissemination

    Ethics

    A major innovation of this study is the integration of a participatory research approach with implementation science; this novel approach has potential for ethical pitfalls. Participatory research and implementation science come from distinct research traditions. Although there are shared goals, they are also distinctly different on their ethical approach.61

    Regarding motivation, both approaches want to improve society. Participatory research is geared more to create social change and build capacity among users; implementation science is geared more to apply knowledge to help users, although not explicitly to build capacity among them. Regarding social location, both approaches want knowledge users involved in the healthcare system. Participatory research is rooted in grassroots, user led action (equalising power differentials); while implementation science is rooted in decisions made by healthcare professionals. The main aim is not to equalise power between researchers and users, although this may occur. Both approaches propose users should be engaged in an ethical manner, although sometimes ethical is defined by consumers in participatory research, but by researchers in implementation science.

    As we conduct this work, we will have to recruit and retain engagement with multiple stakeholders and pay careful attention to inputs of consumers and processes used, to create an implementation strategy and toolkit that truly exemplifies strengths from both traditions. We will need to pay extra attention to work collaboratively, inclusively, and with respect for people living in rural communities, using suggested best practices by experienced community engaged researchers such as using a variety of participation strategies, allowing extra time for building trust, being a regular presence in the community, and including local customs in interventions or implementation.62

    Dissemination

    In this pilot, there is narrow focus on Safety Planning Intervention implementation and Consumer Voice will require adaptation to other evidence-based practices. Although we are collecting preliminary impact data about implementation and patient outcomes, we will be unable to draw strong conclusions about these research questions.

    We plan to use traditional academic modalities of dissemination, including conference presentations and journal publications. We also plan to disseminate findings through meetings with other trainers and teachers in implementation practice so they may adapt or adopt Consumer Voice to meet their needs. Although VHA has no publicly available data repositories, we will make data from our studies available on request.

    Ethics statements

    Patient consent for publication

    Acknowledgments

    Thank you to the Veterans Research Council at the VA Center for Mental Healthcare and Outcomes Research, along with Veteran Service Officers in the state of Arkansas, for their collective feedback on elements of this research design. ENW is a fellow with the Implementation Research Institute (IRI), at the George Warren Brown School of Social Work, Washington University in St. Louis; through an award from the National Institute of Mental Health (5R25MH08091607).

    References

    1. ↵
      1. Kilbourne AM,
      2. Switzer G,
      3. Hyman K, et al
      . Advancing health disparities research within the health care system: a conceptual framework. Am J Public Health 2006;96:2113–21.doi:10.2105/AJPH.2005.077628pmid:http://www.ncbi.nlm.nih.gov/pubmed/17077411
      OpenUrlCrossRefPubMedWeb of Science
    2. ↵
      1. Atkins D,
      2. Kilbourne A,
      3. Lipson L
      . Health equity research in the Veterans Health Administration: we've come far but aren't there yet. Am J Public Health 2014;104:S525–6.doi:10.2105/AJPH.2014.302216pmid:http://www.ncbi.nlm.nih.gov/pubmed/25100414
      OpenUrlPubMed
    3. ↵
      1. VA Office of Health Equity
      . National veteran health equity report—FY2013. Washington, DC: US Department of Veterans Affairs, 2016. http://www.va.gov/healthequity
    4. ↵
      1. Yancey A,
      2. Glenn BA,
      3. Ford CL
      . Dissemination and implementation research among racial/ethnic minority and other vulnerable populations. In: Dissemination and implementation research in health: translating science into practice. 2nd edn. New York, NY: Oxford University Press, 2018: 449–70.
    5. ↵
      1. Baker R,
      2. Camosso-Stefinovic J,
      3. Gillies C, et al
      . Tailored interventions to address determinants of practice. Cochrane Database Syst Rev 2015;308:CD005470. doi:10.1002/14651858.CD005470.pub3pmid:http://www.ncbi.nlm.nih.gov/pubmed/25923419
      OpenUrlPubMed
    6. ↵
      1. Waltz TJ,
      2. Powell BJ,
      3. Fernández ME, et al
      . Choosing implementation strategies to address contextual barriers: diversity in recommendations and future directions. Implement Sci 2019;14:42. doi:10.1186/s13012-019-0892-4pmid:http://www.ncbi.nlm.nih.gov/pubmed/31036028
      OpenUrlPubMed
    7. ↵
      1. Baumann AA,
      2. Cabassa LJ,
      3. Stirman SW
      . Adaptation in dissemination and implementation science. In: Dissemination and implementation research in health. 2nd edn. New York, NY: Oxford University Press, 2017: 285–300.
    8. ↵
      1. Lion KC,
      2. Raphael JL
      . Partnering health disparities research with quality improvement science in pediatrics. Pediatrics 2015;135:354–61.doi:10.1542/peds.2014-2982pmid:http://www.ncbi.nlm.nih.gov/pubmed/25560436
      OpenUrlAbstract/FREE Full Text
    9. ↵
      1. Tiwari T,
      2. Sharma T,
      3. Harper M, et al
      . Community based participatory research to reduce oral health disparities in American Indian children. J Fam Med 2015;2.pmid:http://www.ncbi.nlm.nih.gov/pubmed/26090520
      OpenUrlPubMed
    10. ↵
      1. Fettes D,
      2. Haine-Schlagel R
      . Mixed methods adaptation and pilot testing of a toolkit to enhance parent participation in home visitation programs. 32nd annual San Diego conference on child and family maltreatment. San Diego, CA, 2018.
    11. ↵
      1. Patzer RE,
      2. Paul S,
      3. Plantinga L, et al
      . A randomized trial to reduce disparities in referral for transplant evaluation. J Am Soc Nephrol 2017;28:935–42.doi:10.1681/ASN.2016030320pmid:http://www.ncbi.nlm.nih.gov/pubmed/27738125
      OpenUrlAbstract/FREE Full Text
    12. ↵
      1. Agency for Healthcare Research and Quality
      . AHRQ activities using community-based participatory research to address health care disparities [online]. Rockville, MD, 2014. http://www.ahrq.gov/research/findings/factsheets/minority/cbprbrief/index.html
    13. ↵
      1. Holt CL,
      2. Chambers DA
      . Opportunities and challenges in conducting community-engaged dissemination/implementation research. Transl Behav Med 2017;7:389–92.doi:10.1007/s13142-017-0520-2
      OpenUrl
    14. ↵
      1. Wells KB,
      2. Jones L,
      3. Chung B, et al
      . Community-partnered cluster-randomized comparative effectiveness trial of community engagement and planning or resources for services to address depression disparities. J Gen Intern Med 2013;28:1268–78.doi:10.1007/s11606-013-2484-3pmid:http://www.ncbi.nlm.nih.gov/pubmed/23649787
      OpenUrlCrossRefPubMed
    15. ↵
      1. Cabassa LJ
      . The role of implementation science in reducing healthcare disparities. Implementation research Institute. St. Louis: Washington University, 2018.
    16. ↵
      1. Wells KB,
      2. Jones L,
      3. Chung B, et al
      . Community-Partnered cluster-randomized comparative effectiveness trial of community engagement and planning or resources for services to address depression disparities. J Gen Intern Med 2013;28:1268–78.doi:10.1007/s11606-013-2484-3
      OpenUrlCrossRefPubMed
    17. ↵
      1. Anderson LM,
      2. Adeney KL,
      3. Shinn C, et al
      . Community coalition-driven interventions to reduce health disparities among racial and ethnic minority populations. Cochrane Database Syst Rev 2015;19:CD009905. doi:10.1002/14651858.CD009905.pub2pmid:http://www.ncbi.nlm.nih.gov/pubmed/26075988
      OpenUrlPubMed
    18. ↵
      1. Kirchner JE,
      2. Ritchie MJ,
      3. Pitcock JA, et al
      . Outcomes of a partnered facilitation strategy to implement primary care-mental health. J Gen Intern Med 2014;29:904–12.doi:10.1007/s11606-014-3027-2
      OpenUrl
    19. ↵
      1. Kilbourne AM,
      2. Almirall D,
      3. Goodrich DE, et al
      . Enhancing outreach for persons with serious mental illness: 12-month results from a cluster randomized trial of an adaptive implementation strategy. Implement Sci 2014;9:163. doi:10.1186/s13012-014-0163-3pmid:http://www.ncbi.nlm.nih.gov/pubmed/25544027
      OpenUrlPubMed
    20. ↵
      1. Dickinson WP,
      2. Dickinson LM,
      3. Nutting PA, et al
      . Practice facilitation to improve diabetes care in primary care: a report from the EPIC randomized clinical trial. Ann Fam Med 2014;12:8–16.doi:10.1370/afm.1591pmid:http://www.ncbi.nlm.nih.gov/pubmed/24445098
      OpenUrlAbstract/FREE Full Text
    21. ↵
      1. Ritchie MJ,
      2. Dollar KM,
      3. Miller C
      . Using implementation facilitation to improve healthcare implementation facilitation training manual (Version 3) [online]. Veterans health administration, behavioral health quality enhancement research initiative (QUERI), 2020. Available: https://www.queri.research.va.gov/tools/implementation.cfm
    22. ↵
      1. Powell BJ,
      2. Waltz TJ,
      3. Chinman MJ, et al
      . A refined compilation of implementation strategies: results from the expert recommendations for implementing change (ERIC) project. Implement Sci 2015;10:21. doi:10.1186/s13012-015-0209-1pmid:http://www.ncbi.nlm.nih.gov/pubmed/25889199
      OpenUrlCrossRefPubMed
    23. ↵
      1. U.S. Department of Veterans Affairs
      . VA National suicide data report 2005-2016 [online], 2018. Available: https://www.mentalhealth.va.gov/suicide_prevention/Suicide-Prevention-Data.asp
    24. ↵
      1. McCarthy JF,
      2. Blow FC,
      3. Ignacio RV, et al
      . Suicide among patients in the Veterans Affairs health system: rural-urban differences in rates, risks, and methods. Am J Public Health 2012;102:S111–7.doi:10.2105/AJPH.2011.300463pmid:http://www.ncbi.nlm.nih.gov/pubmed/22390583
      OpenUrlCrossRefPubMed
    25. ↵
      1. Stanley B,
      2. Brown GK,
      3. Brenner LA, et al
      . Comparison of the safety planning intervention with follow-up vs usual care of suicidal patients treated in the emergency department. JAMA Psychiatry 2018;75:894. doi:10.1001/jamapsychiatry.2018.1776pmid:http://www.ncbi.nlm.nih.gov/pubmed/29998307
      OpenUrlPubMed
    26. ↵
      1. Stanley B,
      2. Chaudhury SR,
      3. Chesin M, et al
      . An emergency department intervention and follow-up to reduce suicide risk in the Va: acceptability and effectiveness. Psychiatr Serv 2016;67:680–3.doi:10.1176/appi.ps.201500082pmid:http://www.ncbi.nlm.nih.gov/pubmed/26828397
      OpenUrlPubMed
    27. ↵
      1. Woodward EN,
      2. Matthieu MM,
      3. Uchendu US, et al
      . The health equity implementation framework: proposal and preliminary study of hepatitis C virus treatment. Implement Sci 2019;14:26. doi:10.1186/s13012-019-0861-ypmid:http://www.ncbi.nlm.nih.gov/pubmed/30866982
      OpenUrlPubMed
    28. ↵
      1. Nilsen P
      . Making sense of implementation theories, models and frameworks. Implement Sci 2015;10:53. doi:10.1186/s13012-015-0242-0pmid:http://www.ncbi.nlm.nih.gov/pubmed/25895742
      OpenUrlCrossRefPubMed
    29. ↵
      1. Harvey G,
      2. Kitson A
      . PARIHS revisited: from heuristic to integrated framework for the successful implementation of knowledge into practice. Implementation Science 2015;11.doi:10.1186/s13012-016-0398-2
    30. ↵
      1. Curran GM
      . Implementation science made too simple: a teaching tool. Implement Sci Commun 2020;1:27. doi:10.1186/s43058-020-00001-zpmid:http://www.ncbi.nlm.nih.gov/pubmed/32885186
      OpenUrlPubMed
    31. ↵
      1. Betancourt JR,
      2. Green AR,
      3. Carrillo JE, et al
      . Defining cultural competence: a practical framework for addressing racial/ethnic disparities in health and health care. Public Health Rep 2003;118:293–302.doi:10.1016/S0033-3549(04)50253-4
      OpenUrlCrossRefPubMed
    32. ↵
      1. National Research Council
      2. Institute of Medicine
      . Policies and social values. In: Woolf SH, Aron L, eds. US health in international perspective: shorter lives, poorer health [online]. Washington, DC: National Academies Press, 2013.
    33. ↵
      1. Thomson K,
      2. Hillier-Brown F,
      3. Todd A, et al
      . The effects of public health policies on health inequalities in high-income countries: an umbrella review. BMC Public Health 2018;18:869. doi:10.1186/s12889-018-5677-1
    34. ↵
      1. Melgar Castillo A,
      2. Woodward EN,
      3. True G
      . Examples and challenges of engaging consumers in implementation science activities: an environmental scan. oral symposium presented at: 13th annual conference on the science of dissemination and implementation. Washington, DC, 2020.
    35. ↵
      1. Palinkas LA,
      2. Aarons GA,
      3. Horwitz S, et al
      . Mixed method designs in implementation research. Adm Policy Ment Health 2011;38:44–53.doi:10.1007/s10488-010-0314-zpmid:http://www.ncbi.nlm.nih.gov/pubmed/20967495
      OpenUrlCrossRefPubMed
    36. ↵
      1. Dopp AR,
      2. Parisi KE,
      3. Munson SA, et al
      . A glossary of user-centered design strategies for implementation experts. Transl Behav Med 2019;9:1057–64.doi:10.1093/tbm/iby119pmid:http://www.ncbi.nlm.nih.gov/pubmed/30535343
      OpenUrlPubMed
    37. ↵
      1. Guest G,
      2. Bunce A,
      3. Johnson L
      . How many interviews are enough? an experiment with data saturation and variability. Field Methods 2006;18:59–82.
      OpenUrlCrossRefWeb of Science
    38. ↵
      1. Beebe J
      . Rapid assessment process: an introduction. Walnut Creek, CA: AltaMira Press, 2001.
    39. ↵
      1. Hamilton A
      . Qualitative methods in rapid Turn-Around health services research. HSRD Cyberseminar. US Department of Veteran Affairs, 2013.
    40. ↵
      1. Hsieh H-F,
      2. Shannon SE
      . Three approaches to qualitative content analysis. Qual Health Res 2005;15:1277–88.doi:10.1177/1049732305276687
      OpenUrlCrossRefPubMedWeb of Science
    41. ↵
      1. Helmer-Hirschberg O
      . Analysis of the future: the Delphi method [online]. Santa Monica, CA: RAND Corporation, 1967. https://www.rand.org/pubs/papers/P3558.html
    42. ↵
      1. Domlyn AM,
      2. Wandersman A
      . Community coalition readiness for implementing something new: using a Delphi methodology. J Community Psychol 2019;47:882–97.doi:10.1002/jcop.22161pmid:http://www.ncbi.nlm.nih.gov/pubmed/30688369
      OpenUrlPubMed
    43. ↵
      1. Weiner BJ,
      2. Lewis CC,
      3. Stanick C, et al
      . Psychometric assessment of three newly developed implementation outcome measures. Implement Sci 2017;12:108. doi:10.1186/s13012-017-0635-3pmid:http://www.ncbi.nlm.nih.gov/pubmed/28851459
      OpenUrlCrossRefPubMed
    44. ↵
      1. Rankin NM,
      2. McGregor D,
      3. Butow PN, et al
      . Adapting the nominal group technique for priority setting of evidence-practice gaps in implementation science. BMC Med Res Methodol 2016;16:110. doi:10.1186/s12874-016-0210-7pmid:http://www.ncbi.nlm.nih.gov/pubmed/27566679
      OpenUrlPubMed
    45. ↵
      1. Van de Ven AH,
      2. Delbecq AL
      . The nominal group as a research instrument for exploratory health studies. Am J Public Health 1972;62:337–42.doi:10.2105/AJPH.62.3.337pmid:http://www.ncbi.nlm.nih.gov/pubmed/5011164
      OpenUrlCrossRefPubMedWeb of Science
    46. ↵
      1. Carman KL,
      2. Dardess P,
      3. Maurer M, et al
      . Patient and family engagement: a framework for understanding the elements and developing interventions and policies. Health Aff 2013;32:223–31.doi:10.1377/hlthaff.2012.1133pmid:http://www.ncbi.nlm.nih.gov/pubmed/23381514
      OpenUrlAbstract/FREE Full Text
    47. ↵
      1. Moffatt S,
      2. White M,
      3. Mackintosh J, et al
      . Using quantitative and qualitative data in health services research - what happens when mixed method findings conflict? [ISRCTN61522618]. BMC Health Serv Res 2006;6:28. doi:10.1186/1472-6963-6-28pmid:http://www.ncbi.nlm.nih.gov/pubmed/16524479
      OpenUrlCrossRefPubMed
    48. ↵
      1. Curran GM,
      2. Bauer M,
      3. Mittman B, et al
      . Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care 2012;50:217–26.doi:10.1097/MLR.0b013e3182408812pmid:http://www.ncbi.nlm.nih.gov/pubmed/22310560
      OpenUrlCrossRefPubMedWeb of Science
    49. ↵
      1. Ritchie MJ,
      2. Liu CF,
      3. Townsend JC
      . Time and cost of “extreme” implementation facilitation to address challenging clinical contexts. 4th Biennial “Society for Implementation Research Collaboration”. Seattle, Washington, 2017.
    50. ↵
      1. Woodward EN,
      2. Pitcock JA,
      3. Ritchie M
      . Implementing integrated primary care in late adopter sites: the impact of key events on repeated measures of adoption. 9th annual conference on the science of dissemination and implementation in health. Washington, DC, 2016.
    51. ↵
      1. Thabane L,
      2. Ma J,
      3. Chu R, et al
      . A tutorial on pilot studies: the what, why and how. BMC Med Res Methodol 2010;10:1. doi:10.1186/1471-2288-10-1pmid:http://www.ncbi.nlm.nih.gov/pubmed/20053272
      OpenUrlCrossRefPubMed
    52. ↵
      1. Bauer MS,
      2. Damschroder L,
      3. Hagedorn H, et al
      . An introduction to implementation science for the non-specialist. BMC Psychol 2015;3:32. doi:10.1186/s40359-015-0089-9pmid:http://www.ncbi.nlm.nih.gov/pubmed/26376626
      OpenUrlCrossRefPubMed
    53. ↵
      1. Creswell JW,
      2. Klassen AC,
      3. Plano Clark VL
      . Best practices for mixed methods research in the health sciences. August 2011. National Institutes of Health. [Internet. Washington, DC: Office of Behavioral and Social Sciences Research of the National Institutes of Health, 2011. https://obssr.od.nih.gov/training/online-training-resources/mixed-methods-research/
    54. ↵
      1. Morse JM
      . Critical analysis of strategies for determining rigor in qualitative inquiry. Qual Health Res 2015;25:1212–22.doi:10.1177/1049732315588501pmid:http://www.ncbi.nlm.nih.gov/pubmed/26184336
      OpenUrlCrossRefPubMed
    55. ↵
      1. Orsmond GI,
      2. Cohn ES
      . The distinctive features of a feasibility study: objectives and guiding questions. OTJR 2015;35:169–77.doi:10.1177/1539449215578649pmid:http://www.ncbi.nlm.nih.gov/pubmed/26594739
      OpenUrlCrossRefPubMed
    56. ↵
      1. Proctor E,
      2. Silmere H,
      3. Raghavan R, et al
      . Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health 2011;38:65–76.doi:10.1007/s10488-010-0319-7
      OpenUrlCrossRefPubMed
    57. ↵
      1. Glasgow RE,
      2. Vogt TM,
      3. Boles SM
      . Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health 1999;89:1322–7.doi:10.2105/AJPH.89.9.1322
      OpenUrlCrossRefPubMedWeb of Science
    58. ↵
      1. Leon AC,
      2. Davis LL,
      3. Kraemer HC
      . The role and interpretation of pilot studies in clinical research. J Psychiatr Res 2011;45:626–9.doi:10.1016/j.jpsychires.2010.10.008pmid:http://www.ncbi.nlm.nih.gov/pubmed/21035130
      OpenUrlCrossRefPubMed
    59. ↵
      1. Dobkin BH
      . Progressive staging of pilot studies to improve phase III trials for motor interventions. Neurorehabil Neural Repair 2009;23:197–206.doi:10.1177/1545968309331863pmid:http://www.ncbi.nlm.nih.gov/pubmed/19240197
      OpenUrlCrossRefPubMedWeb of Science
    60. ↵
      1. Luke DA,
      2. Calhoun A,
      3. Robichaux CB, et al
      . The program sustainability assessment tool: a new instrument for public health programs. Prev Chronic Dis 2014;11:130184. doi:10.5888/pcd11.130184
    61. ↵
      1. Jull J,
      2. Giles A,
      3. Graham ID
      . Community-based participatory research and integrated knowledge translation: advancing the co-creation of knowledge. Implement Sci 2017;12:150. doi:10.1186/s13012-017-0696-3pmid:http://www.ncbi.nlm.nih.gov/pubmed/29258551
      OpenUrlPubMed
    62. ↵
      1. Grunbaum J
      . Challenges in improving community engagement in research. In: Principles of community engagement. 2nd edn. National Institutes of Health, 2011.

    Footnotes

    • Contributors ENW conceptualised the study and manuscript, and prepared all written materials, tables and figures. CW helped design methods and integration of mixed methods and edited the manuscript. SJL helped conceptualise the involvement of implementing Safety Planning Intervention and edited the manuscript. LRMH refined the study design and methods and edited the manuscript. KLD helped design analytic plans for qualitative components and edited the manuscript. SO designed analytic plans for quantitative components and edited the manuscript. IAB helped refine methods for Aim 1 (Developing Consumer Voice), edited the manuscript and developed supporting documentation, such as declarations, abbreviations, references and supplemental files. JEK helped develop conceptualisations of the study and manuscript and edited the manuscript.

    • Funding This work was supported by Career Development Award Number IK2 HX003065 from the US Department of Veterans Affairs Health Services Research and Development (HSRD) Service (ENW).

    • Disclaimer The views expressed in this article are those of the author and do not necessarily represent the views of the US Department of Veterans Affairs.

    • Competing interests None declared.

    • Patient and public involvement Patients and/or the public were involved in the design, or conduct, or reporting, or dissemination plans of this research. Refer to the Methods section for further details.

    • Provenance and peer review Not commissioned; externally peer reviewed.