Article Text
PDF
Abstract
Introduction Process evaluations are conducted alongside research projects to identify the context, impact and consequences of research, determine whether it was conducted per protocol and to understand how, why and for whom an intervention is effective. We present a process evaluation protocol for the Getting it Right research project, which aims to determine validity of a culturally adapted depression screening tool for use by Aboriginal and Torres Strait Islander people. In this process evaluation, we aim to: (1) explore the context, impact and consequences of conducting Getting It Right, (2) explore primary healthcare staff and community representatives’ experiences with the research project, (3) determine if it was conducted per protocol and (4) explore experiences with the depression screening tool, including perceptions about how it could be implemented into practice (if found to be valid). We also describe the partnerships established to conduct this process evaluation and how the national Values and Ethics: Guidelines for Ethical Conduct in Aboriginal and Torres Strait Islander Health Research is met.
Methods and analysis Realist and grounded theory approaches are used. Qualitative data include semistructured interviews with primary healthcare staff and community representatives involved with Getting it Right. Iterative data collection and analysis will inform a coding framework. Interviews will continue until saturation of themes is reached, or all participants are considered. Data will be triangulated against administrative data and patient feedback. An Aboriginal and Torres Strait Islander Advisory Group guides this research. Researchers will be blinded from validation data outcomes for as long as is feasible.
Ethics and dissemination The University of Sydney Human Research Ethics Committee, Aboriginal Health and Medical Research Council of New South Wales and six state ethics committees have approved this research. Findings will be submitted to academic journals and presented at conferences.
Trial registration number ACTRN12614000705684.
- mental health
- primary care
- public health
- qualitative research
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Strengths and limitations of this study
Iterative data collection, supported by an Aboriginal and Torres Strait Islander Advisory Group will enable novel theory to be developed concerning the context, impact and consequences of conducting research in primary healthcare services.
Important information will be identified about the feasibility of conducting primary healthcare research that may enhance future research planning.
Results will contribute to the interpretation of a culturally adapted depression screening tool’s validity and acceptability for use by Aboriginal and Torres Strait Islander people, and inform its translation into practice (if optimal validity is established).
Potential limitations are the overlapping roles of researchers which may limit the sharing of negative experiences during data collection, but conversely may facilitate information sharing, analysis and interpretation. As per qualitative research guidelines this is acknowledged a priori.
Introduction
Process evaluations aim to assess how a strategy or programme is implemented; its impact; and how, why and for whom it is effective.1 This understanding is essential to determine whether a strategy is feasible, acceptable and applicable and can inform its roll-out, if it is shown to be effective. Typically, process evaluations are combined with complex health strategies or interventions. However, they can also highlight the unintended consequences of research, such as additional burden on staff or insufficient resourcing to conduct research according to the study’s protocol. For these reasons, process evaluations are increasingly being combined with research projects,2 3 and publication of process evaluation protocols is becoming more commonplace.4 5 This paper describes a process evaluation protocol of an Australian research project — Getting it Right: The validation study (hereafter referred to as the research project).
This national research project6 is focused on the social and emotional well-being (SEWB) of Aboriginal and Torres Strait Islander peoples (hereafter referred to as Indigenous).i It aims to determine the validity of the adapted-Patient Health Questionnaire-9 (aPHQ-9),7 a culturally adapted depression screening tool developed to identify depression. If validity is established, the aPHQ-97 would be the first culturally adapted, free-to-use, nationally validated, depression screening tool for use by Indigenous people and could be recommended for use in primary healthcare (PHC) and other healthcare settings. An understanding of experiences of PHC staff involved with the research project, including their perceptions about how the aPHQ-9 could be implemented, may inform the aPHQ-9’s implementation and future research in this area.
The research project study protocol is published elsewhere.6 In brief, recruitment of 500 Indigenous people attending PHC was completed in 2014 to 2016 at 10 PHC services (hereafter referred to as participating sites) nationally. Conducting the research project required coordination of many processes in the complex PHC setting, including the need to create a good fit alongside existing clinical requirements. This required commitment on multiple levels at each of the participating sites. Study processes were tailored by PHC staff, with support from researchers from The George Institute. SEWB includes mental health within a broad well-being framework and recognises well-being as interconnected with land, culture, family and community and recognises the role of historical, political and cultural determinants.8 9
In Australia, research involving Indigenous Aboriginal and Torres Strait Islander people should address the ethical standards outlined in Values and Ethics: Guidelines for Ethical Conduct in Aboriginal and Torres Strait Islander Health Research (hereafter Values and Ethics guideline).10 Partnerships and community involvement is a central principle of this guideline.10 Increasingly, research teams are describing how research partnerships are formed and operate.11 12 However, descriptions of how the Values and Ethics guideline 10 is used are scarce.
In this paper, we present our process evaluation protocol, including a description of the partnerships we have established to conduct this evaluation and documentation of how the Values and Ethics guideline is met.10 Data collection for this process evaluation began in December 2016 and is ongoing.
Process evaluation aims
In this process evaluation of the research project, we aim to:
Explore the context, impact and consequences (intended and unintended) of conducting the research project at participating sites.
Explore the experiences of PHC staff and community representatives with conducting the research project, including approaches to the research.
Determine if the research project was conducted as outlined in the protocol.
Explore the experiences of PHC staff with the aPHQ-9, including perceptions about potential for use of the aPHQ-9 (if found to be valid) and its acceptability and applicability.
Qualitative data will be considered alongside administrative data for the research project and feedback from PHC patients collected during the research project. This process evaluation may explain any variation in the research project’s results and will provide information on how, why and for whom the aPHQ-9 does or does not work. This will inform implementation of the aPHQ-9 in to clinical practice (if validity is established) and will also be useful when planning future research involving PHC staff and external researchers.
Methods and analysis
Approach to this process evaluation
This process evaluation incorporates components of the Medical Research Council’s guidance on process evaluations of complex interventions (MRC guidance),1 grounded theory13 and a realist theoretical approach.14 We use qualitative methods (semistructured interviews and thematic analysis). These data will be supplemented with administrative data and feedback from PHC patients (quantitative and free-text) about the research project and the aPHQ-9, collected during the research project. Data are collected, analysed and reported according to consolidated criteria for reporting qualitative research.15
The complex intervention under investigation in this process evaluation is the research project’s conduct at participating sites. Our aims differ from those of many process evaluations of complex interventions, where a causal assumption is under investigation, making elements of the MRC guidance 1 unsuitable. For example, rather than applying existing theories (as is common when investigating causal assumptions), we are exploring the context and experiences of staff members as they are presented to us. Therefore, an existing theoretical framework is not necessary or appropriate.
We draw on elements of grounded theory,13 as this is consistent with our aim to explore participants’ experiences as they are presented, to generate new theories. In addition, grounded theory is useful when there is little existing evidence in a research area, as is the case here. Grounded theory does not predefine codes for use during analysis, rather codes are identified from data as they are collected.16 In line with the MRC guidance 1 and grounded theory,16 we are iteratively analysing qualitative data so emerging ideas can be explored in subsequent interviews.
A realist approach14 to evaluations is becoming increasingly common,4 5 as it explores how, why and for whom an intervention is effective, therefore facilitating translation from research to practice. This approach recognises that the intervention itself may not wholly cause an outcome. Instead, it recognises that participants interact with an intervention and that the activities surrounding it (mechanism) within the social and cultural circumstances (context) alongside participant’s circumstances and beliefs, can influence outcomes.17 We use a realist approach to explore the context, mechanisms and outcomes (intended and unintended) related to conducting the research project.
Research partnership and reflexivity
The research team comprises a partnership established to complete this evaluation. This partnership involves an Aboriginal and Torres Strait Islander Advisory Group (the Group; established March 2016) and the research project’s project manager (SF) and chief investigator (MLH). The Group is made up of Aboriginal and Torres Strait Islander researchers and staff members from the research project’s participating sites. The Group provides cultural oversight and local input from sites to enhance and inform data collection, analysis and reporting. The Group’s aims are to:
Provide feedback and oversight of the appropriateness and quality of the semistructured interviews (interviews, setting, questions asked and prompts used)
Identify emerging themes from the data
Guide interview questioning according to the emerging themes (iterative process)
Develop a manuscript of results.
We have jointly developed this protocol in line with the Values and Ethics guideline.10 SF is the lead author of this research and data collected during this project will contribute towards her PhD research.
Qualitative data collection and analysis
We are collecting and analysing qualitative data through semistructured interviews using inductive grounded theory methods and coding data using constant comparison. Open coding is used to identify and label emerging ideas. We are concurrently collecting and analysing data, and are adapting our interview guide during the research. This process informs the coding framework, develops theory and is in line with a realist evaluation perspective.14
An interview guide including prompts is used during interviews. Interviews are face-to-face at the participating sites if possible, or via the phone. SF conducts interviews and another member of the Group joins interviews, when feasible. We plan to continue interviews until thematic saturation is achieved. We estimate approximately 8 of the 10 participating sites will take part, with 3 to 6 participants at each participating site. Therefore, an estimated 40 interviews will be conducted. However, the final numbers will depend on saturation of themes and availability of participants.
SF will code all interview transcripts. Two to three full interview transcripts are independently double-coded by SF and another member of the Group, at three times during data collection (six to eight interviews in total). During double-coding, interview transcripts are independently coded, then codes are compared and discussed until agreement around meaning is reached. SF completes coding for the remaining transcripts based on the agreed coding. Memos are used to document comments and discussion among the Group. Once the coding framework is finalised, we will attempt to relate the results to the values in the Values and Ethics guideline.10 We will consider if and how the codes can be attributed to the values described in the guideline10 using a set of previously developed definitions.18
To address our fourth aim related to the aPHQ-9, process evaluation interviews are conducted after recruitment into the research project is complete and before results are available. This ensures PHC staff and community representatives have recent experience with the research project and using the aPHQ-9, and reduces potential bias that may be introduced by unblinding the interviewer or interviewee during interviews and analysis. The primary interviewer (SF) and members of the Group will be blinded to outcomes for as long as feasible. Should results be made available before the process evaluation is completed, this will be acknowledged during thematic analysis.
Data sources and triangulation
We are collecting qualitative data through semistructured interviews with PHC staff and community representatives at recruitment sites. Administrative and feedback data (quantitative and free-text) will be considered alongside qualitative data to determine the acceptability and applicability and potential for use of the aPHQ-9.
Administrative data include screening logs, communication logs and study tracking documents for the research project. Feedback data include structured (quantitative) and free-text (thematic analysis) feedback from PHC patients about the research project and the aPHQ-9. Feedback was collected immediately after completing the aPHQ-9. In the structured quantitative feedback section, PHC patients were asked to rate their satisfaction with the number, type and wording of the aPHQ-9 questions, level of comfort with the questions, time available to respond to questions and response category options. They were then asked to provide any additional comments in the free-text section.
Data are triangulated in the following way:
Two (or more) members of the team code data and agree on appropriate codes (six to eight transcripts).
The Group reviews and provides feedback on an ongoing basis. Where necessary, further verification is sought from participants. This provides the opportunity for additional member checking.
The coding framework is compared with administrative data from the research project and quantitative and free-text feedback on the research project and aPHQ-9 collected from PHC patients during data collection. This provides further context for the data, opportunity to verify with study records and with PHC patients’ experiences who have experience using the aPHQ-9.
Sampling technique and data management
Participants may include any PHC staff or community representatives involved with some aspect of the research project’s design, approval or conduct. This includes members of community research boards (or alike) involved with community-level review and approval of research. Participants are purposively identified,1 through their existing involvement with the research project.
Participating sites nominate a staff member to facilitate engagement with this process evaluation. This staff member distributes the study information sheet and consent forms to potential participants (refer to online supplementary files 1 and 2). Potential participants will be invited to meet with the interviewer to show interest in the study. Interviews are transcribed verbatim. NVivo 10 for Windows software19 is used to manage data.
Supplementary file 1
Process for input from the Aboriginal and Torres Strait Islander Advisory Group during data collection, analysis and reporting
The following process is facilitated by SF during data collection, analysis and reporting:
Two to three full interview transcripts are independently coded by SF and another member of the Group. Codes are compared and discussed, until agreement is reached. Based on these codes, SF codes the subsequent transcripts.
Two to three full interviews (including the agreed coding) and a summary document is circulated to the Group. Members of the Group are invited to comment and provide feedback.
The interview guide is revised according to the agreed coding and feedback from the Group.
This process is completed after first stage of interviews, midway and at the end of data collection.
SF compiles feedback/comments from the Group. This information will be drafted into a manuscript. The manuscript will be circulated among the Group for comment and input.
How this process evaluation addresses the Values and Ethics guideline
Box demonstrates how the methods and approach used in this process evaluation address the Values and Ethics guideline.10 Further information on how the research project relates to the Values and Ethics guideline 10 has been previously published.6
Demonstration of how the principles of reciprocity, respect, equality, responsibility, ‘survival and protection’, and ‘spirit and integrity’ are considered in the Getting it Right process evaluation
Reciprocity
Getting it Right was initiated following the Men, Hearts and Mind study.29 This responds to community-identified need for social and emotional well-being strategies
Process evaluation follows discussions with primary healthcare staff about their preferences surrounding research conduct. This provides the opportunity for formal feedback to researchers and may enhance capacity by informing improved planning of future research
Process evaluation aims to evaluate research processes. This may contribute to the advancement of the health and well-being of communities by providing useful information on effective and appropriate research processes
It is anticipated that members of the Group* may develop new connections and skills through involvement with this evaluation. This may enhance capacity beyond this research
The Group’s* processes facilitate reciprocal learning between non-Indigenous and Aboriginal and Torres Strait Islander researchers
Option to provide participants with reimbursement (store voucher) for their input. This acknowledges participants’ contributions
Flexibility around interview timing and location. Option for individual or small group interviews. This demonstrates willingness to modify research processes according to communities’ values and aspirations
Respect
The Group* was established to guide the evaluation’s planning, conduct, analysis and reporting. This incorporates local knowledge and experience
Each participating site has the option to nominate a representative to be on the Group*. This acknowledges the diversity of communities
Members of the Group* are authors on research publications. This acknowledges the contribution of individuals and the expertise they provide
Publication plan includes input from participating site via the Group*. Results will be presented to the Group*, and proposed publications discussed including risks and benefits. This process incorporates Aboriginal and Torres Strait Islander knowledge and experience
Processes established around data management and publication. This will protect participants and communities’ identities
Approval for evaluation gained from community research boards (or alike) and ongoing information is provided, as required. This demonstrates community satisfaction with research
Equality
Each participating site has the opportunity to nominate a representative to be on Group*. This demonstrates equality between individuals, communities and researchers
A commitment to list all members of the Group* on the main publication, if they wish to have their contribution acknowledged in this format. This demonstrates equality between researchers
Opportunity for all members of the Group* to contribute to all aspects of the study, as determined by each member. This demonstrates equality between researchers
Research documents use clear concise language. Local processes or documents used (where developed). Researchers attend community research board meetings (when requested). This demonstrates intention to ensure understanding of research by individuals and communities
Participant information sheet and consent forms with clear usable language. This demonstrates the intention to ensure understanding of research by individuals and communities
Responsibility
Ethics approval obtained from eight Human Research Ethics Committees, including three Aboriginal and Torres Strait Islander committees. This demonstrates transparency by researchers and a commitment to ensure research is conducted ethically, the methodologies are appropriate and the research has benefit for people and communities
This manuscript has been reviewed and approved by an Aboriginal and Torres Strait Islander committee. This demonstrates transparency by researchers and a commitment to ensure research is conducted ethically, the methodologies are appropriate and the research has benefit for people and communities
The purpose, methods, conduct, and planned dissemination of results and potential outcomes/benefits of research outlined in an approved study protocol. Publication of the study protocol demonstrates agreements and transparency by researchers
Participants are provided with resources on social and emotional well-being and referral information. This demonstrates responsibility by researchers to ensure participants have access to confidential support, if required
Option to reimburse participants (store voucher) for the time and costs associated with participation. This demonstrates responsibility by reducing potential for harm to participants
A publication plan that involves joint sign off for publication and the protection of individual and community identity
The Group* provides mechanism for representatives to guide feedback of findings to communities
The Group* provides mechanism for ongoing community review of this evaluation
Survival and protection
Opportunity/intention for members of the Group* to participate in data collection through joint completion of interviews. This may protect against discrimination of individuals and cultures
Guidance to non-Indigenous researcher provided by researchers in the Group*. This reduces threats to cultural distinctiveness
Input from community representatives on Group* reduces threats to cultural distinctiveness
An Aboriginal artist was commissioned to design and complete artwork to represent Getting it Right on study-related materials. This provides opportunity for cultural distinctiveness
Spirit and integrity
Efforts by researchers to learn about each community by ensuring adequate time is available when visiting participating site , attending service and community events prior to and throughout study and seeking out and sharing stories. This recognises the diversity of cultures and personal integrity
Budget available for researchers to visit communities multiple times to learn about the community and the local context
*‘The Group’ is an Aboriginal and Torres Strait Islander Advisory Group involving researchers and staff involved with Getting it Right through participating sites.
Ethical considerations
Each participating site’s nominated staff member makes initial contact with potential participants. This ensure participants can consider risks and benefits of participation and do not feel obliged to take part. Identifying information (including names of individuals, participating sites and local references) will be removed from transcripts. To ensure access to SEWB support is available if required, referral information on local services is provided to participants. We identified the potential risk of interview staff working on the research project experiencing vicarious trauma. We provide resources on vicarious trauma to participants. Reimbursement (store voucher) for the time and costs associated with participation is available to participants, as determined by each participating site. Approval for this process evaluation is obtained from each participating site.
Ethical approval for this process evaluation has been provided by the following committees: The University of Sydney Human Research Ethics Committee (2014/361), Aboriginal Health and Medical Research Council of NSW (1044/14), ACT Health HREC (ETH.8.14.207), Queensland Health Metro South HREC (HREC/14/QPAH/503), Central Australian HREC (HREC-15–287), Menzies School of Health Research (2014–2289), Aboriginal Health Council of South Australia (04-17-705) and Western Australian Aboriginal Health Ethics Committee (607).
Dissemination of results
A manuscript will be submitted for publication in an academic journal. The final manuscript will be approved by the Group prior to submission. Findings relevant to the aPHQ-9 will be presented to the research project’s Steering Committee for consideration during the interpretation of research results.
Discussion
Understanding the contexts that surround and shape the way mechanisms can facilitate successful research is important to ensure research is acceptable to communities and results in health gains. Understanding how, why and for whom strategies work (or do not work) is key when translating research into practice, especially in complex and diverse settings with multiple completing priorities. In this evaluation, we aim to address these issues by exploring and documenting the experiences of PHC staff and community representatives involved with a complex national SEWB research project focused on Indigenous people.
The importance of involving community representatives with research is well established,10 20 however, systematic reporting of how this is completed not yet commonplace. This is demonstrated by a recent review of child health research21 which found reporting of if or how involvement was achieved in only 28.6% of the 217 studies included. By describing our research partnership which involves community representatives and external researchers, this paper contributes to the literature in this area.
The need for high-quality research to influence gains in health outcomes among Indigenous peoples is well recognised.22–24 Authors of a recent review25 focused on adolescent health research called for particular attention for SEWB research focused on Indigenous adolescents, due to the lack of evidence in this area. However, there appears to be challenges associated with conducting this research, including identifying Indigenous research staff,26 recruiting participants27 and resourcing.28 This research project will provide much needed SEWB evidence. This process evaluation will describe how the research project was conducted and the experiences of the PHC representatives involved. In this protocol, we describe our partnership established to conduct this evaluation and identify some actions relevant to the Values and Ethics guideline.10
The overlapping roles of some members of the research team are a strength and weakness of this evaluation. These dual roles provide an in-depth understanding of the research project, which may enhance data collection, analysis and interpretation and provide substantial opportunity for verification. However, these roles have the potential to bias data collection and interpretation. The position of the project manager as the main interviewer and the existing relationships may influence the responses provided by participants. We have attempted to identify some key areas where this evaluation addresses the Values and Ethics guideline.10 We acknowledge this is not a comprehensive list and that overlap between values occurs. We recognise that the diversity among Aboriginal and Torres Strait Islander communities means that our findings may not be relevant to other communities.
Conclusion
We are conducting a process evaluation of a large, complex research project focused on the SEWB Indigenous people and conducted at 10 PHC services around Australia. We are exploring the experiences of the PHC staff and community representatives involved with the research project at the participating sites, including their perceptions about how the aPHQ-97 could be implemented into practice. We have established an Aboriginal and Torres Strait Islander Advisory Group to guide this work. We publish this protocol to contribute to the literature and to inform planning of research with Indigenous people, with regard to the Values and Ethics guideline.10
Supplementary file 2
References
Footnotes
↵i The terms ‘Aboriginal’ and ‘Torres Strait Islander people’ are used to identify Australia’s First Peoples and to refer to and recognise the two distinctive Indigenous populations in Australia; Aboriginal and Torres Strait Islander people. ‘Non-Indigenous’ refers to those who do not identify as a member of the community of First Peoples of their respective countries. In this manuscript we use the term ‘Indigenous peoples’ throughout this article to respectfully refer to all Aboriginal and/or Torres Strait Islander peoples of Australia.
Contributors SF leads this research including protocol development, conducting interviews and coordinating feedback from the group. JE is SF’s PhD supervisor and has supported protocol development. A-ME, JF and GG have supported protocol development. BH, MS, KDM, GG, A-ME, JF and JE comprise the Advisory Group. MH is SF’s supervisor. All authors contributed to the manuscript.
Funding Getting it Right: The validation study was supported by National Health and Medical Research Council (NHMRC) Australia grant number APP101767. During the completion of this work, SF was in receipt of a University of Sydney Faculty of Medicine Cross Cultural Public Health Research Award and a George Institute for Global Health John Chalmers Program Grant Scholarship. MLH was in receipt of a National Heart Foundation Future Leader Fellowship no 100034. AME was funded by NHMRC APP1117198 Centre of Research Excellence: Indigenous Health and Alcohol.
Competing interests None declared.
Ethics approval The University of Sydney Human Research Ethics Committee (2014/361), Aboriginal Health and Medical Research Council of NSW (1044/14), ACT Health HREC (ETH.8.14.207), Queensland Health Metro South HREC (HREC/14/QPAH/503), Central Australian HREC (HREC-15-287), Menzies School of Health Research (2014-2289), Aboriginal Health Council of South Australia (04-17-705) and Western Australian Aboriginal Health Ethics Committee (607).
Provenance and peer review Not commissioned; externally peer reviewed.