Introduction There is a growing recognition of the importance of developing learning health systems which can engage all stakeholders in cycles of evidence generation, reflection, action and learning from action to deal with adaptive problems. There is however limited evaluative evidence of approaches to developing or strengthening such systems, particularly in low-income and middle-income settings. In this protocol, we aim to contribute to developing and sharing knowledge on models of building collaborative learning platforms through our evaluation of the Verbal Autopsy with Participatory Action Research (VAPAR) programme.
Methods and analysis The evaluation takes a participatory approach, focussed on joint learning on whether and how VAPAR contributes to its aims, and what can be learnt for this and similar settings. A realist-informed theory of change was developed by the research team as part of a broader collaboration with other stakeholders. The evaluation will draw on a wide variety of perspectives and data, including programme data and secondary data. This will be supplemented by in-depth interviews and workshops at the end of each cycle to probe the different domains, understand changes to the positions of different actors within the local health system and feedback into improved learning and action in the next cycle. Quantitative data such as verbal autopsy will be analysed for significant trends in health indicators for different population groups. However, the bulk of the data will be qualitative and will be analysed thematically.
Ethics and dissemination Ethics in participatory approaches include a careful focus on the power relationships within the group, such that all groups are given voice and influence, in addition to the usual considerations of informed participation. Within the programme, we will focus on reflexivity, relationship building, two-way learning and learning from failure to reduce power imbalances and mitigate against a blame culture. Local engagement and change will be prioritised in dissemination.
- health services administration & management
- health policy
- international health services
- public health
- primary care
- organisation of health services
This is an open access article distributed in accordance with the Creative Commons Attribution 4.0 Unported (CC BY 4.0) license, which permits others to copy, redistribute, remix, transform and build upon this work for any purpose, provided the original work is properly cited, a link to the licence is given, and indication of whether changes were made. See: https://creativecommons.org/licenses/by/4.0/.
Statistics from Altmetric.com
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.
- health services administration & management
- health policy
- international health services
- public health
- primary care
- organisation of health services
Strengths and limitations of this study
It combines realist approaches with participatory action research and allows for testing and refinement over several cycles.
Risks related to power imbalances and insider/outsider tensions are acknowledged, with mitigating strategies planned.
It aims to add to the limited literature on collaborative learning platforms to support learning health systems in low-income and middle-income settings.
The study will also build the field of evaluation of participatory research at multiple levels of the local health system.
One limitation is that results will be specific to this site, however broader engagement is planned to allow for sharing of lessons with other learning sites using related models.
Background on the VAPAR programme
The Verbal Autopsy with Participatory Action Research (VAPAR) programme started in 2017 in Mpumalanga province, South Africa, as a partnership of local and international researchers, community members and health system stakeholders. Its aim is to embed a system of knowledge production and exchange for health systems strengthening in order to improve services and outcomes for vulnerable group’s health locally and with the potential, if successful, for wider learning, uptake and sustainability (www.vapar.org).
In VAPAR, data from verbal autopsy (VA) and participatory action research (PAR) is combined in a series of reflection-and-action cycles based on continuous quality improvement for health systems strengthening, engaging relevant stakeholders at different levels of the health system (figure 1). The programme consists of three learning-and-action cycles over 2017 to 2022, with each cycle of the VAPAR programme including four stages: observe, analyse, plan and act.1
The VA component incorporates new WHO indicators developed with the Mpumalanga Department of Health during pilot work in 2015/2016 on social and health system factors contributing to the ‘circumstances of mortality’1 within VA data gathered through the Medical Research Council (MRC)/Wits Agincourt Unit's Health and Socio-Demographic Surveillance System (HDSS) in Mpumalanga. These VA outputs are shared during PAR with village-based groups, which identify priority topics, analyse root causes and the impact of the problem, identify stakeholders and plan action to address these with key stakeholder groups. During the pilot phase of the programme in 2016, the areas of under-5 mortality and HIV infection were nominated by researchers and the Department of Health; however in the main phase communities nominated access to clean water, alongside alcohol and drug abuse as key local health priorities.2–4 The last two topics were carried through by communities into the main phase in 2017. Data from VA and PAR have then been interpreted with district and provincial stakeholders (within the health sector but also beyond, as relevant) in order to reach common understanding on problems and root causes, leading to actionable agendas and promoting learning from that action in ongoing processes.
The research is informed by established post-positivist paradigms asserting that all truths are partial. The work is thus rooted in and draws from critical,5 constructivist6 and participatory/cooperative traditions.7 Through paradigmatic interweaving, we seek to deepen understanding in and of health systems as complex adaptive systems and social constructions8 located in and continually (re)shaped by wider social, political and historical contexts, and cooperative enquiry as political participation in collaborative practical knowing, action and transformation. Drawn together, the key underlying assumption is that practical, experiential knowledge that is co-constructed, self-reflective and embedded in complex, adaptive social and health systems will support and inform the organisation and delivery of public goods that are equity-oriented and people-centred. The beneficiaries are intended to be people and practitioners in resource-constrained systems, collectively possessing rights and responsibilities for health, healthcare and wider public services.
VAPAR is situated in Mpumalanga, South Africa, a rural province of 4.6 million people in the northeast, bordering Swaziland and Mozambique. Conditions in poor and rural villages are comparable with many other settings in the region: there is limited piped water, rudimentary sanitation, underdeveloped roads, unaffordable electricity and high unemployment.9 The burden of HIV in South Africa is high and highly unequal. Prevalence in black populations is 40 to 50 times that of white and in adolescents, risks are eight times higher in female adolescents than males.10 Age-adjusted HIV prevalence in the study area is 26% in women and 19% in men.11
In spite of entrenched social and health inequities, the post-apartheid policy context in South Africa is progressive and inclusive. There is a constitutional commitment to the right to health and community participation for primary care,12 and National Health Insurance was launched in 2012 as a clear commitment to Universal Health Coverage.13 Despite a ‘near-ideal’ policy context, there is chronic underinvestment in public services. This has resulted in human resource crises, corruption, poor stewardship and deteriorating infrastructure — and deep disconnects between policy and implementation as a result.14 The health system also faces a complex ‘quadruple’ burden of socially patterned disease comprising: chronic infectious diseases (characterised by HIV/AIDS and tuberculosis), non-communicable conditions, maternal and child mortality and mortality owing to injury and violence.15 16
Purposes of evaluation
In this paper, we lay out the approach which will be taken by our evaluation, which will be participatory among our key constituencies and will aim to understand whether and how VAPAR contributes to its aims, and what can be learnt for this and similar settings. We present the theory of change of the programme, how it was developed and is evolving and how it will be tracked using mixed methods. We discuss our positionality and some of the risks and ethical issues arising.
In addition to informing the development of the programme, the evaluation aims to build evidence on how to develop collaborative reflection and action for health through multiple levels of engagement and more authentic community engagement, which has been highlighted as a key gap area for health systems research17 generally as well as in the province over the period of engagement. More broadly, it aims to enrich learning on PAR and its evaluation, as well as contributing to current debates on learning health systems18 and on health system strengthening, with many of the areas of focus of VAPAR mapping onto suggested process goals for a stronger health system.18
Methods and analysis
Participatory evaluation is a growing field,19 however, evaluation of VAPAR faces additional challenges of evaluating an intervention which is itself participatory, multilevel, multicycle, pragmatic, emergent and embedded in rapidly changing contexts. Our approach therefore includes the following elements, which reflect the programme design:
It is post-positivist in epistemology, recognising that knowledge is valid only relatively to a specific context, society, culture or individual and is socially (and potentially cooperatively) constructed.6
It is participatory and embedded in that it will build on reflections and insights of partners and wider stakeholders which are generated as part of the PAR cycles.
It will be adaptive, to allow for changes in the programme and its environment which may occur over time.
It is theory-based and looking for contribution (https://www.betterevaluation.org/en/plan/approach/contribution_analysis) not attribution, starting from a hypothesised theory of change and examining actual changes against that.
It draws from critical realist evaluation5 20 in trying to identify mechanisms of change operating in specific contexts, and the outcomes to which they lead.
The focus on actors and institutions also allows us to probe into political economy factors — incentives, power relationships, ideas and ideologies21 — which will be important explanatory factors for why and how change does or does not occur.
We will in addition record resource intensity, including intangible costs for participants, as part of good practice for thinking about replication and scalability.
It is important to note that this is an evaluation of the VAPAR approach as a whole (a meta-evaluation, rather than focussing on local actions triggered individually). Methodological points of interest and innovation will include being able to test and refine our theory of change over repeated cycles, and being able to test the adaptation of realist approaches to participatory processes.
Stages of evaluation
Developing the initial theory of change
During the first PAR cycle in 2017 to 2019, the research team developed an initial theory of change for VAPAR, based on continuous interactions with community, health system and wider public administration stakeholders over a number of prior years (in pre-pilot and pilot phases), as well as wider literature and secondary data. This theory of change considers the challenges and resources in the context, the expected causal pathways for addressing challenges, including change mechanisms and their underpinning assumptions, and desired outcomes. The expected causal pathways were discussed with stakeholders at workshops in cycle 1 and the start of cycle 2.
Testing and refining the theory
In each cycle, data will be collected which will allow us to refine our understanding of the intervention. This will include qualitative and quantitative data collected by the programme (on context, inputs, activities, outputs, outcomes and assumptions), supplemented by end-of-cycle interviews and workshops. These will be reviewed by the programme team and key stakeholders at the end of each cycle, leading to refined engagement and a more developed theory of change.
This will bring together the learning from across the programme, describing the starting situation, the initial theory of change, how this was adapted over the years, the evidence of interplay of context, mechanisms, actors and outcomes, leading to the final theory of change for future testing and to inform sustainability and potential scale up and replication.
Theory of change
Figure 2 presents the initial theory of change.
The context factors outline some of the key challenges which the programme is seeking to address — such as lack of constructive engagement between communities and health system and organisational culture issues within the health system — as well as some of the resources and opportunities, such as a progressive policy environment and a growing network of institutions collecting data on community health.
The inputs represent the envisaged contribution of the VAPAR programme in (1) supporting co-production of timely and relevant local evidence on health and other challenges faced by the communities in our focal area, and (2) enabling exchange and engagement with stakeholders within the health and public administration system to develop local solutions, collaboratively implement them and reflect on implementation.
In relation to mechanisms of change, three channels are hypothesised:
Greater confidence in and commitment to co-producing and using evidence by all stakeholders, including as an input to services.
Improved relationships and trust between communities, researchers and health authorities.
Increased motivation and capacity for community involvement and localised evidence-based primary healthcare in health and other sectors.
Outputs are expected to include the establishment of a learning platform or space, ownership and uptake of locally-relevant evidence and ensuing collective action and reflection on action in an ongoing manner. A learning platform in this context is understood as a neutral, respectful forum in which to co-produce, exchange and use evidence for action, and to learn from that action.
Outcomes are broken into short-term changes, such as improved policy implementation within existing resources; medium-term changes, such as improved health service organisation, resourcing and delivery; and long-term changes, such as improved health behaviours and outcomes, but also the sustaining and transfer of the learning from the programme.
Although these are presented in a linear fashion, it is clear that these stages are connected, fluid and in continuous interaction, with the mechanisms key to bringing about change. The learning cycles present opportunities to engage in and analyse these repeated interactions over time.
Underlying assumptions were identified, many of which are themselves potentially influenced by the programme, such as:
The research institutions, Department of Health and local communities being able and willing to engage over time and being open to dialogue.
The three core constituencies having some flexibility of resources to be able to respond to new co-produced evidence.
There being sufficient social coherence to support movement towards shared priorities and actions.
There being sufficient stability in the health sector for receptivity to programme outputs.
There being a wider interest in distributed and collaborative PAR (in relation to transfer of lessons to other settings).
The main questions to be examined by the evaluation are summarised in table 1 below, which also points to the data source for answering them.
The evaluation will be led by a team member who has some independence from the VAPAR process but who is embedded and able to facilitate reflections and learning from the main stakeholders in the research team, communities and health system. Different perspectives will be compared, noting synergies and tensions across the group. The focus will be on joint learning, and understanding the explanatory factors as much as the outputs and outcomes.
Most of the evaluation ‘indicators’ are qualitative, reflecting the focus of the programme on changing ‘software’ such as relationships, trust, attitudes and skills (for example, in communication and evidence use). Many will be extracted from routine programme and secondary data sources compiled by the research team throughout the VAPAR process.
Data collection and analysis will be continuous, collaborative and inclusive, with reviews at the end of each cycle to aid partner learning and inform improvements to VAPAR engagement. Results will be discussed with key actors from local to provincial level, such as community groups, PHC supervisors, facility managers, community health workers, front-line health staff, health programme and research managers and agencies in other sectors such as water and sanitation, housing and the environment.
Quantitative data such as VA will be analysed for significant trends in health indicators for different population groups. However, the bulk of the data will be qualitative and analysed thematically, focussing on understanding relevant changes, their drivers, their perceived impact on different groups and their relationships with one another, how they interact with other changing context features, any unintended consequences (including negative) and the implications for future interventions to build similar learning platforms.
Patient and public involvement
The VAPAR programme emerged from and further develops participatory action research pilots working with community members in the district, so members of the public (not patients in this context) were integrally involved in its shaping, in the selection of priority topics, in the generation of evidence and its interpretation, as well as participating in discussion and dissemination events. This approach will continue to be followed in the evaluation activities, as it is a core component of our programme and evaluation approach.
Ethics and dissemination
While we emphasise a participatory approach, in line with our programme aims, this does create a risk of an overly optimistic assessment, with both insiders and external respondents seeking to emphasise positive outcomes. This risk will be mitigated by the continuity of data collection and the multitude of sources; findings on impact on collaborators will not, for example, be drawn from a single interview but from repeated observations and interactions across time. Regular practice of reflexive and self-critical thinking will also be practiced within the research team, drawing on our own learning regarding the construction of safe spaces in which constructive, respectful reflection and critique can be encouraged.
A degree of insider/outsider tension22 is acknowledged during this process.
Risk may arise due to a lack of involvement or commitment on the part of participants in communities and the health systems, lack of data and/or controversial and negative results. Steps will be taken to promote the evaluation through regular contact and dialogue with all involved. This will be underpinned by constructive accountability to mitigate against blame, negativity and a punitive focus, embracing failures as learning opportunities and negative results as well as successes with a view to understanding key mechanisms in both.
Principles of research ethics related to un-harmful ways to treat individuals will apply. Ethical considerations are also anticipated related to the interdisciplinary, relativist and transformative nature of the work, the less rigid distinctions between researchers, implementers and advocates and the commitments to knowledge for action.
In each phase, there will be actions to minimise potential harm or negative consequences to participants. Informed consent will be gained from all participants. Participants will be informed about the nature of the research, its aims, objectives, procedures and outcomes. Participants will be assured that identifying information will be anonymised, and will not be disclosed beyond the research team without permission. Preliminary results will be fed back to, and verified with, participants before being disseminated more widely. Participants will be reimbursed for time spent participating in the research via provision of subsistence and travel expenses. All participants will be free to leave the study at any time and for any reason. Efforts to develop partnerships and processes beyond the programme will be sought throughout.
Ethical arrangements that apply to the routine surveillance in MRC/Wits Agincourt Unit will apply to the VAs that will be acquired in the proposed research. Specific ethical issues relate to the PAR and health systems elements. PAR is underpinned by relativist and transformative epistemologies, and is a dynamic and context dependent process. These features may be unfamiliar to, or viewed as unscientific by, medical research ethics committees and discussion may be necessary on these features, which may incur delays in the ethical approval process. Furthermore, PAR is concerned with transferring power through the research process towards those most directly affected by the issues investigated.23 24 Ethical conduct is therefore considered in terms of continual checking and rechecking of categories and dynamics of power by those intended to benefit from the process. It is also acknowledged that the changing of situations of social exclusion through empowerment gained via learning from knowledge and action may be open to stigmatisation and negative consequence in communities for individuals involved. Potential risks from participating in the long-term acting on information will be explicitly considered with participants, investigators and the International Steering Committee (which also supports quality assurance within the programme). We will work with the MRC/Wits Agincourt Unit's Public Stakeholder Engagement Office in the event of disputes or other difficulties in transparent and constructive dialogue with communities or other stakeholders to address and resolve issues where necessary.
In the PAR and health systems consultations, protecting the identities of participants may not be possible or necessary. Time will be taken at the outset with participants to ensure that ethical principles are agreed, respected, implemented, revisiting fit and function during the process. Ethical challenges also arise related to anonymity and confidentiality of visual data. Participants using visual methods receive training on how and why to secure permissions from the subjects of images. Where photographic material is collected and used, separate release permissions will be secured.
The plans centralise the social nature of knowledge creation and transfer. Acknowledging that significant knowledge emerges from the combination of disparate perspectives,25 time will be invested to build relationships, trust and shared understandings where partners accommodate and learn from different cultures and systems and where control is shifted as far as possible to stakeholder groups. It is acknowledged that operating between diverse sectors and perspectives may introduce problems. Where intersectoral tensions are identified, they will be supportively and constructively addressed. The process will invest in understanding and aligning perspectives, acknowledging and accommodating differences and distinguishing roles, with the overall purpose of identifying and progressing collective agendas through partnerships that span boundaries for positive change. Potential difficulties will be mitigated against through a supportive and well-structured process, protected time, reliable data, shared dialogue, effective training and dedicated staff underpinned by principles of two-way learning.25 If conflicts, tensions or problems ultimately threaten the process, ad-hoc sessions (in person wherever possible) will be held. The focus will be to respect and safeguard the partnerships. If the process fails, then reasons why and lessons learnt will be documented as a contribution to the methodological literature.26
Acknowledging the relevance of where and how outputs are disseminated, reporting will be balanced between academic-practitioner literature and public media. Partner voices will be given space wherever possible, seeking to privilege the ‘local gaze’27 and provide lessons which support further local action and benefits. Quantitative data on VA will be available for scrutiny through Agincourt HDSS, however qualitative data will be curated by the programme team. Findings will be shared through local meetings, briefs, social media sites, conferences and academic publications. Wider collaboration and lesson sharing with other centres in South Africa and beyond which are testing learning health system models is also planned.
Collaborators We would like to acknowledge the support and inputs of all collaborators and research group members, including co-investigators Steve Tollman, Kathy Kahn, Peter Byass, Duduzile Mdluli and Barry Spies.
Contributors SW led on the development of the evaluation methods and drafting of the paper. LD, MvM, RT, JH, DM and GG worked on the development of the theory of change framework with SW. All authors contributed to the overall programme development, as well as the drafting and finalisation of the paper.
Funding The research is supported by the Health Systems Research Initiative from Department for International Development (DFID)/ Medical Research Council (MRC)/Economic and Social Research Council (ESRC) (MR/N005597/1, MR/P014844/1), South African Department of Science and Innovation, the University of the Witwatersrand, and the Medical Research Council, South Africa, and previously the Wellcome Trust, UK (grants 058893/Z/99/A; 069683/Z/02/Z; 085477/Z/08/Z; 085477/B/08/Z).
Competing interests None declared.
Patient consent for publication Not required.
Ethics approval Ethical approval has been obtained from the authors’ institutes and from the provincial health authority.
Provenance and peer review Not commissioned; peer reviewed for ethical and funding approval prior to submission.