Table 1

Evaluation questions

DomainEvaluation focus/indicatorsSources
Inputs
  • Provision/co-production of robust/timely evidence on local situation

  • Research spaces and processes enabling engagement and exchange of local (and wider, as relevant) evidence

  • Degree and quality of stakeholder engagement (by whom, and for which activities; participant understanding and perception of processes)

  • Relevance of data which is co-produced for local needs (match to known burdens of disease, link to priority and actionable topics, etc)

  • Intensity of activities and match to programme plans (number of community meetings, meetings with health system stakeholders, etc)

  • Use of budgeted resources by all partners

  • Changes made to programme approach based on learning through activities

  • Research briefs

  • Reflection from communities and authorities during each stage

  • Summary programme reports

  • End of each cycle review interviews/workshops

  • Published VAPAR research papers

  • Social media platforms – the conversation pieces, etc

  • Programme budget and expenditure

Mechanisms
  • Greater confidence/ commitment to co-producing, using and acting on evidence by all stakeholders

  • Improved relationships and trust between communities, researchers and health authorities

  • Increased motivation/capacity for community involvement by health system and research stakeholders

  • Jointly authored outputs

  • Regularity of meetings and other collaboration between stakeholders

  • Changes to stakeholder perceptions of relevance of partnership and evidence

  • Changes to stakeholder skills, engagement, confidence and self-efficacy, self-reported and as observed during interactions

  • Changes to stakeholder relationships (eg, better communication, less hierarchical blockages and punitive relationships)

  • Any process changes noted (including for wider Agincourt HDSS – for example, more proactive engagement with health system actors)

  • Participant feedback

  • End of cycle interviews/workshops

  • Systematic noting of observations and reflections on change from team members

  • Important emails stored in shared drive

  • Invitations to events between partners

  • Stakeholder mapping by team and study participants, repeated over time to record change

Outputs
  • Learning platform

  • Ownership/uptake of locally relevant evidence

  • Collective action and learning derived from it

  • Continued commitment to process (eg, attendance and active participation at meetings and in joint activities – in claimed and invited spaces)

  • Behaviour change by any of key stakeholders (greater focus on uptake by researchers, greater use of evidence by system, more community inputs into both)

  • Collective action plans and extent of their completion

  • Programme reports

  • Local action plans and follow-up reporting

  • End of cycle interviews/workshops

Outcomes
Short-term/foundational
  • Improved engagement researchers/communities/authorities

  • Improved awareness of and shared local health priorities

  • Improved healthcare processes/policy implementation with existing resources

  • Evidence of demand for continued exchange by all stakeholders (eg, independent meetings or collaboration, not linked to VAPAR)

  • Value given to different forms of evidence and inclusion of different evidence in decision-making processes

  • More evidence citation and use in local planning and review within health and other relevant sectors

  • Inclusive strategic review and reflection used to plan and prioritise locally

  • Changes to service planning and organisation linked to VAPAR and VAPAR-inspired processes

  • Programme reports and team observations

  • Cycle 3 interviews/workshops

  • Secondary reports, for example, district health plan, annual performance plan, integrated development plans, quarterly and annual reviews

  • Local action plans and other PAR outputs (eg, photovoice, root causes mapping, Venn diagrams)

  • Engagement and collaboration by other research in MRC/Wits-Agincourt

Medium-term
  • Legitimate learning platforms to produce and exchange local knowledge

  • Improved understanding of and commitment to equitable health priorities, including across sectors

  • Improved health service organisation, resourcing and delivery

  • Improved understanding of and experiences of health services by users

  • Perceptions of stakeholders on ownership, utility, impact on them personally of VAPAR-catalysed exchanges

  • Any notable changes in resources mobilised for health and how these are used

  • Reductions in reported problems for health services in study area (eg, stock-outs, unfilled posts, but also for outreach activities, for example, increased effectiveness of CHWs)

  • User satisfaction increased, as expressed through PAR, client surveys, reduction in community protests, etc

  • Cycle 3 interviews/workshops

  • Quarterly district health review reports on challenges, changes in organisation, resource use for each year

  • Satisfaction trends (from routine reports, VAPAR data, any relevant additional Agincourt data, press reports)

Long-term
  • Sustained legitimate learning platforms to produce and exchange local evidence

  • Organisational culture favouring evidence of different types (within health system and research institutions)

  • Supported decision-making to serve vulnerable and underserved populations

  • Policy and planning informed by local evidence

  • Improved health behaviours and outcomes

  • Improved distribution of behaviours and outcomes

  • Transferable process – shared learning

  • Continued support for VAPAR-inspired fora and activities

  • Greater use of local data in policy and programme documents in province

  • Clearer focus on marginalised communities in provincial health plans and reporting

  • Engaging and relationship building across sectors, horizontally and vertically, in support of Primary Health Care (PHC)

  • Greater access to and utilisation of essential health services

  • Reduced morbidity and mortality

  • Any other social impacts raised by participants

  • Trends in inequity: health outcomes, behaviours and services will be assessed in terms of gender, age, ethnicity and income

  • Dissemination and training materials and activities, deployed nationally and internationally

  • Uptake of VAPAR learning and approach in other provinces of South Africa (eg, through SAPRIN, the new network of HDSS sites in South Africa or other health system research centres) and potentially beyond

  • PAR narratives and visual data

  • Other relevant research reports (including VAPAR publications tracking specific health issues and training materials)

  • Funds to sustain and progress VA, PAR, local health policy and systems research, VAPAR

  • Provincial and district health plans and reports

  • District health information system data disaggregated

  • HDSS data disaggregated

  • VA data - trends

  • South African Population Research Infrastructure Network (SAPRIN) healthcare utilisation data

  • Reports by programme partners (eg, WHO, StatsSA, INDEPTH network, Code4SouthAfrica)

Context
Relevant changes to context will be tracked, including:
Opportunities, such as:
  • Supportive policy and legislative environment for health service delivery and community involvement

  • Research infrastructure exists in HDSS and expanding in South Africa


Also challenges, for example,
  • Top down/hierarchical governance in sector limits operational autonomy

  • Low accountability to service providers and users

  • System operates ‘in the dark’ in the absence of local data

  • Lack of communication/trust communities and authorities

  • Lack of power and representation of community

  • Limited incentives for researchers to engage with health system

  • Policy documents, including national (eg, on NHI, PHC re-engineering and relating to relevant other sectors, such as water and alcohol and drugs)

  • Annual provincial health expenditure data (from annual reports)

  • End of cycle interviews/workshops

  • Programme documents and data

  • Wider literature

  • VAPAR outputs, for example, tracking of decision space in the province for health)

  • Social media and other interactions such as webinars

  • Research infrastructure development – for example, SAPRIN/Agincourt

  • Systematic noting of observations and reflections on change by team members

  • News articles

  • Other MRC/Wits Agincourt Unit research

  • CHWs, community health workers; HDSS, Health and Demographic Surveillance System; MRC, Medical Research Council; NHI, National Health Insurance; PAR, participatory action research; VA, verbal autopsy; VAPAR, Verbal Autopsy with Participatory Action Research.