Description of in-DEPtH framework

We developed a framework that can be used to incorporate locally relevant data and multi-factor considerations from various stakeholders to inform priority setting and decision making in the context of complex health programme interventions. We term this the in-DEPtH (evidence-informed, co-creation framework for the Design, Evaluation and Procurement of Health services) framework. in-DEPtH aims to provide a systematic approach to incorporate evidence on health programmes, local context realities and stakeholder multi-factor considerations to co-create specific health programme features that are relevant and applicable to individual health agencies and their areas of service. It is envisaged that prioritised health programme features can be directly used as specifications for the procurement of health services.

The framework has several steps. First, a search for relevant studies (both quantitative and qualitative) is conducted. Quantitative studies, if appropriate, undergo meta-analysis while qualitative studies are used to identify barriers and facilitators to the health programme of interest. The identified barriers and facilitators are converted into pertinent programme features. Through engagement and discussion with stakeholders, the pertinent features are synthesised (ie, research evidence and expert opinions are aggregated) and refined to suit local context and realities at the coalface. For each synthesised programme feature, published and grey literature are searched to inform the costs and effects of each feature, which is then shared back with local stakeholders. The synthesised features are then prioritised through a series of Delphi rounds to identify the most important features. Finally, the ranked programme features are then recommended to decision-makers so that they can decide in an informed and transparent manner, with the assurance that the considerations and assessments of various Delphi participants (stakeholders, clinicians, users and communities) have been systematically incorporated.

The individual steps of the framework are explained as follows:

1. Define research question and search inclusion criteria: Population, intervention, comparison and outcomes for quantitative effectiveness studies, and population, intervention of interest and contextfor qualitative studies. This should be jointly defined with the commissioning health agency to ensure the right and relevant research question and health outcomes are addressed.

2. Search for studies (quantitative and qualitative) that are relevant for the research question, inclusion criteria and context.

3. For included quantitative studies conduct meta-analysis (if appropriate).

4. For included qualitative studies:

   1. Identify and extract barriers and facilitators (until data saturation is reached).

   2. Group similar barriers and facilitators.

   3. Condense and convert to statements on programme features (be as specific as possible).

   4. Relate features back to included quantitative studies, and whether these studies reported positive or negative results (this is to understand success/failure factors and to incorporate learning points back into the extracted programme features, eg, factors on dosage/intensity).

5. Synthesise extracted programme features from primary evidence together with features included in corroborated positive trials, and existing programme specifications of the health agency, if any (ie, co-create desired health programme features via synthesis of data and engagement with stakeholders).

6. For each synthesised programme feature:

   1. Conduct a search for evidence (published or grey literature) on outcomes and costs.

   2. Analyse data to generate estimates of the costs and effects of each programme feature.

7. Incorporate inputs from stakeholders via Delphi process to rank programme features for priority setting:

   1. Two initial Delphi rounds: (a) First round: rank based on level of benefits to patients/consumers. (b) Second round: rank based on level of difficulty to implement (ie, deliverability).

   2. Third round: Results of the first two rounds will be shared with Delphi participants to inform the third round. Taking into account results of prior two rankings, features to be finally ranked.

8. Recommend prioritised programme features to senior executive management for decision making.

Figure 1 shows the steps of the framework in a graphical representation.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Graphical representation of the evidence-informed, co-creation framework for the Design, Evaluation and Procurement of Health services.

The structured synthesis of evidence and local stakeholder considerations provides a transparent and systematic basis for stakeholders to contribute to the commissioning of health programmes, thus improving stakeholder satisfaction and acceptance of the changes. Also, with the assurance that research evidence, local context and inputs of various stakeholders (health agencies, healthcare providers, clinicians, users and communities) have all been systematically incorporated and prioritised, decision-makers will be better empowered and equipped to act.

Methodologies applied in framework

The in-DEPtH framework uses concepts from the methodologies of realist review, qualitative descriptive analysis and integrative review. The adaptation of these methodologies to the framework is described as follows.

Realist review

A realist review is based on a realist philosophy of science and considers the interaction between context, mechanism and outcome.3 The realist reviewer seeks to answer the question: ‘What works for whom under what circumstances, how and why?’3 4 In a realist review, the following steps are involved. First, the main ideas behind the intervention is elicited from literature (ie, the intervention programme theory is formulated). Next, the relevance and effectiveness of each theory idea is verified using various types of evidence (such as qualitative, quantitative, comparative and administrative) from both published and grey literature sources. Searches are conducted iteratively and are purposive and targeted in approach to answer or test specific questions or theories.4 For each theory idea, the reviewers aims to understand the contextual factors that triggers the mechanisms which, in turn, generates the outcome of interest. By comparing the intervention programme theory to empirical evidence, the reviewers seek to determine the many contingencies and circumstances (ie, what works for whom in what situations) that affect the ability of the intervention to generate the intended outcomes. The information gained from a realist review helps policy makers to interpret and use an explanation of why a programme works better in one context than another.4 The last step in a realist review is to implement, test and evaluate recommendations with stakeholders in particular contexts.

Complex health programme interventions can be viewed as complex systems consisting of multiple service delivery features/components and being subject to various dynamically changing and interacting factors such as human behaviours/perceptions, skills level/capacity, macro/local policies, physical structures and resource allocations. This social reality cannot be measured directly, but can be known indirectly3 and because a realist review adopts an approach that sits between positivism and constructivism,5 it is particularly suited for examining and analysing complex health programme interventions.3 4

For our framework, we adapted the approach of a realist review to suit the nature and scope of our research aim. The realist review approach serves to guide us in examining complex health programme interventions, in conducting iterative, purposive searching to understand specific aspects as new information is uncovered, and in testing and evaluating the recommendations. The realist approach also enables us to understand contextual and social relational factors surrounding the programme features. However, unlike the realist review, we do not formulate a programme theory and our approaches towards data appraisal and synthesis is based on methods of qualitative descriptive analysis6 7 and integrative review.8

Using the approaches of descriptive analysis, we identify and extract barriers and facilitators of a health programme from qualitative primary studies and, due to quality considerations, these primary studies are searched from published, peer-reviewed journals (framework steps 4i and 4ii). The analysis and synthesis of data follows an integrative review approach, where similar data is grouped, reduced into summary statements, compared with other data and displayed in tables and matrices (framework steps 4iii, 4iv and 5). In framework step 6, we again adopt the realist approach of searching for evidence for each synthesised feature, that is we use iterative and purposive searches to answer specific questions and find data for economic costing and outcome effect sizes. For these searches, both published and grey literature are included to maximise data collection and refinement.

Qualitative descriptive analysis

The process to identify specific health programme features/components starts with the descriptive analysis of qualitative research evidence. Qualitative descriptive analysis involves the identification of findings that are close to the data, with minimal transformation (ie, with low imputation of meaning by researchers).6 9 The descriptive analysis approach enables us to identify and extract data to describe specific barriers/facilitators, what works/does not work as they are presented, that is, manifest context is described with low interpretation of data.7 We use the descriptive analysis approach for this step of the framework, as it allows us to understand the actual nature of the difficulties and challenges faced by those involved in specific local contexts, thereby giving us a sharper resolution of the problems that are being encountered on the ground, without any loss of details or contextual meaning.

The qualitative research evidence is usually generated from interviews and focus group discussions with various ‘actors’ of the health programme intervention being examined. The qualitative research evidence used in descriptive analyses could be studies that are conducted before the start of a health programme, after implementation of a health programme or even quantitative trials assessing the fidelity and effectiveness of health programmes when they contain relevant information on what works/does not work. The pre-program studies serve to examine local contextual factors and are particularly useful in understanding the barriers and facilitators at ‘ground zero’. The post programme studies examine the out-workings of certain implemented health programmes and serve to describe the programme processes, identify what worked/did not work and suggest potential improvements. To maintain the quality of evidence, such studies should come from published, peer-reviewed sources.

Integrative review

The next stage of the framework involves the analysis and synthesis of data. For these steps, the data analysis and presentation stages of an integrative review are applied, as it enables us to summarise data, compare iteratively across data, synthesise data elements into an integrated summation and to present conclusions with a logical chain of evidence.8 Integrative review has the potential to capture the complexity of varied perspectives and provide a comprehensive understanding of problems relevant to healthcare and policy,8 10 11 which makes the principles of an integrative review particularly applicable for this stage of the framework.

The application of the integrative review (data analysis and presentation stages) for steps 4iii to 4iv and step 5 is described as follows. From the descriptive analysis, a list of barriers/facilitators or problems/solutions is identified. This list of data is sorted and those that addressed a similar aspect are grouped together. See table 1 for an illustrated example. Relevant data (ie, components) about a similar aspect is extracted from each study, and studies with no relevant inputs are left empty. The extracted information is then condensed into a programme feature on the right. During the formulation of the programme feature, attention should be paid to retain as much specific details as possible, as this will facilitate the use of these programme features as specifications in procurement documents. In table 2, an exemplar case study has been applied to further illustrate how a programme feature is extracted from individual studies.

In the exemplar case study, a programme feature on access to general practitioners (GPs) by aged care residents is illustrated. Relevant data about access to GPs was extracted from the qualitative studies identified in a review of evidence on multi-component interventions (three studies did not report relevant inputs on access to GPs). In the example above, the data was condensed and aggregated to an identified programme feature that captures the essence of GP access across all studies. Factors reported in the reviewed studies were included in the identified programme feature to provide as much specific details as possible.

The quantitative studies included for meta-analysis are then compared against each of the identified programme features. Programme components that address the identified programme features are extracted from each quantitative study and grouped according to the identified programme feature. This generates a table that links identified programme features with components of the programmes evaluated by the identified quantitative studies. Please see an illustrated example under table 3.

This approach allows us to iteratively compare the quantitative studies against the identified programme features: horizontally we can compare the different intensities/dosages of the programme components that relate to each programme feature and vertically we can compare the extent to which studies addressed the identified programme features. The aim is to generate an understanding of the reasons why studies did or did not report positive trial results. This comparative understanding can also help us to interpret the result of the meta-analysis, either reinforcing or contradicting it. In the case that the aggregated result of the meta-analysis is inconclusive, we can compare the trialled outcomes of individual studies with the comparative understanding gained. If the comparative understanding explains why a particular study should be effective and this is also corroborated by the positive trial outcome of that study, we have a good concurrence between the two. Table 4 shows an exemplar case study to illustrate the comparison between identified programme features and quantitative studies.

In the exemplar case study, a programme feature on availability of clinical expertise was compared across corresponding components extracted from four quantitative studies. From the comparison, we found that Fan et al12 had the most components related to the feature (ie, higher intensity/dosage). By comparing with other identified programme features (ie, by comparing vertically [not illustrated here]), we found that the Fan et al12 addressed most of the identified programme features. The Fan study also reported positive trial results, and all this led us to conclude that the Fan trial was a positively corroborated study.

Next we compare the list of identified programme features against corresponding programme components of any identified corroborated studies that have positive trial outcomes. At this stage, if current programme specifications from the health agency are available, they should be incorporated into the analysis framework. Data from the health agency programme specifications have to be extracted and grouped according to the list of identified programme features. This process enables us to horizontally compare the descriptions of identified programme features across corresponding component descriptions of the corroborated studies and the health agency specifications. We then abstract and synthesise these similarly grouped component descriptions into synthesised programme features. Please see an illustrated example under table 5. Table 6 shows an exemplar case study to illustrate the synthetisation of a programme feature.

The exemplar case study shows the synthetisation of an identified programme feature on advanced care directives (ACDs). We compared the ACD programme feature across corresponding descriptions from the corroborated Fan et al12 study and the current programme specifications of a health agency. Here, the Fan et al12 study had no relevant data on ACD. However the current programme specifications stated the use of an ACD tool (7 Step Pathway - community version), which was subsequently added to the ACD programme feature to form the synthesised version. In an actual case study, further inputs would have to be sought from stakeholders in order to solicit their views on the syntax of the synthesised feature and incorporate their expert opinions into the synthesised programme feature.

In this way, local contextual factors reflected in programme specifications of the health agency, positively trialled components from corroborated quantitative studies and identified programme features extracted from qualitative evidence sources are all incorporated into a synthesised feature that is both locally relevant and evidence-based. To further enhance its relevance and acceptability, stakeholders should be engaged to review the descriptions of synthesised programme features jointly with researchers. Another benefit of this approach is that it allows both stakeholders and researchers to understand the differences between the various sources of data, especially the gaps between programme specifications of the health agency and identified programme features that have been extracted from primary evidence sources. This enables a richer and more thorough discussion during the synthesis of programme features.

Patient and public involvement

The research question is developed from the prevalent public health understanding that there is a translational gap between evidence and the practicalities of health service commissioning. During the co-creation process, we aim to involve patients and health consumers, together with clinicians, service providers and the commissioning health agencies to achieve a better translation from evidence to practice. For example, the patients and consumers could be involved to formulate the research question, jointly review the evidence with commissioning health agencies and participate in the stakeholder Delphi process.

Co-creation with stakeholders

The co-creation process of the framework is critical to ensuring that changes to the health programmes are acceptable, feasible and sustainable. By systematically and thoughtfully incorporating various local contextual factors and stakeholder considerations, we seek to achieve priority setting and decision making that is localised and relevant for the specific population area, where the health programme is to be commissioned.

The co-creation process with stakeholders takes place systematically over steps 1, 5, 6 and 7 of the framework. In step 1, the research question and scope are jointly defined with the commissioning health agency and stakeholders so that the purpose, programme context and outcomes of the research are correctly understood and agreed to by all involved parties. Next, under step 5 synthesis of the programme features occurs. Here, data extracted from evidence is aggregated with the existing programme specifications of the commissioning health agency, and contextual limitations and enablers are also incorporated into the synthesised programme features. We expect synthetisation to occur via workshops with stakeholders, where the research team will present on the findings and a facilitator will assist to guide the discussion to reach consensus. Specifically, stakeholders would jointly deliberate with the research team on the specific phrasings to include/exclude for a particular programme feature. Through such facilitated workshops, the various stakeholders will have to consider the relevance and applicability of the research evidence to the local context. The facilitated workshops will encourage stakeholders to be engaged and be explicit about the rationale for their decisions. Through this process of joint dialogue and exchange of ideas, the desired health programme features are refined and ‘polished’. This stage of the process is expected to be the most time consuming, but it would be prudent not to rush through step 5 and instead devote more time to reach an understanding and consensus, as the subsequent steps 6 and 7 will depend on the outcome of this step.

How the programme features are organised for the Delphi process is also determined at this stage. For example, if there is a positive corroborated study of programme features that are relevant to the local context, the health agency may well choose to simply adopt those features which are reflected in the study. For those remaining synthesised features that are not present in a corroborated study, these could then proceed for prioritisation via the Delphi method. This is just an illustration of what might occur at this stage of the framework. The actual organisation of the synthesised programme features will vary, depending on how the co-creation with the health agency progresses.

It is also to note that steps 5 and 6 are inter-connected and can be iterative. The cost and outcome data obtained for each synthesised feature could influence the aggregation of the programme features. There should be flexibility and regular communication with the health agency and stakeholders for steps 5 and 6 in order to maximise the synergies of data and stakeholder inputs.

In step 7, a wider range and quantity of stakeholders is engaged in a Delphi process to seek their expert opinions in order to prioritise the synthesised features. The wider group can extend to health agencies, healthcare providers, community and consumers. This stage of the framework serves to co-create the combination of features that are beneficial and truly valued by the specific population area. At the same time, the co-creation provides insights into the levels of deliverability by the healthcare providers.

Application of diverse analysis methodologies

We used several methods of review and analysis in our in-DEPtH framework, namely: realist review, descriptive analysis and integrative analysis. The reasons to do so are twofold. First, the nature of health programme interventions is complex and we need to be able to ‘dissect’ the programme system into components of data that can be worked and analysed. Second, we need to be able to generate specific details of evidence-informed programme features that both reflect on-the-ground realities and which can be synthesised to incorporate further evidence and stakeholder inputs for priority setting. These synthesised features should be clear and specific, with the intention that these features can be used directly as procurement specifications. Given the outputs required, we find that a ‘one size fits all’ approach of using one particular method of analysis is not suitable. Instead, using a combination of methods allows us to achieve outputs that are practically useful for commissioning activities.

Combining quantitative meta-analysis findings with qualitative findings

In steps 3 and 6ii of the framework, meta-analysis is conducted for quantitative data. However meta-analysis may not be appropriate if the trial data pertain to intervention programmes that are not homogeneous.13 14 This would apply particularly to step 3, as at this stage of the framework, it is likely that quantitative studies are trialling complex programme interventions which consist of different components. If the trial programme components are different between studies, then in essence each trial is testing different aspects of the intervention, even though the trial population, context and outcome measures are similar. In our experience, given the complex nature of health programmes, it is likely that meta-analysis would not be appropriate at step 3.

At step 6ii, however, it may be appropriate to conduct meta-analysis, as at this stage, we are looking at individual programme features and not the health programme as a whole. At this ‘smaller-scope’ level, the trials would be measuring outcomes and cost for a more specific intervention scope. Hence it is likely that the trial studies will be more homogeneous and hence possibly suitable for meta-analysis. The challenge, though, is the availability of data as there might not be studies which focuses on specific areas, given that trials are resource intensive and usually aim to test a wider scope. The availability of empirical cost data for individual programme features is also likely to be limited. Nevertheless, if quantitative data is available, it can provide useful intelligence to the synthesised programme features.

Limitations of framework

Prioritising research efforts and maximising use of existing evidence

Under steps 2 and 6i of the framework, searches for research evidence are performed. If there is insufficient primary evidence to address the pertinent research question defined by health agencies under step 2, the need for primary research is highlighted. Also, the framework, through the use of evidence to value programme features under step 6i, searches for evidence on cost and effects data, and where features warrant further evidence, future research efforts can be directed to generate data. On the other hand, where evidence is available and sufficient (both in terms of quantity and quality), then a particular feature has been adequately addressed. In this manner, the framework identifies evidence gaps and the need to prioritise research efforts to generate more data to address these gaps, while also maximising the use of existing evidence for those areas where there are adequate evidence.

Insufficient research evidence

In the absence of sufficient evidence to address the research question, the contribution of other sources of data (such as expert opinions and local intelligence) would gain greater emphasis. The framework is still expected to inform the definition and selection of programme features, even in the presence of minimal research evidence. Though not fully evidence based, the framework outputs could still be considered valid, in view of the contribution of expert opinions. Over time, as research efforts are directed towards addressing the gaps, the framework could be updated to incorporate the new evidence.

Generalisability to other healthcare systems

It is envisioned that the framework can be used in the healthcare systems of different countries, as the methods of the framework are context neutral. The important starting point is to define the research questions, health outcomes and search inclusion criteria, such that they fit the nature of the intervention to be investigated at the local contextual setting. The subsequent steps of evidence synthesis and Delphi consultations will then be relevant and meaningful to local stakeholders, such that the outputs from the framework will be appropriate to the specific environment where the health programme intervention is to be adopted.

Engagement with health partners

We are currently engaging health service partners to apply the framework in their commissioning activities. Through this process of real-world testing for specific health priorities (such as aged care, mental health, etc), we hope to evaluate the feasibility and acceptance of such a framework and identify areas for improvements. On completion of the engagement, we aim to publish our findings in later articles.