Objective and aims

The objective of the trial is to examine what impact the SPIRIT intervention has and how it works in the field. Our primary outcome measure is an objective assessment made by an expert panel, our secondary measures utilise self-informant or key informant reports and our process evaluation includes coded and descriptive accounts of the intervention delivery and participation made during observations of sessions, semistructured interviews and written participant feedback. We will examine these data across the components of the SPIRIT Action Framework (capacity, research engagement actions and use of research evidence); this will enable us to examine the impact of the SPIRIT intervention programme and to test the causal relationships hypothesised by the model. By capacity, we mean the extent to which the agency values research, has systems in place to use research and the skills of its staff. By research engagement actions, we mean the extent to which research is accessed, appraised and generated, and where interaction with researchers occurs, in relation to the development of a specific policy or programme document. By research use, we mean the extent to which research is sought and used in developing the policy or programme document, taking into account barriers and facilitators.

The specific aims are:

Primary outcome

1. To determine, using an objective measure, whether the SPIRIT intervention results in an increase in the extent to which research engagement actions are undertaken, and research is used in the development of policy and programme documents.

Secondary outcome

1. To determine, using the self-report of staff members, whether the SPIRIT intervention results in an increase in:

   • The capacity of health policy agencies to use research in terms of: (A) the value placed on the use of research evidence by individual policymakers and by the organisation; (B) the confidence of policymakers in undertaking research actions and using research and (C) the systems and tools the organisation has in place to support research use;

   • Research engagement actions;

   • The use of research.

Process evaluation: The aim of the process evaluation is to complement the outcome measures by exploring how and why the intervention worked or did not work in different contexts. This includes descriptions of:

1. The implementation of the intervention programme, including the delivery of the essential elements (fidelity);

2. Participation in and responses to the intervention (eg, how people interacted with the programme, how they evaluated different aspects of it and what sort of change they experienced, if any);

3. Contextual factors that might have affected the programme and responses to it (eg, agency priorities, practice norms, other training or organisational change initiatives, relationships with external bodies, legislative reform).

Cost analysis

1. To document the cost of delivering the intervention.

Methods

SPIRIT started in October 2012, and at the time of writing, approximately one-half of the total intervention period of 30 months has been completed.

Trial design

SPIRIT is a stepped wedge cluster intervention trial (figure 2) involving six agencies. Two agencies are randomly assigned to each step. This design is a variation on a cross-over design where each intervention unit is measured in the control and intervention phases, except that the step wedge design involves a cross-over in one direction only. The key feature of this design is that the intervention is rolled out sequentially over a number of time periods, such that all trial units have received the intervention by the end of the trial, although the order in which they receive it is determined at random.15 ,16 Outcome measures are obtained at the same time in all sites, at baseline and after implementation in each site. This design was selected in accordance with the UK Medical Research Council17 recommendations because the intervention is delivered at the level of the health policy agency (cluster) and recruiting and intervening with a large enough number of policy agencies to power a traditional randomised controlled trial was unfeasible due to the resource-intensive nature of SPIRIT and its status as a new, untested programme.

Participants

Intervention

Development of the intervention programme

The intervention programme is underpinned by the programme logic shown in figure 1.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

The programme logic for Supporting Policy In health with Research: an Intervention Trial (SPIRIT).

The intervention was designed to be appropriate for policy agencies, taking into account that they are sophisticated, complex organisations with skilled staff and diverse policy priorities. The design of the programme was strongly influenced by cognitive behavioural theory,18 ,19 system science,20 ,21 organisational change theory22–25 and adult learning theories.26 ,27 We used these approaches, together with information from interviews with policymakers and experience from existing programmes in Australia28–30 and internationally,5 ,6 ,10 ,14 to identify factors that have been associated with significant individual or organisational change. Telephone interviews were then conducted with nine senior Australian health policymakers to canvass their views about the kinds of strategies being considered for inclusion in the intervention. They endorsed our approach, indicating that the proposed change strategies were acceptable and would most likely make a difference in improving the use of research evidence in policymaking.

Components of the intervention and all three measures were piloted in a policy agency that was not part of SPIRIT prior to the start of the study. This pilot broadly confirmed that the proposed methods were acceptable and appropriate and allowed fine tuning of some aspects of the intervention.

Timing of recruitment, intervention delivery and follow-up

As shown in figure 2, the SPIRIT programme runs for 12 months in each health policy agency. The total measurement period is 30 months with measures being collected at the beginning of the study period, and then every 6 months thereafter in each agency. There are thus six measurement points in total. The first measurement point is a preintervention measure for all agencies and the final measurement point is 6 months after the completion of the SPIRIT programme in the last randomised pair of agencies.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2

SPIRIT design. LP, liaison person; SPIRIT, Supporting Policy In health with Research: an Intervention Trial.

Outcomes

Primary outcome

1. The primary outcome is an objective measure (SAGE, described below) of whether the SPIRIT intervention results in an increase in the extent to which research engagement actions are undertaken and research is used in the development of policy and programme documents.

Secondary outcomes

1. To determine, using the self-report of staff members (SEER and ORACLe, described below), whether the SPIRIT intervention results in an increase in:

   • The capacity of health policy agencies to use research in terms of: (A) the value placed on the use of research evidence by individual policymakers and by the organisation; (B) the confidence of policymakers in undertaking the research actions and using research; and (C) the systems and tools the organisation has in place;

   • Research engagement actions;

   • The use of research.

Process evaluation: The third set of outcomes was collected as part of a process evaluation and describes the implementation of SPIRIT in health policy agencies including data around:

1. SPIRIT components selected by each agency;

2. Participants’ engagement;

3. The fidelity of delivery;

4. The cost of delivery.

Outcome measurement

All outcome measures used in SPIRIT have been developed specifically for the trial as we were unable to identify any suitable measures in the literature. All measures were extensively piloted prior to the start of the trial and papers outlining their development and validation will be published in peer-reviewed journals. Table 2 provides a summary of the primary and secondary study outcomes and the accompanying outcome measurement tools.

1. Staff Assessment of enGagement with Evidence (SAGE) is an objective measure of the primary outcome—the extent to which agencies apply research findings in the development of the policy and programme documents they produce. An expert panel will be assembled to assign SAGE scores on the basis of interview data and document review. At each of the six data points, agencies are asked to provide the four policy documents finalised during the past 6 months that best represent the use of research in their policy and programme development work. The level of measurement is the policy document. Information about each document is collected via a structured qualitative interview with an individual who was heavily involved in the development of the document, nominated by the LP. The interview runs for approximately 1 h. A separate interview is conducted for each nominated document. The domains SAGE assesses are: (A) research engagement actions (accessing research, appraising research (for quality and relevance), generating new research or analysis and interacting with researchers)—each of these six dimensions receives a score from 0 to 9. Scores can be summed or reported separately; (B) outcome: research use (four types of research use are considered: instrumental, tactical, conceptual and imposed). Each of these four dimensions receives a score from 0 to 9. Scores can be summed or reported separately.

2. Seeking, Engaging with and Evaluating Research (SEER) measures individual policymakers’ capacities, research engagement actions and research use. SEER is administered via an online self-report survey to eligible policymakers identified by the LP. At each of the six measurement points, the LP from each agency provides a list of eligible policymakers to the research team.

Health policy or programme staff are regarded as eligible to complete SPIRIT measures if:

1. They write health policy documents or develop health programmes, or make or contribute significantly to policy decisions about health services, programmes or resourcing;

2. They are employed at a mid-level or higher in their agency;

3. They are over 18 years of age and consent to participate in the study.

Individual agency staff members are excluded if they are contractors and/or work across several of the participating agencies. All nominated policymakers are emailed an invitation to participate in the SPIRIT trial outcome measure in question along with standard participant information and consent forms. SEER measures (A) capacity (four subscales: predisposing factors related to individual values, perceptions of organisational values, perception of organisational systems and individual knowledge); (B) research use actions (accessing research, appraising, generating new research or analyses and interacting with researchers); and (C) research use (conceptual, instrumental, tactical and imposed research use) and extent of research use.

1. Organisational Research Access, Culture and Leadership (ORACLe) is used to assess each agency's capacity to use research findings. ORACLe data are collected via a structured qualitative interview with a senior staff member from each agency, nominated by the agency LP. The qualitative data are transcribed and used to assign scores to seven dimensions related to an organisation's capacity to use research. Two researchers separately score the responses (0=no, 1=yes—some or to a limited extent, 2=yes—very much so) of each participant to the 21 questions which make up the ORACLe scale. Inter-rater reliability has thus far been high (95%). Scores for each domain and a total score are then generated using an algorithm developed via a discrete choice experiment. This experiment harnessed the views of leaders in policy and knowledge exchange on the most important agency-level factors influencing the use of evidence.

ORACLe assesses agencies across the following seven domains: (A) processes that encourage or require the examination of research in policy and programme development; (B) tools and programmes to assist leaders of the organisation to actively support the use of research in policy and programme development; (C) strategies to provide staff with training in using evidence from research in policy and in maintaining these skills; (D) organisational strategies to help staff to access existing research findings; (E) methods to generate new research evidence to inform the organisation's work; (F) methods to ensure adequate evaluations of the organisation's policies and programmes; and (G) strategies to strengthen research relationships.

Process evaluation

A mixed methods process evaluation is conducted in parallel with the intervention in each of the six agencies. It collects data about, and analyses the interactions between, three domains:

1. The delivery of the intervention, including implementation of essential elements (fidelity). All intervention sessions are observed and audio recorded by process evaluation staff. Initial essential elements (ie, aspects of each intervention component that are considered critical to its success) are identified, and codes developed to monitor their delivery descriptively and, where possible, quantitatively. Essential elements and codes are tested in the early stages of the programme delivery, refined and further tested throughout the trial. Revised elements are applied retrospectively to earlier sessions to enable comparison across intervention sites.

2. Participation in and responses to the intervention: Participation attributes are recorded during session observations, and self-report evaluation forms are used to collect feedback on the relevance, applicability and value of each intervention session from the participants’ point of view, including whether they are likely to use knowledge/skills developed in the session. Following the intervention, five to seven semi-structured interviews are conducted in each site which explore participants’ responses to the programme, including changes in relation to individual and organisational capacity to use research, research engagement actions and actual research use.

3. Contextual factors that might have affected the intervention and responses to it: Session observations are used to collect data on participants’ views and experiences of research use in their work. Preintervention interviews with participants explore workplace priorities and culture, how they perceive their organisations’ support for research use, the role of research within the mix of other information and any other contextual factors that affect their use of research or may affect how they will engage with and respond to the intervention, including exposure to other research use drivers.

The process evaluation will complement the outcome measures by describing how and why the intervention worked or did not work, including why the same intervention strategies may have had different effects in different contexts.31–33 In addition, it will aid future programme improvement by identifying beneficial design and implementation strategies and suggesting contextually responsive programme adaptation for different recipients and settings. The full protocol for the SPIRIT process evaluation is described in detail elsewhere.

Cost analysis

A cost analysis is being conducted as part of SPIRIT because: the intervention will include non-market goods for which standardised cost estimates are less likely to exist. This cost analysis is conducted from the perspective of the intervention provider; detailed data on the resources used to deliver the intervention and their values are collected via interviews with central intervention staff, logging of the costs involved in hiring session facilitators, interviews with agency liaison people regarding the time taken to complete their SPIRIT tasks, recording the length of interviews, monitoring the time taken to complete the SEER online surveys and careful record keeping regarding the duration of sessions. Microcosting (a bottom-up approach methodology) is being used.34

Randomisation

The health policy agency is the unit of randomisation and intervention in the SPIRIT trial, with two sites randomised to start the intervention at each ‘step’. The six participating agencies were each assigned a code. The list of agency codes was sent to a biostatistician who was not involved in any other aspect of the trial, who assigned each agency code a computer-generated random number sequence and resorted them. The first two agencies in the list were allocated the October 2012 start date (step 1), the second two the April 2013 start date (step 2) and the final two the October 2013 start date (step 3).

Blinding

The SPIRIT staff and those investigators involved in delivering aspects of the intervention, by necessity, are not blind to allocation. All other investigators are blind to allocation, although analysis will need to know when sites start the intervention. They will not be informed which agencies are in the active intervention phase, and all questionnaire responses and interview records that they receive will be de-identified for individual participants. The agency allocation is concealed from those scoring SAGE and ORACLe to limit bias; this is particularly important since the intervention team cannot be blinded to the active intervention agencies.

Statistical methods

Analysis will be undertaken using generalised linear models with a link function and error distribution appropriate to the outcome measure. In each model, the intervention effect will be estimated as the difference between the postintervention and preintervention levels of the outcome after adjusting for time. The unit of analysis will be the agency for the SAGE and ORACLe outcome measures and the individual for the SEER outcome measure. Since there are too few clusters to employ either a linear mixed model or a generalised estimating equation (GEE) approach, all models will adjust for the clustering by using agency as a fixed effect.

For the study design outlined in figure 2 and with the assumption of an intraclass correlation coefficient of 0.01, a total sample size of 144 documents, 4 per site per time period, will give 89% power to detect a 1 SD difference in the mean SAGE total score at the 5% significance level.16

Data management and monitoring

Data for the SEER measure are entered online by participants. SEER is primarily multiple choice. In order to enhance data quality, the online survey has built in skip functions and participants are unable to progress from one page to the next until all questions are answered. Data from SAGE and ORACLe are transcribed (the scoring and checking procedure was described in the outcome measures section).

All data and documents are stored on password protected computers and on a network server accessible only to the researchers. Any paper documents are stored in a securely locked filing cabinet located at the Sax Institute, which has day-to-day management of CIPHER. The Sax Institute is a secure facility requiring a unique staff access code to enter; visitors are escorted on-site and are required to sign in.

A data monitoring committee was not required for this trial as the risk of harm to participants is negligible.

Ethics and dissemination

All participants provide electronic consent to participate in outcome measures via return email, and signed consent for process evaluation data collection. All participating agencies and individuals are free to decline to participate in any and all aspects of SPIRIT at any time with no explanation required (example participant information and consent forms in online supplementary appendix 1).

Participating agencies receive feedback on their results at regular intervals during the active phase of the intervention and again at the conclusion of the trial. The findings of this study will be disseminated more broadly through peer-reviewed publications (authorship determined by BMJ guidelines) and presentations at conferences and used to inform future strategies.