Design

A realistic evaluation framework will be used, which is a theory-driven framework that aims to explore ‘what works, for whom, in what context and to what extent’.14 It focuses on the ways that the context (ie, settings) of a social intervention interacts with a mechanism of action (ie, underlying processes or structures) to produce outcomes. Multiple methods and data sources are used to help provide a comprehensive understanding of what mechanisms for change are triggered by an intervention and how they impact on the existing social processes sustaining the behaviour or circumstances that are being targeted for change. The evaluation of the SAFE programme will last 3 years. The quantitative arm of the evaluation will focus on outcome data from all participating wards; the qualitative arm of the evaluation will focus on a detailed process evaluation of the ward-level implementation of huddles in a subset of participating wards. Figure 1 shows the SAFE programme theory of change, which the evaluation aims to assess.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

The SAFE programme theory of change.

Study design and participants

The SAFE quantitative outcome evaluation draws on data collected on all participating wards from the 12 sites implementing SAFE (six specialist children's hospitals and six district general (community) hospitals). All wards will be collecting safety outcome indicators (see table 1). Within each ward, a sample of around 30 staff members will be asked to complete safety culture surveys at 6-month intervals. The sample will be selected based on the staff lists provided by each ward to ensure representation of medical staff, nursing staff, healthcare assistants and allied health professionals. We estimate the overall number of surveys collected across wards and across time points to be ∼1000. As part of implementation, all parents of children staying on the participating wards (and children themselves where appropriate) are also being asked to complete experience of care surveys, which cover general experience of care and perceptions of safety throughout the implementation period of the SAFE programme (see table 1). Ethnographic field notes will also be taken as part of quarterly implementation site visits to all 12 participating hospitals.

The SAFE qualitative process evaluation draws on a subsample of two district general hospitals and two specialist children's hospitals. In order to maximise the study of context, these four sites will be sampled to allow variation in geographical region (north and south of the country), ward specialism (High Dependency Unit (HDU) or general paediatric inpatient) and ward size (small and large). The qualitative arm of the study is comprised of three components. First, interviews with five to six parents and children (aged 8 years or older, children younger than 8 will be excluded from the sample) from each of the four sites will be conducted. The parents (or young person, if they are aged 16 or older) will complete an expression of interest form on the ward at the point of their child's discharge, if they are interested in potentially taking part in an interview. The research team will collect the completed expression of interest forms from the sites and will then contact each parent to try and arrange their interview. This sample size is anticipated to be feasible for the research team in terms of their capacity, and also pragmatic in terms of potential difficulties in recruiting parents for interview after their child has been discharged from the ward.

Second, interviews will be conducted with staff, with 15–20 interviews conducted at each site at Time 1 (start of SAFE implementation); a smaller sample of 10–15 staff will be interviewed at each site at Time 2 (6 months after baseline) and Time 3 (12 months after baseline). These sample sizes are anticipated to be feasible for the staff members on the wards in terms of their capacity to take part in interviews (potentially across all three time points) during their shifts.

Third, at these three time points, huddles will be observed and audio recorded. A non-participant huddle Observation Tool (HOT) (J Edbrooke-Childs, J Hayes, E Sharples, et al. Development of the Huddle Observation Tool (HOT) for structured case management discussions to improve situation awareness in pediatric wards. in preparation) (see Measures) was designed to describe the core components of huddles in practice, including the structure of the huddle, the extent of disruptions to the huddle from the external environment, the collaborative culture of the huddle and whether there were clear opportunities to identify risks and discuss concrete plans to mitigate these risks.

Measures

Safety outcomes. To capture safety outcomes, an agreed suite or battery of indicators has been agreed across sites, which is shown in table 2. This suite was produced to allow different study sites to collect relevant data.

Ward experience of care and safety. To measure ward experience of care and safety, the 10-item ward experience of care and safety questionnaire will be used, drawing on existing measures. The questionnaire captures two domains. First, experience of care received, drawing on a subset of five items from the Experience of Care Questionnaire15 (eg, ‘I feel that the people who have seen my child listened to me’). Although a child self-report version will be used, as the majority of children on participating wards are under the age of 8 (so not eligible to complete the questionnaire), it is likely that most respondents will be parents. Children and parents will respond to items on a three-point scale from 1 (certainly true) to 3 (not true). Second, parents’ experience of safety is assessed via the overall perception of safety subscale of the Hospital Survey on Patient Safety Culture comprising four items16 (eg, ‘The procedures and systems in this unit are good at preventing errors from happening’). Parents will respond to items on a five-point scale from 1 (strongly disagree) to 5 (strongly agree). Both measures have demonstrated reliability and validity in previous studies.15 ,17 ,18

Staff Safety culture. To measures staff safety attitudes, an adapted version of the Safety Climate Survey (SCS)19 will be used. The survey used in the present study includes 19 items from the original instrument20 with an additional two items from the Safety Attitudes Questionnaire.20 The SCS measures staff perceptions of safety in their wards and perceived management commitment to safety. Example items include ‘The senior leaders in my hospital listen to me and care about my concerns’ and ‘I am encouraged by my colleagues to report any safety concerns I may have’. The SCS is a widely used measure of safety climate recommended by the Institute for Healthcare Improvement and has demonstrated reliability and validity in previous studies.21

Huddle Observation Tool. This instrument has been developed specifically to capture the team processes that take place during huddles. It comprises four domains, each of which has one item rated on a five-point scale, from 1 (strongly disagree) to 5 (strongly agree), with free text response sections for notes: risk management (ie, ‘Were there opportunities to identify risks and come up with concrete plans for these risks?’), structure (ie, ‘Did the huddle have a clear structure?’), collaborative culture (ie, ‘Did everyone have the opportunity to contribute and were all points of view respected?’) and ‘environment’ (ie, ‘Was the huddle free from distractions?’). Huddles observed using the HOT will also be audio recorded. The HOT has being developed specifically for this evaluation and the data collected through completing the tool during research team site visits. Inter-rater reliability for the tool will be established as part of the evaluation.

Procedure and analytic strategy

Realistic evaluation involves the examination and testing of a theory of change.14 Our procedure for examining each of the four components of the SAFE programme theory of change—input, mechanisms, context and outcomes—is described below. The overarching analytic strategy will involve synthesis across the different strands of data collection to enable triangulation.22

Input. The SAFE evaluation aims to describe implementation at the ward level and how this has been experienced through quantitative and qualitative data capture. General characteristics of implementation will be recorded by each ward (eg, number of huddles per day, attendance of huddles) and reported using descriptive statistics. These data will be complemented with observational assessments captured using the HOT. Audio recordings from all Time 1 observations will be subjected to conversation analysis23 to capture early emerging practice in terms of how huddles take place, what core elements are included, how concerns are conveyed and who leads and steers discussions. The conversation analysis will be conducted to explore what huddle implementation looks like at each site, and to examine cross-site differences and similarities.

Qualitative interviews with ward staff will also provide further information about what implementation of SAFE looks like across a range of wards. All interviews will be transcribed verbatim. The framework approach24 will initially be employed to manage the qualitative interview data set. This will involve sorting extracts from the transcripts into a framework of predefined, ‘top-down’ categories (context, mechanisms, impact and barriers and facilitators). The emergence of ‘bottom-up’ themes within each of these predefined categories will then be explored using thematic analysis.25 Thus, overall, a combination of inductive and deductive analyses will be used. Our specific process will be as follows: two researchers will code the interview transcripts to all of the predefined categories in the top-down framework. One of the researchers will then conduct a thematic analysis of the content coded to a selected category (depending on the research question) and will develop a preliminary thematic framework (bottom-up themes) of the content coded within this category. The second researcher will then recode a subset of the content coded to this category to test the first researcher's preliminary thematic framework. The two researchers will discuss any discrepancies in coding and refine the thematic framework accordingly. In mid and late implementation, exploration of ‘input’ will consider (1) reach, as evidenced by observations of who is discussed in huddles and patients and parents’ awareness and experience of huddles; (2) adoption by ward staff on participating wards, as evidenced through records of huddle attendees from the HOT and staff reports of involvement in interviews; and (3) implementation consistency and adaptation, and the reliable maintenance of practices, based on huddles observations, field notes and staff-reported implementation.26

Mechanisms. The mechanisms being explored are changes in safety culture, and improved situational awareness, captured using the Safety Climate Survey19 (see Measures) and through interviews with ward staff. Descriptive and inferential statistics will be used to examine change over time within wards and differences between wards. Mediation tests may also be conducted, with staff safety culture mediating the potential relationship between implementation of SAFE and changes in safety outcomes. Additional factors of particular relevance, such as increased awareness and better anticipation of risk, will be captured using semistructured interviews with ward staff.

Context. The context includes the range of different conditions within each hospital or ward hypothesised to determine the extent to which input and mechanisms affect outcomes. Each site is different in the type of ward and the acuity of patients. Contextual data will primarily be derived from the routinely collected data from sites, for example, hospital and type, staffing and patient throughput. Staff interviews and interviews with parents and children, in combination with ethnographic field notes taken during implementation site visits, will provide further contextual information.

Outcomes. Change in patient safety outcomes, in terms of incidents of harm, is the high-level outcome of SAFE. Analysis of run charts spanning pre and postimplementation time periods will be carried out to identify any potential patterns (eg, shifts, trends and runs)27 in incidents of harm that might be associated with SAFE implementation. Analysis will be considered at individual ward level and as an aggregate across wards. Similarly, patient-reported and parent-reported measures will be used for group-level comparisons of perceived care between wards within time points, individual wards over time and aggregated across wards over time. Changes in these outcomes, identified based on ward-level run charts, will be triangulated with information about mechanisms of change drawn from quantitative and qualitative data to identify those that appear to be most strongly associated with improvements in safety outcomes.

The qualitative interviews with staff members will capture what the huddle looks like at each site, the impact of huddles and other safety improvement techniques associated with SAFE on the ward environment and on patient safety and barriers and facilitators to implementation and impact from staff members’ perspectives. The qualitative interviews with patients and parents will seek to capture their experiences of care and perceptions of safety on the wards, as well as their perceptions of and opinions on huddles.