Methods and analysis

Conceptual framework

We take the perspective that quality of care can be considered a system property that is dependent on how the organisation providing care performs as a whole.34 Improving the quality of care is thus dependent on changing organisational behaviour, which implies changing the way clinicians interact and perform their clinical processes.35 36 Change in organisational behaviour is a complex social process that involves a range of different organisational activities.37 If external inspection is to contribute to improvement in the quality of care, it should have an impact on those activities involved in organisational change, here defined as any modification in organisational composition, structure or behaviour.38

We have previously conducted a systematic review of published research to identify the mechanisms of how external inspections can contribute to improving quality of care in health organisations.39 By combining empirical evidence and theoretical contributions, we found evidence to support that external inspections need to affect both organisational ideas and organisational change activities to improve the quality of care. Organisational ideas encompass theoretical constructs like organisational readiness for change, awareness of current practice and performance gaps and organisational acceptance that change is necessary.40 41 Organisational change activities refer to key activities involved in quality improvement like setting goals, planning and implementing improvement measures and evaluating effect of such measures.42 43

Figure 1 depicts our overall conceptual framework, how the elements of the framework relate to the different study aims and the corresponding data and effect measures. We suggest that inspections can affect organisational ideas and initiate change activities, which in turn can lead to organisational change. We collect qualitative data to explore how the inspections affect the involved organisations. Moreover, we suggest that organisational change and change in the process of detecting and treating patients with sepsis can contribute to improve the quality of care. To measure change in the process of detecting and treating sepsis, we collect data that reflect this process, for example, time to triage, time to initial assessment by physician and time to treatment with antibiotics. We refer to these data as process indicators, because they reflect how the process of detecting and treating sepsis is carried out. To measure change in the quality of care we use two outcome measures, length of hospital stay and 30-day mortality rate.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Conceptual study framework.

Data collection

Quantitative data

Figure 2 outlines the clusters and the data collection. For each inspection, we collect data at four different time points, referred to as P0, P1, P2 and P3. P0 is the baseline measurement for all hospitals before the Norwegian Board of Health Supervision announced the inspection campaign. The campaign is part of the regular planned inspection activities. These activities are transparent for the hospitals and are announced in advance, in this particular case 6 months before the first inspection. Due to practical reasons, the inspection teams collect data for P0 and P1 at the same time right before the inspection. Data for P0 are thus collected retrospectively, but it is predefined that the patients that will be included are the last patients with suspected sepsis that were admitted prior to 1 October 2015.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2

Illustration of clusters and data collection.

Data for P0 are collected from the time period right before the inspection campaign was announced. All hospitals know that they can be inspected. By collecting data before the campaign was announced, we can track changes throughout the inspection cycle and assess to what extent changes are implemented before the inspections are undertaken. P1 is the preinspection measurement and P2 and P3 are post-inspection measurements. The regional inspection teams collect data during the inspection and audits conducted 8 (P2) and 14 months (P3) following the initial inspection. These data serve two purposes. They are used to guide the judgements on whether the inspected hospitals comply with the requirements and to evaluate how inspections affect the clinical processes involved in diagnosing and treating sepsis.

We use a two-step approach to identify eligible patients who might have sepsis on arrival at the hospital. First, a record search of the National Patient Register (NPR) is carried out, using diagnostic codes for infections and sepsis. The NPR is a national common register of all patients and the treatment they have received in Norwegian hospitals. This record search will produce a list of patients with an identification number that will enable the inspection teams to access the corresponding patient records at inspected hospitals. Second, the inspection teams assess the individual patient records for eligibility. The inclusion criteria are clinically suspected infection and at least two systemic inflammatory response syndrome (SIRS) events, excluding elevated leucocytes.53 The inspection teams extract data from the included patient records.

Based on the literature, previous research and discussion with clinical experts, we identified relevant effect measures that reflect how hospitals handle patients admitted to the hospital with suspected sepsis.28 54 These indicators are displayed in box 3. In addition, data are collected for the following control variables: patient sex and age, sepsis with organ dysfunction and hospital size.

Box 3

Effect measures

Process measures

• Percentage of patients who have been triaged within 15 min of arrival in the emergency room.

• Percentage of patients who have been assessed by a medical doctor within the time frame set during triage.

• Percentage of patients in which vital signs have been evaluated within 30 min.

• Percentage of patients in which blood lactate has been measured within 30 min.

• Percentage of patients from which supplementary blood samples have been taken within 30 min.

• Percentage of patients from which a blood culture is taken prior to administration of antibiotics.

• Percentage of patients in which adequate supplementary investigation to detect locus of infection has been undertaken within 24 hours.

• Percentage of patients who have received antibiotics treatment within 1 hour of arrival in the emergency room.

• Percentage of patients who have received intravenous fluids within 30 min.

• Percentage of patients who have received oxygen therapy within 30 min.

• Percentage of patients for whom an adequate surveillance regime has been established.

• Percentage of patients who have been adequately discharged from the emergency room for further treatment in the hospital (written statement indicating patient status, treatment and further actions).

Outcome measures

• Hospital length of stay.

• 30-day mortality rate.

We use 30-day mortality rate as our key outcome measure, defined as the ratio of patients with sepsis who are dead within 30 days of hospital admittance. Mortality measures based on in-hospital deaths alone, can be misleading as indicators of hospital performance.55 Our measure also includes out-of-hospital deaths. Using the unique personal identification number provided to all citizens of Norway, we are able to link the patient record data with data from the National Registry to calculate the 30-day mortality rate. Thirty-day mortality rate is an established, national quality indicator for Norwegian hospitals,56 and this indicator has been shown to have better validity as a hospital performance measure than in-hospital mortality for selected medical conditions.57 Thirty-day mortality rate has also previously been used to assess effects of measures to improve care for patients with sepsis.58

We consider the percentage of patients with organ dysfunction who have received antibiotic treatment within 1 hour as the key measure, because it is associated with our outcome measure, 30-day mortality rate.27 28 Furthermore, we suggest that early triage and early assessment by medical doctor are key activities to secure timely treatment with antibiotics and that the corresponding process measures consequently are important too.

The power calculations were performed using the stepped-wedge function in Stata/IC V.14.0 (StataCorp LLC) software for Windows, developed by Hemming and Girling.59 The statistical power in a stepped-wedge design depends on the total number of intervention sites, the total number of data collection points for each intervention site, the number of patient records included at each data collection point, the correlation between clustered observations on the same hospital (intracluster correlation) and the implementation period.45 We based our calculations on 24 intervention sites, 4 data collection points and 33 patient records per collection point at each intervention site. As the intracluster correlation may vary between samples and between process and outcome measurements, it is not straightforward to specify an intracluster correlation in advanced. In addition, we could not find any estimated intracluster correlation in previous trials of patients with sepsis. Consequently, we chose an intracluster correlation of 0.05, which is in line with that estimated for several patient outcomes in a cluster randomised trials of heart failure patients.60 Type I and II errors were assumed to be 0.05 and 0.20, respectively.

Because assessment of patient records must be done manually, data collection is resource demanding. Thus, we must balance the design and power of the study against the available resources, a manageable amount of data collection and likely detectable changes in the key effect measures. For the process data, we have powered the study according to the key clinical process indicator, antibiotic treatment within 1 hour of arrival at the hospital. To detect an absolute improvement of 15 percentage points, for example, from 50% to 65% of patients receive antibiotic treatment within 1 hour of arrival, we must include a minimum of 2376 patient records. We intend to include n=3168 patients (24 hospitals×4 time points×33 patients); we assert that this sample is large enough to examine the other relevant process indicators described in box 3.

To reach sufficient statistical power to detect a significant change in patient outcome measures, such as 30-day mortality rate, we need to include more than the 33 patients that we include to detect changes in process measures per each time point. We have powered the study to detect a reduction from 15% to 11.5% in the mortality rate by including 60 patients for each hospital at each time point. The total intended number of included patients for outcome measures is n=5760 patients (24 hospitals×4 time points×60 patients).

Qualitative data

We conduct focus group interviews based on an interview guide. We developed the interview guide based on theory about practices involved in implementing organisational change61 along with our model for how inspections can contribute to improving quality of care by affecting organisational change activities and ideas.39 The key research questions that will be further explored in the qualitative study are displayed in box 4.

Box 4

Key research questions of the qualitative study

• How do inspection teams plan, conduct, follow-up, and finalise the inspections?

• How do the inspected organisations prepare for inspections?

• Do inspections contribute to creating awareness about current performance and possible performance gaps in the clinical system for delivering care to patients with sepsis?

• Do inspections contribute to engaging leaders and clinicians in improvement of their work?

• Do hospitals initiate improvement activities aimed at enhancing the quality of sepsis care prior to inspections?

• Do inspections affect organisational ideas, for example, understanding of the clinical system and commitment to change?

• Do clinicians and leaders reflect on the performance of their clinical system before and after the inspections?

• What kind of change activities do hospitals initiate following the inspections, and how do these affect the quality of care delivered to patients with sepsis?

• What is the impact of other contextual factors on the improvement process?

• How do follow-up audits of patient records affect the change processes?

We intend to include a strategic sample of six cases; we assert that such a data sample can provide sufficiently rich data about the phenomenon we intend to study.62

Strengths, potential limitations and biases

Regulatory measures within healthcare seem to be increasing, and the burden and effect of such measures are being questioned.69 71 Previous studies have evaluated different approaches to regulation and surveillance in healthcare, but we have limited knowledge about the actual effect of such measures on the quality of care.72 73 External inspections are core components of regulatory regimes and are frequently used within healthcare.1 2 It is therefore of importance to gather more knowledge about the effects of this regulatory measure and how inspections might contribute to improving healthcare quality. Few studies have used an experimental design to assess the effects of external inspections.10 We suggest that the reason for this is that external inspections represent contemporary events involving a whole range of autonomous actors in society, which makes it challenging to apply an experimental design. In our case, it involves the Norwegian Board of Health Supervision, 18 autonomous county governors, 24 hospitals and the NPR. The main strength of our study is the fact that we have been able to persuade all involved actors to cooperate and commit to conducting the inspections using a stepped-wedge design.

The stepped-wedge design enables us to track changes in outcome measures for sepsis detection and treatment over time. The changes that we might observe in the outcome measures are not necessarily attributable to the inspections alone. There can be other factors beside the inspections that can affect sepsis detection and treatment during the study period. Our qualitative data can help identifying such factors and provide insight into how they might interact with the inspections. By combining findings from the qualitative and quantitative data, we assert that we can assess how sepsis detection and treatment develops over time and substantiate how inspections along with other factors can affect the development.

Our study is based on an overall framework suggesting that the inspections need to affect organisational ideas and activities to facilitate change in organisational behaviour and thereby improve quality of care for patients with sepsis (figure 1). Moreover, we have developed a more detailed theory of change for the inspections, suggesting how they can contribute to affect organisational change. We collect quantitative data that indicate whether organisational behaviour and the quality of care improve after the inspections. By combining these findings with our qualitative data that provide insight into how the inspections affect organisational change, we can test our theories about how inspections can contribute to improve the quality of care. We suggest that our study can contribute to build theory about how inspections can improve the quality of care and thus have relevance for inspections covering other topics.

Initially, we had intended to use data from the NPR to calculate the 30-day mortality rate and hospital length of stay. After we had begun planning our study and had received ethical approval, a new definition for sepsis and septic shock was issued.74 Patients with sepsis are identified in the NPR using diagnosis codes of the International Classification of Diseases, 10th revision. Previous research has shown that coding practice for sepsis varies between hospitals and can change over time.75 76 We assert that the new sepsis definition will affect how Norwegian hospitals code sepsis; therefore, we cannot use available routine data to calculate our outcome measures. Instead, we must use a more labour-intensive approach and manually scan all patient records that we include.

Though more time consuming, including patients based on manual reviews of patient records, instead of sepsis codes, is strengthening our study. The patients in this study are recruited from a population identified through a standardised search in the NPR that includes sepsis codes and the most commonly used infection codes. Patients are included based on SIRS criteria. We assert that a change in coding practice owing to the new sepsis definition will not alter the patient population identified by our standardised search. The new sepsis definition is narrower; it is therefore likely that fewer patients will be coded with sepsis. Patients that do not fulfil the criteria for sepsis under the new definition will still be assigned an infection code and will therefore be encompassed in the patient population identified by our standardised search.

The inspections in our study are contemporary and transparent events, thus it is not possible to mask who is exposed to the intervention. Nor is it possible to mask the data collection. Because there is no available routine date about sepsis detection and treatment, such data are collected as part of the intervention in our study. The inspections teams access the patient records manually, and they will therefore know which hospital the patients belong to and whether the patient was admitted before or after the inspection. The fact that data are collected as part of the intervention can be viewed as a limitation. Doing data collection during the inspection is however standard procedure, and thus not atypical for the inspections in our study.48 Given the nature of the intervention—that is collecting data, reporting and giving recommendations—what we are in fact measuring is, at least in part, the effect of data collection.

To enhance data quality and reduce bias, the Norwegian Board of Health Supervision has developed a framework and detailed criteria on which data collection is based. Moreover, data collection is done by experienced inspectors with particular medical expertise, and the whole process is overseen by the external medical experts and the leaders of the inspection teams. The data collection is transparent for the inspected hospital. They can verify all the collected data and give feedback if they encounter missing data or data that has been misinterpreted.

The patients included in this study have a suspected serious infection that potentially requires rapid treatment. The process data that we collect addresses sepsis detection and treatment in the initial phase of the hospital stay, for example, the time to triage, assessment by a physician and antibiotics treatment. Such process measures are still relevant for the patient population included in our study, regardless of whether they fulfil the criteria for sepsis using the new definition. Because our main objective is to track changes over time in the quality of care delivered to a defined patient population, we assert that our study design and process data are still valid and relevant despite the introduction of a new definition of sepsis.

We had to balance the power of the study against the available resources and a manageable amount of data collection. We designed and powered the study to detect realistic changes in our outcome and process measures. However, there is an inherent risk that the study can become underpowered to detect changes in our effect measures. A potential challenge is information missing in the patient records. Lack of documentation itself can be an important finding because this can be associated with poorer patient outcomes.77 If information regarding process measures is missing in the patient records, it will affect the power of the study, which can potentially become underpowered. If there is an association between substandard care and lack of documentation in the patient records, our preinspection data may be biased. Poor care will only be indicated by missing data but will not be supported by actual empirical data. Following inspection, it may be easier to improve the documentation practice within a hospital than actually improving the delivery of care. Thus, there is a risk that the postintervention data will be biased because they include substandard care that would not have been documented prior to inspection.

Postinspection data are collected 8 and 14 months after inspection. There is a delay of about 2 months after the on-site visit before the hospitals receive their final report. Implementing true change in a clinical system can be time consuming. To give the hospital time to plan and implement changes, we schedule the first follow-up audit 8 months after the initial site visit. We assert that the hospitals have begun to implement change at this point. The feedback after the first audit will not be as extensive as that in the report following the site visit, and the feedback will be provided to hospitals without much delay. The hospitals can use this feedback to adapt and reiterate their ongoing improvement activities. We therefore schedule the second follow-up audit 6 months after the first.

A likely challenge in our study is possible contamination of the control group, that is, hospitals that have not yet received the intervention. All hospitals in Norway are aware of the nationwide inspection campaign on sepsis treatment now that the inspections are under way. There is a possibility that hospitals that have not yet been inspected will begin to make improvements prior to inspection. We therefore collect data at P0 to obtain a baseline measurement for all hospitals before announcing the inspection campaign. Collecting quantitative data from P0, P1, P2 and P3 as well as qualitative data will enable investigation of the extent to which hospitals implement changes before the inspections and explore how change processes unfold in the inspected hospitals.

Factors like socioeconomic status and comorbidity can affect the outcomes. We do not have access to such data and can therefore not adjust for these factors. In a stepped-wedge design, all sites are exposed for the intervention and we compare change in the effect measures before and after the intervention. We collect the data in a standardised way, and the intervention itself should not affect which patients that are admitted to the hospitals. Consequently, we have no reason to believe that confounding factors should be unevenly distributed in the study population before and after the intervention.

The size of the included hospitals differ. The order of the inspections are randomised in clusters of four hospitals, and all the clusters include hospitals with different sizes. To account for intracluster correlation, we will use generalised estimating equations methodology,65 and we will include hospital size as a covariate in our analytic models. Due to Norwegian topography with long travel distances, patients with suspected sepsis are typically sent to the nearest acute hospital for initial diagnosis and treatment. In some cases, patients with septic shock can be transferred to a larger hospital later. We know the number of patients with organ dysfunction admitted to the various hospitals and can adjust for this in our analysis. We do however not have access to data about comorbidity for the included patients and can therefore not fully adjust for case mix differences between the included hospitals. Our process measures cover the initial steps of the diagnostic and treatment process, which is done in all hospitals irrespectively of size. We compare changes in the effect measures before and after the intervention, and the intervention itself should not affect which patients that are admitted to the different hospitals. Consequently, we have no reason to believe that case mix differences between the inspected hospitals should change before and after the intervention.

External inspections can have intended and unintended outcomes.78 The aim of our study is to assess the effects of inspection on the intended outcome, that is, improved care for patients admitted to the hospital with suspected sepsis. However, there can be intended and unintended outcomes of inspections that this study does not explicitly address. One such intended outcome is to create legal safeguarding. Governmental or other public institutions often perform external inspections as one of many measures within a larger regulatory system that aims to ensure delivery of services according to certain requirements. An independent system of external control can contribute to confidence that certain services meet relevant quality requirements, both for the recipients of said services and for the public in general. By having an independent controlling mechanism, patients need not examine and control the safety and quality of services themselves.79 Inspections can thus be regarded as a means of legally safeguarding users and the public. This is an intangible outcome that is difficult to quantify and evaluate, and it will not be addressed in our study.

Another outcome that we only partially address is the possibility of goal displacement. Whatever is being inspected tends to receive attention.80 81 An unintended effect of inspections can therefore be that members of the inspected organisation become so preoccupied with responding to an ongoing inspection that they neglect more important improvement efforts initiated by the organisation itself. Furthermore, inspections can influence how the organisation prioritises its resources,22 81 82 and there is an inherent danger that the inspection can draw resources away from areas in greater need of resources. Our qualitative data can provide insight into these unintended consequences of inspection to some extent, but we will not be able to quantify these effects.

A basic precondition for external inspections is that the inspected organisation is accountable for implementing necessary changes following an inspection. The effect of an inspection thus depends on delivery of the inspection and the inspected organisation’s capacity to implement change.11 83 The latter is in turn dependent on a range of factors, for example, engagement by leaders and workers, organisational culture, available resources and knowledge improvement.42 84 85 We assert that the outcome measures used in this study reflect both the inspection delivery and the improvement capacity of the inspected organisation. Delivery of the inspection cannot be evaluated in isolation from the inspected organisation’s ability to effect change. Still, we argue that a lack of improvement following an inspection can primarily be caused by the inspected organisation’s lack of capacity to implement change and not by the inspection itself. Our qualitative data can provide insight into the impact of inspection on the organisation and the organisation’s capacity to implement change. To a certain extent, we can therefore interpret the absence of change as either being primarily a consequence of weaknesses in the delivery of inspection or owing to the inspected organisation lacking the capacity for improvement. However, one limitation of our study is that we cannot quantitatively distinguish between these two causes.

There is a striking mismatch between the widespread use of external inspections in healthcare and the paucity of evidence of their effect.10 We combine quantitative and qualitative data in an innovative way that can contribute to shed light on how inspections affect the quality of care delivered to patients with sepsis. Despite the limitation of our study, we argue that it is robust enough to provide new and needed knowledge about the effects of external inspections.

Ethics and dissemination

The Regional Ethics Committee of Norway Nord (REC) reviewed the study (Identifier: 2015/2195). The REC considered the study to be health service research and not a clinical trial because there is no intervention applied directly to patients. According to Norwegian legislation, the REC is mandated to grant exemptions from the requirement of confidentiality for this kind of research. The REC considers this study to be of essential interest for society and that the welfare and integrity of patients is attended to. Data will be deleted at the end of the project. On this background, the REC has granted an exemption from the requirement of consent and confidentiality for the data used in this study. The Norwegian Data Protection Authority has reviewed and approved the study (Identifier: 15/01559). The study was registered with ClinicalTrials.gov in April 2016 (Identifier: NCT02747121). The study results will be reported in international peer-reviewed journals.