Article Text

Does the quality and outcomes framework reduce psychiatric admissions in people with serious mental illness? A regression analysis
  1. Nils Gutacker1,
  2. Anne R Mason1,
  3. Tony Kendrick2,
  4. Maria Goddard1,
  5. Hugh Gravelle1,
  6. Simon Gilbody3,
  7. Lauren Aylott4,
  8. June Wainwright4,
  9. Rowena Jacobs1
  1. 1Centre for Health Economics, University of York, York, UK
  2. 2Primary Care and Population Sciences, University of Southampton, Aldermoor Health Centre, Southampton, UK
  3. 3Department of Health Sciences, University of York, York, UK
  4. 4Service User, UK
  1. Correspondence to Dr Rowena Jacobs; rowena.jacobs{at}york.ac.uk

Abstract

Background The Quality and Outcomes Framework (QOF) incentivises general practices in England to provide proactive care for people with serious mental illness (SMI) including schizophrenia, bipolar disorder and other psychoses. Better proactive primary care may reduce the risk of psychiatric admissions to hospital, but this has never been tested empirically.

Methods The QOF data set included 8234 general practices in England from 2006/2007 to 2010/2011. Rates of hospital admissions with primary diagnoses of SMI or bipolar disorder were estimated from national routine hospital data and aggregated to practice level. Poisson regression was used to analyse associations.

Results Practices with higher achievement on the annual review for SMI patients (MH9), or that performed better on either of the two lithium indicators for bipolar patients (MH4 or MH5), had more psychiatric admissions. An additional 1% in achievement rates for MH9 was associated with an average increase in the annual practice admission rate of 0.19% (95% CI 0.10% to 0.28%) or 0.007 patients (95% CI 0.003 to 0.01).

Conclusions The positive association was contrary to expectation, but there are several possible explanations: better quality primary care may identify unmet need for secondary care; higher QOF achievement may not prevent the need for secondary care; individuals may receive their QOF checks postdischarge rather than prior to admission; individuals with more severe SMI may be more likely to be registered with practices with better QOF performance; and QOF may be a poor measure of the quality of care for people with SMI.

  • MENTAL HEALTH
  • PRIMARY CARE
  • HEALTH ECONOMICS

This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/

Statistics from Altmetric.com

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Strengths and limitations of this study

  • This is the first study to investigate the relationship between general practitioner (GP) practice quality, as measured by four mental health Quality and Outcomes Framework (QOF) indicators, and psychiatric admissions in the English NHS.

  • The data covered all practices in England and the results were found to be representative.

  • The study used a consistent set of primary care quality indicators over the entire study period and employed longitudinal panel data estimation, therefore improving the robustness of results compared to previous research.

  • A comprehensive set of GP practice and patient population characteristics were included in the models.

  • An array of sensitivity analyses was undertaken and results were found to be robust.

  • Aggregate practice-level data were used to examine the association between QOF and admissions and so we cannot be sure whether admitted patients had received a QOF review or had been exception reported.

  • These are observational data and our results may be affected by unobserved confounders.

Introduction

The quality of care of people with mental health problems is of international concern.1 ,2 While primary care is central to the provision of mental healthcare in England, there is increasing focus on the interface between primary and secondary care, and the potential for better quality primary care to reduce avoidable hospital admissions and contain health expenditures.3 The role of general practitioners' (GP) care in preventing admissions is a matter of continuing debate,4 ,5 including the mechanisms through which this might operate,6 but the tough economic climate implies this will be a subject of focused attention.7

Serious mental illness (SMI) includes schizophrenia, bipolar disorder and psychoses with considerable disability, prevalence8–10 and an estimated economic burden of £14 billion.11 In the UK, around 30% of people with SMI are treated solely by primary care clinicians12 and compared with those without mental health problems, people with SMI are in contact with primary care services for a longer cumulative time.13 ,14 The pay-for-performance scheme in primary care—the Quality and Outcomes Framework (QOF)—includes targets to incentivise GPs to improve the quality of care for people with SMI.15

Although not an explicit aim of the QOF, several studies have examined whether better quality primary care can reduce hospital admissions. Analyses of relationships between QOF performance for coronary heart disease, asthma and chronic obstructive pulmonary disorder and hospital admissions found no effects,16–18 but better performance for diabetes19 ,20 and stroke21 had a small negative association with emergency admissions. We hypothesise that this may be plausible for SMI. No previous study has tested this association for people with SMI. We investigated whether higher achievement on the SMI QOF indicators was associated with fewer psychiatric admissions for people with SMI.

Methods

We carried out a retrospective analysis of routine data at GP practice level, estimating the effect of four QOF indicators (table 1) on psychiatric admissions to hospital using random effects Poisson regression.

Table 1

Overview of QOF indicators for SMI used in the analyses

Sample

Our data set included all GP practices in the English NHS between April 2006 and March 2011. We chose this period because the QOF definition of SMI was constant and there was a stable set of mental health indicators. We excluded practices with fewer than 1000 registered patients within a year as unrepresentative of the way in which primary care is normally organised. Practices were also excluded if the registered number of patients with SMI was below 5 as their QOF achievement was prone to large variations over time. However, we did not apply this exclusion to the analysis of bipolar disorder because registered numbers of patients with bipolar disorder were low for most practices. We excluded practices reporting inconsistent numbers of patients with SMI or bipolar disorder across indicators within a year and practices where the registered number was fewer than the number admitted to hospital. Finally, we excluded all admissions for patients who changed practice within a year as it was unclear which practice affected the need for inpatient care.

Data sources

We linked administrative data sets including the General and Personal Medical Statistics (GMS) data, the Attribution Data Set (ADS), the QOF data set and the annual GP Patient Survey data (table 2) using unique practice-year identifiers. Census (2001) data from the Office for National Statistics, measured at small-area level (ie, Lower Super Output Areas), were linked to practices on the basis of their practice population distribution as reported annually in ADS. Annual admission rates were calculated from Hospital Episodes Statistics (HES). All data sources are reported in the online supplementary appendix table 1.

Table 2

Covariates included in the regression models: descriptive statistics

Hospital admissions

HES records diagnoses using the International Classification of Diseases (ICD) 10 classification system, whereas the QOF inclusion criteria are based on Read codes used in primary care. To identify the relevant population, we used the Health and Social Care Information Centre cross-mapping from Read to ICD-10. Psychiatric admissions were defined as hospital inpatient episodes in patients aged 18 years or over with a main diagnosis of SMI (ICD-10 codes: F20-F31); for the subset of bipolar disorder admissions, defining main diagnoses were ICD-10 codes F30-F31.

Just under one-quarter (23.3%) of psychiatric admissions in our data were coded as elective. However, some providers class all mental health admissions as emergencies; hence, elective and emergency psychiatric admissions cannot be consistently distinguished in routine data due to variation in coding.22 On the advice of our study steering group (including policy experts, clinicians and people with SMI) we therefore pooled all admissions irrespective of how they were coded by the provider, and conducted sensitivity analyses.

Measures of practice quality

During our study period, the QOF mental health domain included five indicators to incentivise proactive disease management for a population where low adherence to medication, or drug levels outside of a therapeutic range, may lead to relapse and hospitalisation (table 1).23 Two of these indicators (MH6 and MH9) apply to all registered patients with a diagnosis of SMI, whereas the two lithium indicators (MH4 and MH5) apply only to patients with bipolar disorder. Our analyses excluded indicator MH7, which encourages follow-up of patients with SMI failing to attend their annual review, because practices that review all eligible patients cannot score on MH7 and so would not contribute to the analysis. We selected indicators on the basis of consistency over time within our study period (see online supplementary appendix table 2).

Under the QOF, practices may ‘exception report’ patients,24 that is, remove inappropriate patients from the denominator used to calculate achievement. While exception reporting may reflect good-quality care, for example, because patients are deemed unsuitable for clinical reasons, it could also reflect ‘gaming’ by GPs, who can increase the number of points they earn by reducing the eligible population inappropriately.25

The set of patients for each QOF indicator (all patients with SMI for MH6 and MH9, or all patients with bipolar disorder for MH4 and MH5) were divided into three mutually exclusive categories: those for whom the indicator was achieved (A), those who were exception reported (E), and those for whom the indicator was not achieved (NA). For each indicator, we calculated two measures of practice performance. The QOF incentive regime rewards GPs on the basis of reported achievement:Embedded Image 1which is set to 0 if all patients are exception reported.

Given that the appropriate level of exception reporting is uncertain and we cannot distinguish admissions for patients who were exception reported from those who were not, we followed Kontopantelis et al26 in using:Embedded Image 2as our preferred practice performance measure in our main analysis. We used three sensitivity analyses to investigate the effect of including exceptions in calculating achievement. First, proportions of exception-reported patients included in the denominator were successively increased by 10 percentage points from 0% (equation 1) to 100% (equation 2) to identify potential switching points (ie, the levels at which the sign and statistical significance of the estimated coefficient changed). Second, we stratified practices into tercile groups by their level of exception reporting and interacted the respective achievement rate with the population exception rate:Embedded Image 3This sensitivity analysis was restricted to indicators MH6 and MH9 as there was insufficient variation in exception rates for the bipolar indicators to classify them into terciles. Finally, we also tested a model including the population exception rate as a separate covariate.

Covariates

Data on hospital admissions and practice quality were linked to GP practice characteristics, their patient population characteristics and population characteristics including deprivation and other potential confounders recorded at small-area level (table 2). We also controlled for measures of access to care and modelled Primary Care Trust fixed effects to account for differences in resourcing of crisis resolution and home treatment teams providing alternative home care in emergencies and playing a ‘gatekeeping’ role in hospital admissions.27–29 Year indicator variables were used to account for temporal trends. In order to reduce potential bias from unobserved practice-specific confounders, we included presample baseline admission numbers per practice (averaging financial years 2003/2004 to 2005/2006).30

Analysis

Random effects Poisson regression models were estimated to relate the number of psychiatric admissions per practice to its QOF achievement, conditioning on potential confounding factors and a normally distributed GP practice random effect with zero mean and constant variance.31 ,32 The numbers of practice-registered patients with SMI or bipolar disorder were used as exposure terms. We obtained cluster-robust Huber-White SEs to account for potential overdispersion.31 Coefficient estimates are presented as incidence rate ratios (IRR) with 95% CIs, so that a coefficient less than 1 indicates that the variable reduced admissions and vice versa. We also calculated the average effect of a 1% increase in QOF achievement on admissions. All analyses were conducted in Stata V.13.

Separate models were estimated for the SMI and bipolar admissions. The model for all SMI admissions included MH6 and MH9 as explanatory variables and for bipolar admissions included MH4 and MH5. Achievement scores were introduced as sets because in practice they are likely to be achieved jointly (the Pearson's correlation between MH4 and MH5 was 0.369, and between MH6 and MH9 was 0.585).

We conducted several additional robustness checks. First, we tested the effect of including QOF indicators separately (rather than as sets). Second, we estimated the model only on within-practice variation, where any time-constant (un)observed practice effects were conditioned out of the likelihood. Third, we used a dependent variable based on numbers of individuals admitted at least once in any given year (rather than total numbers of admissions) to test whether individuals admitted frequently (‘revolving door’ cases)33 distorted observed admission rates. Fourth, we tested the effect of using only admissions coded as emergencies. Fifth, we ran models using 1-year, 2-year or 3-year lags between QOF scores and admissions. Lastly, we estimated separate (cross-sectional) models for each study year to check for temporal effects not already accounted for in our regressions.

Results

Descriptive statistics

Our sample included 8234 GP practices that treated people with SMI during the 5-year period (38 774 practice-year observations; mean follow-up 4.8 years). The number of practices (8052) and practice-year observations (37 573) were lower for the bipolar sample because not all practices treating people with SMI also treated people in the subsample of bipolar disorder. The median number of people with SMI per practice was 39 (interquartile range (IQR)=22–64) and the median number of people with bipolar disorder was 6 (IQR=3–10). The median number of annual admissions per practice was 3.5 (IQR=1–5) for SMI, and 1.1 (IQR=0–2) for bipolar disorder.

Over time, average practice QOF achievement improved across all four indicators, whereas the exception-reporting rate declined (table 3).

Table 3

Average practice population Quality and Outcomes Framework (QOF) achievement and exception rates, 2006/2007 to 2010/2011

Main analysis

There was a consistent positive association between QOF achievement rates and hospital admissions for all indicators apart from MH6 (documented comprehensive care plan) (table 4). For MH9 (annual review), an additional 1% in achievement rates was associated with an average increase in the practice admission rate of 0.19% (95% CI 0.10% to 0.28%) or 0.007 patients (95% CI 0.003 to 0.01). Corresponding figures for bipolar disorder indicators were 0.16% (95% CI 0.01% to 0.30%) or 0.002 patients (95% CI 0.0001 to 0.003) for MH4 (thyroid-stimulating hormone (TSH)/creatinine checks for those on lithium) and 0.10% (95% CI 0.01% to 0.19%) or 0.001 patients (95% CI 0.0001 to 0.002) for MH5 (lithium level within therapeutic range).

Table 4

Regression results for main specification and sensitivity analyses based on population achievement rates

Reported achievement, where exception-reported patients were excluded, was not statistically significantly associated with admissions for any of the four indicators (see online supplementary appendix table 3). However, the association between achievement on MH9 and psychiatric admissions was significant when at least 30% of exception-reported patients were included in the denominator (figure 1). Given the median SMI practice register of 39 patients, the overall (pooled) MH9 exception rate of 12.5% (table 3), and a switching point of 30%, we calculated that approximately 1.5 exception-reported patients per practice year (=39×0.125×0.3) needed to be included in the denominator for the positive effect of MH9 on admissions to be statistically significant. The association between QOF achievement on MH4 [MH5] and admissions for bipolar disorder was statistically significant if at least 0.02 [0.08] exception-reported patients were included in the denominator.

Figure 1

Association between achievement on the Quality and Outcomes Framework (QOF) indicators and admission rates (incidence rate ratios), by percentage of exception-reported patients included in the denominator. (A) Admissions for patients with serious mental illness (SMI). (B) Admissions for patients with bipolar disorder (IRR, incidence rate ratio; TSH, thyroid-stimulating hormone).

Covariates generally had anticipated plausible and significant effects (see online supplementary appendix table 4 for results of all coefficients for the main specifications). Results for reported achievement rates (equation 1) for the main analyses and sensitivity checks are provided in online supplementary appendix table 3.

Sensitivity checks

Findings were generally robust to sensitivity analyses, including patients admitted at least once, within-practice effects, lagged QOF achievement scores and for only admissions coded as emergencies (table 4) although results were not always statistically significant. The effect of population achievement rates on admissions was similar across practices with high/medium/low exception rates. When the model also included the separate population exception rate, only achievement on MH4 had a positive and statistically significant effect on admissions. For all indicators, population achievement was significantly associated with higher admissions if the indicators were included separately rather than together. Cross-sectional analyses broadly supported the results for MH9, but showed that the estimates’ association diminished over time. In addition, the lithium indicators (MH4 and MH5) were not significantly associated with admissions in individual years. None of the sensitivity checks found that the QOF indicators were associated with significantly lower hospital admissions.

Discussion

Our study is the first to investigate the relationship between GP practice quality, as measured by four mental health QOF indicators, and psychiatric admissions in the English NHS. Contrary to expectations, we found that better performance was associated with a higher psychiatric admission rate on three of these indicators, although the likely magnitude of any effect was small.

The potential for higher quality primary care to reduce emergency admissions is an important issue3 ,7 and existing research has addressed this in several disease areas, with mixed results.34–37 The evidence on the effectiveness of the QOF on admissions is similarly mixed.16–21 However, we are not aware of any finding that better quality care is associated with a significantly higher rate of admissions. Our study used longitudinal panel data rather than cross-sectional data, which allows us to control for unobserved time-constant confounders, therefore improving the robustness of results compared with previous research.

We explored the robustness of results to variations in exception reporting in practices and showed the results were sensitive to this. A previous study examining exception reporting found wide variation across practices and across indicators.38 We found a high level of exception reporting for MH9. These may be because a patient is deemed unsuitable for clinical reasons, or because a patient received at least three invitations for review during the preceding 12 months, but did not attend, or they refused to be treated. It is not possible to identify an appropriate level of exception reporting, but some people with SMI may be harder to reach due to the nature of their mental health problems; thus, practices may face difficulties in establishing and maintaining contact, and some degree of exception reporting is, therefore, expected.

There are a number of limitations to our study. As with all observational studies, we cannot ascertain causality. Our results may be affected by unobserved time-varying confounders that correlate with achievement rates. If these factors are positively associated with admission rates and with achievement rates then our estimate of the effect of achievement rate on admissions will be biased upwards. Also, since QOF data are reported at practice level, we cannot be sure whether admitted patients had been reviewed or exception reported. This makes it difficult to ascertain timing and causality. For example, it is possible that those admitted then received a QOF check arranged on discharge from hospital which could explain the positive association. Alternatively, patients with SMI whose problems are more severe may be preferentially registered with practices that are better equipped to provide their care. These practices may achieve higher QOF scores, but also uncover more unmet needs and have more admissions because of their case-mix. Both explanations would imply that the estimated positive association between QOF performance and admissions is not causal.

A second data limitation is that a few of our control variables are time invariant because they are based on census data. It may be possible that the underlying factors (eg, the ethnic composition of neighbourhoods) have changed over our study period, which may bias our results in unknown directions.

Two further limitations arise from our defined inclusion criteria and outcomes. First, we counted all admissions for people with bipolar disorder even though some may not have received lithium therapy, reducing the likelihood of finding effects for MH4 or MH5. Second, we implicitly assumed that psychiatric admission is a poor outcome. However, QOF checks may uncover mental health problems best addressed by admission and we cannot distinguish appropriate admissions from those representing avoidable failures in care.

Finally, it is possible that QOF indicators do not accurately measure the quality of primary care for SMI. The QOF, like any other pay-for-performance scheme, may result in tunnel vision39 or a focus on areas of activities which are incentivised, sometimes at the expense of other non-incentivised activities.40 Thus, high QOF attainment may not necessarily reflect high-quality care.

Further avenues are ripe for exploration including analysis at the patient level, rather than practice level, which would allow for detailed exploration of the entire patient pathway, including identification of the timing of QOF checks for admitted patients. Other priorities include consideration of non-QOF measures of primary care quality that might reduce admissions more effectively and could be incentivised through pay-for-performance; identification of types of secondary care for which utilisation may be affected more by primary care; and investigating whether some practices are more successful than others in getting patients admitted and whether this correlates with their QOF achievement, particularly relevant in the light of high occupancy levels due to closures of mental health beds.41 Finally, greater understanding of unmet needs for people with SMI is essential.42 Prevalence of an unmet need relates to mental healthcare provision and to socioeconomic circumstances—the less integrated and continuous care and the poorer the life situation, the higher is the unmet need.43 More precise estimates of unmet needs can inform policy initiatives to ensure primary care is appropriately equipped and incentivised.

While current policy places an emphasis on ‘upstream’ prevention and ‘early intervention’ to reduce the need for more intensive and expensive specialist care,44 ,45 our findings raise doubt about whether improvements in primary care quality, as measured by the QOF, are likely to achieve this. This chimes more broadly with concerns about the effectiveness of current approaches to reduce avoidable secondary care use.46 While the QOF was not explicitly designed to reduce hospital admissions, there may be effective alternative primary care interventions that could be incentivised in the QOF.

Acknowledgments

The authors would like to thank the NHS National Clinical Classifications Service at NHS Connecting for Health for providing us with the cross-mapping results for Read codes V.2 and V.3 to ICD-10 diagnosis codes used in HES admissions data. They would also like to thank the following steering group members for their invaluable contributions to this project: June Wainwright, Lauren Aylott, Suzanne McBain, Peter Bower, Paul Blenkiron, Liz England and David Daniel.

References

Supplementary materials

  • Supplementary Data

    This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

    Files in this Data Supplement:

Footnotes

  • Contributors NG led the data assembly, analysed the data and contributed to the manuscript drafting. ARM derived the key explanatory variables, and contributed to the analysis and manuscript drafting. TK contributed to the study design, interpretation of results, provided clinical input and to the writing of the manuscript. MG and HG contributed to the study design and interpretation of results, and to the writing of the manuscript. SG contributed to the study design, provided clinical input and helped to interpret findings. JW and LA contributed as service users, contributed to the interpretation of results and implications for practice, and commented on the manuscript. RJ was the principal investigator, overseeing all aspects of the study and is the guarantor.

  • Funding This research was funded by the National Institute for Health Services & Delivery Research programme (project no: 10/1011/22).

  • Competing interests TK has been a member of the NICE national QOF Advisory Committee for clinical indicators since 2009. SG is a member of the HTA Clinical Evaluation and Trials Board.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Data sharing statement HES Copyright © 2006–2010, re-used with the permission of The Health and Social Care Information Centre. All rights reserved. No additional data are available.