Analytical approach

Ideally, a representative sample of the population would be randomised to receive the treatment (renting with SL) or not. The causal effect of the treatment could then be measured by comparing the treatment group with the control group. However, random allocation of housing policy interventions is rarely possible and other approaches must be used. In this case, we do not know the exact allocation mechanism for treatment which is decided by local authorities based on locally held data. Therefore, we will follow a DiD approach with propensity score-matched controls (PSM).25–27 The controlled design will primarily strengthen causal inference by creating a counterfactual and secondarily enable capture and pooling of APS interviews from the fine-scale geography of the intervention, for example, from sets of electoral wards or census tracts.

Study setting

Adult, individual respondents in APS with residence in Greater London between 2011 and 2019. Among these, we will primarily study private renters. Homeowners and social renters will be studied in parallel for feasibility checks. For the area-level outcomes, the data concern 921 treated Lower Layer Super Output Areas (LSOA) and 3684 LSOA including three controls per treated area.

Eligibility criteria

HMOs are covered by separate legislation13 and would add noise to the measurement if not excluded from the study sample. The question about the ‘number of households at sample address’ in APS will be used to exclude HMO respondents. Only areas that were either ever fully treated OR never treated will be included in the study.

Interventions

We will obtain details of the spatial extent and time period of all current and historic SL schemes through Freedom of Information requests to each of Greater London’s 33 boroughs from when first enacted in 2006 to the end of 2019. Local authorities can choose the geography of the schemes. To standardise the data for data fusion and analysis, conversion weights will be calculated based on the number of 2011 Census enumeration postcodes28 falling into small intercepts between the de facto geographical unit and the unit of analysis, Lower LSOA 2011 (LSOA; approximately 1700 average population).29 LSOA units that are only partially under treatment (conversion weights >0 and <1) will be removed from both the treatment and control pool prior to analysis (preliminary analysis showed that this only concerns a small proportion of cases). We furthermore plan to exclude data from two boroughs that introduced street-level schemes, that is, Hammersmith & Fulham and Southwark, and a single electoral ward that was used as a pilot in Newham 2 years before the rest of the borough, that is, Little Ilford. For the temporal scale, the effects will be measured in whole years from the year of initiation. The earliest scheme, Newham, will, thus, only have a single preintervention observation and some schemes will only fall into the evaluation set once more years of outcome data have accrued.

Outcomes

The four subjective health and well-being questions in APS were designed for ONS (aka. ONS4) and intended as four separate dimensions rated from 0 to 10.30 We will study the anxiety question as the primary outcome given its more direct relevance to mental health, and the other questions on subjective well-being as secondary outcomes. The APS also includes other data relevant to residents’ experience of housing, such as residential stability (how long the respondent has lived at the address). The years at address responses will be recoded to category mid-point. The top category, 10+years, will be recorded to 15 years.

Alongside the analyses in APS, we propose a neighbourhood-level analysis with a similar design using SAMHI18 as primary outcome and incidence of police-recorded ASBs19 20 as secondary outcome.

The proportion of dwellings that were privately rented in 2011 Census was 26% (IQR 15–35) in treated areas and 23% (IQR 13–32) in non-treated areas. Any effects measured at the area-level will, thus, be the net effect of designating an area to the scheme (the implications of this are discussed below under Counterfactual).

Statistical methods

A DiD approach with 3:1 PSM will form the basis of the primary analysis.26 31 The propensity score matching will as far as possible use preintervention housing and neighbourhood characteristics from the 2011 Census, IMD and official dwelling age data (table 2)32–34 All matching variables were from 2011 with two exceptions, that is, housing affordability (2012) and Built pre-1945 (2014). Different matching algorithms, that is, propensity score or multivariate distance,35 will be compared using matching diagnostics and the algorithm with the best balance across the matching variables will be selected.36 As a sensitivity check of bias potentially introduced by matching and trimming, we will also conduct the DiD analysis without PSM matching.37 Various sensitivity checks will be conducted such as placebo-in-time and placebo-in-space.

A complicating feature of the intervention for the analysis is that it was not introduced everywhere at the same time. A staggered design is, therefore, proposed for estimating the effect before and after the intervention. We will analyse the effects of the interventions over multiple areas and time periods jointly. This is notably an area of ongoing software development and we will apply the latest techniques and discuss choices and assumptions in the process.31 38–41 With the new multiple time period methods, it is possible to measure a combined effect of all schemes and this will be our preferred method relative to the standard DiD method. However, the individual-level data from APS could end up with panel units that are too sparsely populated for the new methods. We, therefore, also plan to analyse the APS data following a standard DiD setup analysing the data by year of treatment initiation. This will also enable adjusting for differences in time-varying sociodemographic variables in APS (age group, sex, born UK and National Social Occupation Classification42). This adjustment is justified on the grounds that the APS analysis is a repeated cross-sectional study where the outcome between different years could be confounded by the sociodemographic composition alone. It is at the same time assumed that treatment allocation is independent of these confounders.

We will assume that the propensity score matching combined with adjusting for sociodemographic variables collected before and after the intervention will balance out biases introduced by non-response and unequal selection probabilities in APS. To this end, we are planning balance checks such as comparing the values of the provided year weights across treatment and control groups. The neighbourhood analyses will use the same selection of treatment and control areas as the APS analyses.

Sample size

In APS, a yearly average of 13 316 adult respondents were resident in Greater London (2012–2019). Of these, 3017 were private renters. The proportion of London’s population affected by selective licence schemes was 21%. The estimated number of interviews in the treatment group would, thus, be 633 (21% of 3,017). We propose to match treatment to controls 1:3, which would give us 1899 controls and a sample size of 2532 (4*633). We will exclude interviews conducted at addresses with more than one household (HMO by definition). Preliminary analyses suggest that this is a very small proportion of APS households. The unit of analysis for the area-level impacts will be London’s 4835 LSOA. By 2018, 921 LSOA have been included in SL schemes (exclusion criteria are listed under the Methods section). The study size including 3:1 matched controls would be 3684, accordingly.

The power calculations are found in table 3 (two sample t test). Assuming that the outcomes follow a normal distribution, the SD is the same for both the control and treatment group, and the matching will control for all confounding, the study would have 80% power to detect a minimum change in scores of 0.1 for ln(ASB) and SAMHI, 0.25 for population turnover, 0.25 for Satisfied and Worthwhile, 0.5 for Anxious, Happy, Years at address (table 3).

Expected study outcomes

Improving standards in the private rental sector is an important policy area with the potential to impact on population health and health equity. The findings from this study of SL in Greater London will provide an early indication of the impact of such schemes. SL schemes is an example of an intervention that modifies an important social determinant of health and health inequality (ie, housing). As such the schemes fall within a local authority’s discretionary powers and local decision-makers must choose whether or not to implement and renew a scheme in their jurisdiction. Robust evidence of individual and area level of impacts on mental health and well-being, and social outcomes such as crime, should inform these decisions.

The study will also provide data relevant to planning a national evaluation, including information relevant to power calculations, analytical approach, outcomes and data sources and descriptive data relevant to local schemes.

Current study status

As of April 2021, we have recruited two postdoctoral researchers for the quantitative and qualitative strand, respectively. We have successfully obtained data on all existing SL schemes in Greater London as well as most area-based matching and outcome data. We have also started recruiting stakeholders for the qualitative analyses and identified Patient and Public Involvement representatives from London Borough of Hackney. We anticipate submitting the study report for publication by August 2022.

Patient and public involvement

As of April 2022, we have recruited two Patient and Public Involvement representatives. We will seek feedback from them throughout the project.