Multiple deprivation indicators are frequently used to capture the characteristics of an area. This is a useful approach for identifying the most deprived areas, and summary indices are good predictors of mortality and morbidity, but it remains unclear which aspects of the residential environment are most salient for health. A further question is whether the most important aspects vary for different types of residents. This paper focuses on whether associations with neighbourhood characteristics are different for men and women. The sociopolitical and physical environment, amenities, and indicators of economic deprivation and affluence were measured in neighbourhoods in the UK, and their relationship with self-rated health was investigated using multilevel regression models. Each of these contextual domains was associated with self-rated health over and above individual socioeconomic characteristics. The magnitude of the association was larger for women in each case. Statistically significant interactions between gender and residential environment were found for trust, integration into wider society, left-wing political climate, physical quality of the residential environment, and unemployment rate. These findings add to the literature indicating greater effects of non-work-based stressors for women and highlight the influence of the residential environment on women's health.