Table 2

Quality assessment and risk of bias

Uncontrolled before and after evaluations
StudyQuality criteriaRisk of biasNotes
Grant 2000
RCT
Sequence generationLowSealed opaque envelopes prepared by research team. Stratification by practice and blocks of six used (three intervention/three control).
Allocation concealmentUnclearSequentially numbered envelopes opened. In two practices, there was evidence that the randomisation process was initially misunderstood: six patients excluded.
Blinding of participants and personalNot possible
Blinding of outcome assessmentUnclear
Incomplete outcome dataHigh32% loss to follow-up at 4 months
Selective outcome reportingUnclear
Other potential threats to validityUnclearNumbers potentially eligible but not recruited unknown
Recruited general practices were not a random sample: participating doctors were likely to be more interested in the research question and may have managed psychosocial problems more actively, which could have diminished reported estimates of effects
Maughan 2016
CBA
Is there a suitable comparison group?YesOne intervention and one control group, drawn from the same general practice with similar patient characteristics.
Models environmental costs (in terms of carbon footprint)
Data were retrospectively collected from GP health records for a 2-year period.
Two participants in intervention group excluded from analysis
Financial and environmental impacts calculated for each outcome using national averages or accepted conversion factors
Do the authors use theory to underpin the project/evaluation?No
Were appropriate methods used for data collection and analysis?Yes
Were efforts made to assess patient experience?No
Uncontrolled before and after evaluations
StudyQuality criteriaRisk of biasNotes
Dayson 2014Was the study question or objective clearly stated?YesSmall sample of those referred (N=1607) participated in evaluation—HES data at 6 months N=451, at 12 months N=108; well-being data at 3–4 months 280/819
Methods of qualitative analysis of patient experience unclear
Were eligibility/selection criteria for the study population prespecified and clearly described?Not reported
Were the participants in the study representative of those who would be eligible for the test/service/intervention in the general or clinical population of interest?Yes
Were all eligible participants that met the prespecified entry criteria enrolled?Not reported
Was the sample size sufficiently large to provide confidence in the findings?No
Was the test/service/intervention clearly described and delivered consistently across the study population?Not reported
Were the outcome measures prespecified, clearly defined, valid, reliable and assessed consistently across all study participants?Yes
Were the people assessing the outcomes blinded to the participants’ exposures/interventions?Not reported
Was the loss to follow-up after baseline 20% or less? Were those lost to follow-up accounted for in the analysis?No
Did the statistical methods examine changes in outcome measures from before to after the intervention? Were statistical tests done that provided p values for the pre-to-post changes?Yes
Were outcome measures of interest taken multiple times before the intervention and multiple times after the intervention (ie, did they use an interrupted time-series design)?No
If the intervention was conducted at a group level (eg, a whole hospital, a community, etc), did the statistical analysis take into account the use of individual-level data to determine effects at the group level?Not applicable
Friedli 2012Was the study question or objective clearly stated?YesDetails of preintervention and postintervention outcomes not reported
Small sample size
Timing of post intervention assessment not reported
Methods of qualitative analysis of patient and provider/referrer experience unclear
Were eligibility/selection criteria for the study population prespecified and clearly described?No
Were the participants in the study representative of those who would be eligible for the test/service/intervention in the general or clinical population of interest?Yes
Were all eligible participants that met the prespecified entry criteria enrolled?Not applicable
Was the sample size sufficiently large to provide confidence in the findings?No
Was the test/service/intervention clearly described and delivered consistently across the study population?Not reported
Were the outcome measures prespecified, clearly defined, valid, reliable and assessed consistently across all study participants?No
Were the people assessing the outcomes blinded to the participants’ exposures/interventions?No
Was the loss to follow-up after baseline 20% or less? Were those lost to follow-up accounted for in the analysis?No
Did the statistical methods examine changes in outcome measures from before to after the intervention? Were statistical tests done that provided p values for the pre-to-post changes?No
Were outcome measures of interest taken multiple times before the intervention and multiple times after the intervention (i.e., did they use an interrupted time-series design)?No
If the intervention was conducted at a group level (eg, a whole hospital, a community, etc), did the statistical analysis take into account the use of individual-level data to determine effects at the group level?Not applicable
Grayer 2008Was the study question or objective clearly stated?YesGP practices volunteered and may not be representative of practices overall
Patients who consented to participate in evaluation were more likely to speak English as a first language than those who did not consent
No significant differences at baseline between those successfully followed up and those lost to follow-up
95% CIs (no p values) reported for changes in GHQ-12, CORE-OM and WSAS scores
Were eligibility/selection criteria for the study population prespecified and clearly described?Yes
Were the participants in the study representative of those who would be eligible for the test/service/intervention in the general or clinical population of interest?Yes
Were all eligible participants that met the prespecified entry criteria enrolled?No
Was the sample size sufficiently large to provide confidence in the findings?No
Was the test/service/intervention clearly described and delivered consistently across the study population?Yes
Were the outcome measures prespecified, clearly defined, valid, reliable and assessed consistently across all study participants?Yes
Were the people assessing the outcomes blinded to the participants’ exposures/interventions?Not reported
Was the loss to follow-up after baseline 20% or less? Were those lost to follow-up accounted for in the analysis?No
Did the statistical methods examine changes in outcome measures from before to after the intervention? Were statistical tests done that provided p values for the pre-to-post changes?Yes
Were outcome measures of interest taken multiple times before the intervention and multiple times after the intervention (i.e., did they use an interrupted time-series design)?No
If the intervention was conducted at a group level (eg. a whole hospital, a community, etc), did the statistical analysis take into account the use of individual-level data to determine effects at the group level?Not applicable
Kimberlee 2014Was the study question or objective clearly stated?YesSROI analysis presents data for all baseline completers and the smaller percentage who were followed up; possible bias towards positive finding for intervention
Unclear whether calculations of mean differences in scale scores used all baseline data or baseline data for follow-up completers only
p values reported for change from baseline at 3 months in PHQ-9 depression scores
Were eligibility/selection criteria for the study population prespecified and clearly described?No
Were the participants in the study representative of those who would be eligible for the test/service/intervention in the general or clinical population of interest?Yes
Were all eligible participants that met the prespecified entry criteria enrolled?Not applicable
Was the sample size sufficiently large to provide confidence in the findings?No
Was the test/service/intervention clearly described and delivered consistently across the study population?Not reported
Were the outcome measures prespecified, clearly defined, valid, reliable and assessed consistently across all study participants?Yes
Were the people assessing the outcomes blinded to the participants’ exposures/interventions?Not reported
Was the loss to follow-up after baseline 20% or less? Were those lost to follow-up accounted for in the analysis?No
Did the statistical methods examine changes in outcome measures from before to after the intervention? Were statistical tests done that provided p values for the pre-to-post changes?Yes
Were outcome measures of interest taken multiple times before the intervention and multiple times after the intervention (i.e., did they use an interrupted time-series design)?No
If the intervention was conducted at a group level (eg. a whole hospital, a community, etc), did the statistical analysis take into account the use of individual-level data to determine effects at the group level?Not applicable
  • CBA, Controlled Before and After study; CORE-OM, Clinical Outcomes in Routine Evaluation-Outcome Measure; GP, general practice; GHQ, General Health Questionnaire; HES, hospital episode statistics; PHQ, Patient Health Questionnaire; RCT, randomised controlled trial; SROI, Social Return on Investment; WSAS, Work and Social Adjustment Scale.