Quality assessment and risk of bias
Uncontrolled before and after evaluations | |||
---|---|---|---|
Study | Quality criteria | Risk of bias | Notes |
Grant 2000 RCT | Sequence generation | Low | Sealed opaque envelopes prepared by research team. Stratification by practice and blocks of six used (three intervention/three control). |
Allocation concealment | Unclear | Sequentially numbered envelopes opened. In two practices, there was evidence that the randomisation process was initially misunderstood: six patients excluded. | |
Blinding of participants and personal | Not possible | ||
Blinding of outcome assessment | Unclear | ||
Incomplete outcome data | High | 32% loss to follow-up at 4 months | |
Selective outcome reporting | Unclear | ||
Other potential threats to validity | Unclear | Numbers potentially eligible but not recruited unknown Recruited general practices were not a random sample: participating doctors were likely to be more interested in the research question and may have managed psychosocial problems more actively, which could have diminished reported estimates of effects | |
Maughan 2016 CBA | Is there a suitable comparison group? | Yes | One intervention and one control group, drawn from the same general practice with similar patient characteristics. Models environmental costs (in terms of carbon footprint) Data were retrospectively collected from GP health records for a 2-year period. Two participants in intervention group excluded from analysis Financial and environmental impacts calculated for each outcome using national averages or accepted conversion factors |
Do the authors use theory to underpin the project/evaluation? | No | ||
Were appropriate methods used for data collection and analysis? | Yes | ||
Were efforts made to assess patient experience? | No | ||
Uncontrolled before and after evaluations | |||
Study | Quality criteria | Risk of bias | Notes |
Dayson 2014 | Was the study question or objective clearly stated? | Yes | Small sample of those referred (N=1607) participated in evaluation—HES data at 6 months N=451, at 12 months N=108; well-being data at 3–4 months 280/819 Methods of qualitative analysis of patient experience unclear |
Were eligibility/selection criteria for the study population prespecified and clearly described? | Not reported | ||
Were the participants in the study representative of those who would be eligible for the test/service/intervention in the general or clinical population of interest? | Yes | ||
Were all eligible participants that met the prespecified entry criteria enrolled? | Not reported | ||
Was the sample size sufficiently large to provide confidence in the findings? | No | ||
Was the test/service/intervention clearly described and delivered consistently across the study population? | Not reported | ||
Were the outcome measures prespecified, clearly defined, valid, reliable and assessed consistently across all study participants? | Yes | ||
Were the people assessing the outcomes blinded to the participants’ exposures/interventions? | Not reported | ||
Was the loss to follow-up after baseline 20% or less? Were those lost to follow-up accounted for in the analysis? | No | ||
Did the statistical methods examine changes in outcome measures from before to after the intervention? Were statistical tests done that provided p values for the pre-to-post changes? | Yes | ||
Were outcome measures of interest taken multiple times before the intervention and multiple times after the intervention (ie, did they use an interrupted time-series design)? | No | ||
If the intervention was conducted at a group level (eg, a whole hospital, a community, etc), did the statistical analysis take into account the use of individual-level data to determine effects at the group level? | Not applicable | ||
Friedli 2012 | Was the study question or objective clearly stated? | Yes | Details of preintervention and postintervention outcomes not reported Small sample size Timing of post intervention assessment not reported Methods of qualitative analysis of patient and provider/referrer experience unclear |
Were eligibility/selection criteria for the study population prespecified and clearly described? | No | ||
Were the participants in the study representative of those who would be eligible for the test/service/intervention in the general or clinical population of interest? | Yes | ||
Were all eligible participants that met the prespecified entry criteria enrolled? | Not applicable | ||
Was the sample size sufficiently large to provide confidence in the findings? | No | ||
Was the test/service/intervention clearly described and delivered consistently across the study population? | Not reported | ||
Were the outcome measures prespecified, clearly defined, valid, reliable and assessed consistently across all study participants? | No | ||
Were the people assessing the outcomes blinded to the participants’ exposures/interventions? | No | ||
Was the loss to follow-up after baseline 20% or less? Were those lost to follow-up accounted for in the analysis? | No | ||
Did the statistical methods examine changes in outcome measures from before to after the intervention? Were statistical tests done that provided p values for the pre-to-post changes? | No | ||
Were outcome measures of interest taken multiple times before the intervention and multiple times after the intervention (i.e., did they use an interrupted time-series design)? | No | ||
If the intervention was conducted at a group level (eg, a whole hospital, a community, etc), did the statistical analysis take into account the use of individual-level data to determine effects at the group level? | Not applicable | ||
Grayer 2008 | Was the study question or objective clearly stated? | Yes | GP practices volunteered and may not be representative of practices overall Patients who consented to participate in evaluation were more likely to speak English as a first language than those who did not consent No significant differences at baseline between those successfully followed up and those lost to follow-up 95% CIs (no p values) reported for changes in GHQ-12, CORE-OM and WSAS scores |
Were eligibility/selection criteria for the study population prespecified and clearly described? | Yes | ||
Were the participants in the study representative of those who would be eligible for the test/service/intervention in the general or clinical population of interest? | Yes | ||
Were all eligible participants that met the prespecified entry criteria enrolled? | No | ||
Was the sample size sufficiently large to provide confidence in the findings? | No | ||
Was the test/service/intervention clearly described and delivered consistently across the study population? | Yes | ||
Were the outcome measures prespecified, clearly defined, valid, reliable and assessed consistently across all study participants? | Yes | ||
Were the people assessing the outcomes blinded to the participants’ exposures/interventions? | Not reported | ||
Was the loss to follow-up after baseline 20% or less? Were those lost to follow-up accounted for in the analysis? | No | ||
Did the statistical methods examine changes in outcome measures from before to after the intervention? Were statistical tests done that provided p values for the pre-to-post changes? | Yes | ||
Were outcome measures of interest taken multiple times before the intervention and multiple times after the intervention (i.e., did they use an interrupted time-series design)? | No | ||
If the intervention was conducted at a group level (eg. a whole hospital, a community, etc), did the statistical analysis take into account the use of individual-level data to determine effects at the group level? | Not applicable | ||
Kimberlee 2014 | Was the study question or objective clearly stated? | Yes | SROI analysis presents data for all baseline completers and the smaller percentage who were followed up; possible bias towards positive finding for intervention Unclear whether calculations of mean differences in scale scores used all baseline data or baseline data for follow-up completers only p values reported for change from baseline at 3 months in PHQ-9 depression scores |
Were eligibility/selection criteria for the study population prespecified and clearly described? | No | ||
Were the participants in the study representative of those who would be eligible for the test/service/intervention in the general or clinical population of interest? | Yes | ||
Were all eligible participants that met the prespecified entry criteria enrolled? | Not applicable | ||
Was the sample size sufficiently large to provide confidence in the findings? | No | ||
Was the test/service/intervention clearly described and delivered consistently across the study population? | Not reported | ||
Were the outcome measures prespecified, clearly defined, valid, reliable and assessed consistently across all study participants? | Yes | ||
Were the people assessing the outcomes blinded to the participants’ exposures/interventions? | Not reported | ||
Was the loss to follow-up after baseline 20% or less? Were those lost to follow-up accounted for in the analysis? | No | ||
Did the statistical methods examine changes in outcome measures from before to after the intervention? Were statistical tests done that provided p values for the pre-to-post changes? | Yes | ||
Were outcome measures of interest taken multiple times before the intervention and multiple times after the intervention (i.e., did they use an interrupted time-series design)? | No | ||
If the intervention was conducted at a group level (eg. a whole hospital, a community, etc), did the statistical analysis take into account the use of individual-level data to determine effects at the group level? | Not applicable |
CBA, Controlled Before and After study; CORE-OM, Clinical Outcomes in Routine Evaluation-Outcome Measure; GP, general practice; GHQ, General Health Questionnaire; HES, hospital episode statistics; PHQ, Patient Health Questionnaire; RCT, randomised controlled trial; SROI, Social Return on Investment; WSAS, Work and Social Adjustment Scale.