Objective The primary aim is to identify, summarise and quality assess the available literature on the cost-effectiveness of implementing low back pain guidelines in primary care. The secondary aim is to assess the transferability of the results to determine whether the identified studies can be included in a comparison with a Danish implementation study to establish which strategy procures most value for money.
Design Systematic review.
Data sources The search was conducted in Embase, PubMed, Cochrane Library, NHS Economic Evaluation Database, Scopus, CINAHL and EconLit. No restrictions were made concerning language, year of publication or publication type. The bibliographies of the included studies were searched for any studies not captured in the literature search.
Eligibility criteria for selecting studies To be included, a study must be: (1) based on a randomised controlled trial comparing implementation strategies, (2) the guideline must concern treatment of low back pain in primary care and (3) the economic evaluation should contain primary data on cost and cost-effectiveness.
Results The title and abstract were assessed for 308 studies; of these, three studies were found eligible for inclusion. The Consensus Health Economic Criteria (CHEC) list showed that the 3 studies were of moderate methodological quality while application of Welte's model showed that cost results from two studies could, with adjustments, be transferred to a Danish setting. It was questionable whether the associated effects could be transferred.
Conclusions Despite the resemblance of the implementation strategies, the 3 studies report conflicting results on cost-effectiveness. This review showed that transferring the results from the identified studies is not straightforward and underlines the importance of transparent reporting. Future research should focus on transferability of effects, for example, development of a supplement to Welte's model.
- HEALTH ECONOMICS
- Guideline implementation
- Primary health care
- Low back pain
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Strengths and limitations of this study
Policymakers could potentially make wrong decisions if they are not aware of the difficulties in transferring costs and effects between countries.
To the best of our knowledge, this is the first review to assess transferability concurrently with methodological quality.
Only studies focusing on both the costs and effects of strategies for implementing low back pain guidelines in general practice were included in this review.
Only limited research concerns the cost-effectiveness of implementing low back pain guidelines in primary care.
A comparison of all relevant alternatives should be carried out when assessing the cost-effectiveness of new a intervention.1 This may be challenging if not all alternatives have been evaluated in the decision-makers' jurisdiction and, as it is very time-consuming and expensive to conduct research, the utilisation of foreign data may be of great value. Utilisation is, however, conditional on transferability of the foreign data from the study country to the decision country.2
In connection with the dissemination of a clinical guideline on the management of low back pain (LBP) in primary care, a cluster randomised controlled trial was carried out in Denmark.3 ,4 The objective of this trial was to assess the effectiveness and cost-effectiveness of an extended implementation strategy to enhance the general practitioners' adherence to the guidelines compared with usual dissemination.5 ,6 It would significantly add to this study if the results of the economic evaluation could be contrasted with results from other studies assessing different strategies towards implementing LBP guidelines in primary care. This could provide insight into which strategy procures most value for money.
The primary aim of this systematic review is to identify, summarise and assess the quality of economic evaluations concerning strategies used in the implementation of guidelines for management of patients with LBP in primary care. The secondary aim is to assess the transferability of the results by application of the model of Welte,7 which is a decision chart for assessing the transferability of results from economic evaluations between countries. By use of the model of Welte,7 the transferability appraisal aims to inform whether the existing studies can be included in a comparison with the Danish guideline implementation study,6 followed by a discussion of the applicability of Welte's model.
The reporting of this systematic review follows, where applicable, the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA).8 ,9 See online supplementary file 1 for the PRISMA 2009 checklist.
For studies to be included in the systematic review, the following eligibility criteria must be met: (1) the study must be based on a randomised controlled clinical trial comparing implementation strategies, (2) the guideline must concern treatment of patients with LBP in primary care, and (3) the economic evaluation should contain primary data on cost and cost-effectiveness. To identify all relevant studies, a literature search was conducted in the following databases: Embase, PubMed, Cochrane Library, NHS Economic Evaluation Database, Scopus, CINAHL and EconLit. No restrictions were made concerning language, year of publication or publication type. The keywords ‘guideline implementation’, ‘low back pain’, ‘primary health care’ and ‘cost and cost-analysis’ were searched for in combination by use of the Boolean operator ‘AND’. As far as possible, thesaurus terms were applied and adjusted to the individual databases and used in combination with free text search. The last search was run on 13 February 2016. See online supplementary file 2 for a complete search history.
After removal of duplicates, two authors (CEJ and AR) screened the title and abstracts of the recovered studies applying a five-point scale: (1) the study does not concern LBP, (2) the study does not concern LBP guideline implementation in primary care, (3) the study is not based on a randomised controlled clinical trial, (4) the study is not an economic evaluation, or (5) the study is eligible for full-text screening. Studies assigned a value from one to four were excluded, while studies assigned the value five were full-text screened and included if eligible. Any uncertainty about whether or not to include a study in the systematic review was resolved by consensus among the authors. Following this, the bibliographies of the included studies were searched for any studies not captured in the systematic literature search.
Information about study country, study design, type of economic evaluation, population characteristics, interventions, perspective, time horizon, effectiveness measures, incremental cost-effectiveness ratio (ICER) and sensitivity analyses was extracted from the included studies. Data extraction was carried out by CEJ and KDP.
Assessment of quality
The Consensus Health Economic Criteria (CHEC) list, developed for use in systematic reviews involving economic evaluations, was applied to assess the methodological quality of the included studies.10 The CHEC list is composed of 19 questions, where each question is assigned either a ‘yes’, indicating that the item was either covered adequately or reported in an appropriate way, or ‘no’, indicating that the item was not met.11 Studies receiving a ‘yes’ score higher than 75% were defined as high-quality evaluations, while studies with a score higher than 50% and up to 75% were defined as moderate quality evaluations, and studies with a score of up to 50% were defined as low-quality evaluations.
Assessment of transferability
To assess the transferability of the included studies from the study country to a Danish setting, the decision chart for assessing and improving the transferability of economic evaluation results between countries developed by Welte et al7 was applied.
Welte's model is a transferability decision chart method that includes both general and specific knockout criteria to assess if a study can be transferred to the decision country.7 Initially, the study must meet three general knockout criteria. The first general knockout criterion requires that the evaluated alternative is compatible with the one that may be used in the decision country. The second general knockout criterion entails that the comparator must likewise be compatible with one that is relevant in the decision country. Finally, the third general knockout criterion requires that the study must be of acceptable methodological quality.7 In this systematic review, results based on the CHEC list were used to assess the third general knockout criteria.
For studies passing the three general knockout criteria, an additional 14 specific knock out criteria are applied afterwards.7 For each of the 14 specific knockout criteria, the estimated relevance of each criterion, the correspondence between the study country and the decision country, and the expected effect on the cost-effectiveness ratio (CER) must be determined. Estimated relevance and correspondence is graded on a five-level scale from very high to very low. According to Welte et al,7 a very high or high correspondence between the study country and the decision country, regardless of estimated relevance of the criterion, is needed for an unbiased estimate of the CER. CEJ carried out the transferability assessment, which was discussed afterwards and adjusted in collaboration with all authors.
Through the systematic literature search, a total of 459 studies were identified. Duplicates (n=151) were removed and the title and abstract were assessed for the remaining 308 studies. Of the assessed studies, 136 (44%) were excluded as they did not deal with LBP. An additional 149 studies (48%) did not concern specific strategies aimed at implementing LBP guidelines in primary care. Furthermore, 13 studies (4%) were excluded, as these were not based on randomised clinical trials, and seven studies (2%), as these were not economic evaluations. In total, 305 studies (99%) were excluded based on title and abstract, leaving no more than three studies (1%) eligible for full-text screening and inclusion. No additional studies were identified through a search of the bibliographies of the included studies. Figure 1 is a depiction of this process.
In the study by Mortimer et al,12 a cost-effectiveness analysis was carried out alongside a cluster randomised controlled trial. To increase Australian general practitioners’ adherence to the LBP guidelines, a multifaceted and theory-based implementation strategy (IMPLEMENT) was developed. The hypothesis was that the IMPLEMENT intervention, when taking into account reductions in health service use, would be cost-effective compared with standard dissemination. The intervention consisted of two facilitated interactive workshops concentrating on two key messages from the guidelines. The first workshop focused on reducing X-ray referrals, as these are rarely beneficial, and how to handle patients with X-ray seeking behaviour. The second workshop centred on the importance of patients remaining active to reduce pain and disability. For the primary outcome, the number of X-ray referrals, the IMPLEMENT intervention dominated standard dissemination when excluding development costs. However, from a healthcare sector perspective, >95% probability of cost-effectiveness could not be achieved.
The study by Becker et al13 is likewise a cost-effectiveness analysis carried out alongside a cluster randomised controlled trial. In the clinical trial, two different implementation strategies were compared with the usual strategy of dissemination. In the first intervention group ‘Guideline Implementation’ (GI), the implementation strategy consisted of physician education, where the general practitioners were trained in using the LBP guideline. In addition to physician education, the implementation strategy in the second intervention group ‘Guideline Implementation and Motivational Counselling’ (GI+MC) consisted of motivational counselling by practice nurses, who had a 20-hour training course in how to motivate patients with LBP for physical activity. In the German setting, both interventions appeared less costly, when excluding implementation costs, and more effective for the primary outcome functional capability. From a societal perspective, the probability of GI being cost-effective compared with dissemination was 97% at a threshold value of €67 per point on the functional capability scale, while there was a 99% probability at €99 when comparing GI+MC with dissemination.
Hoeijenbos et al14 undertook a cost-utility analysis conducted alongside a clinical trial. The aim of this economic evaluation was to evaluate the cost-effectiveness of an active implementation strategy aimed at physiotherapists working in primary care compared with dissemination. The active implementation strategy featured a training session, where the LBP guideline was explained as well as discussed with the physiotherapists, followed by special skills practice. For the following 4 weeks, the physiotherapists had to practise using the guideline after which a second session was conducted, where experiences and problems, if any, were discussed and the physiotherapists received feedback on their current management of patients with LBP. Since no significant difference in either costs or effects was found between the intervention and the control groups, no assessment of the cost-effectiveness of the active implementation strategy was made.
Assessment of quality
Application of the CHEC list indicated that all three studies were of moderate methodological quality (table 2).
Both Mortimer et al12 and Becker et al13 clearly described the characteristics of the study population (item 1). Mortimer et al12 failed to include patients in the clinical trial, necessitating a departure from the planned analyses. This entailed that the general practitioners became the study participants and as a reference is made to the analysis of the effectiveness,17 this item is categorised as fulfilled. Hoeijenbos et al,14 however, received a ‘no’ on this item, as no eligibility criteria were described for the study population and no reference was made for additional information. Moreover, Hoeijenbos et al14 received a ‘no’ on item 3 concerning a well-defined research question, because, in addition to clearly identifying the alternatives compared, the research question should also state the population.11 With regard to identifying, measuring, and valuing relevant costs in accordance with perspective (item 7–9), only Mortimer et al12 successfully met all three criteria. Item 16, which is centred on whether the conclusion follows from the data reported, was only covered by one study,14 while only Mortimer et al12 addressed any ethical or distributional implications (item 19).
Assessment of transferability
The first step in assessing the transferability of the included studies to a Danish setting was to apply the three general knockout criteria. A review by Koes et al19 has shown that guidelines on managing LBP in primary care are quite similar over a range of countries, including Australia, Germany and the Netherlands. Hence, all studies12–14 pass the first two general knockout criteria. Application of the third general knockout criterion is, however, not as straightforward.
Although the study by Mortimer et al12 was evaluated as being of moderate quality using the CHEC list, it had to deviate from the planned analyses20 and use an intermediate outcome. This is associated with several limitations and results should be interpreted with caution and, furthermore, precludes comparison with the other studies included. On these grounds, the study is considered not to pass the third general knockout criterion.
Becker et al13 likewise have some methodological challenges influencing the transferability assessment. This is primarily related to the lack of results presented, including consumption of resources, valuation and effects. Only incremental cost and effect are presented. Furthermore, the base case ICER does not include implementation costs and is based on complete case analysis, possibly introducing selection bias. Adjusting for baseline difference in costs and clustering of data is done in a sensitivity analysis and presented only in an incremental cost-effectiveness plane showing an estimated ICER close to zero. Since the authors may provide further information or a simple adjustment based on purchasing power parities could be carried out, it is relevant to proceed with the 14 specific knockout criteria (table 3).
The study by Hoeijenbos et al14 differs from other studies, as the implementation strategy is aimed at physiotherapists working in primary care. Since the Danish guideline on management of LBP in primary care is also aimed at physiotherapists,3 it is relevant to evaluate whether it would be more cost-effective to increase their adherence to the guideline as neither general practitioners nor physiotherapists have high adherence.21 Application of the 14 specific knockout criteria are shown in table 4. Lack of accordance in the approach to productivity cost estimation between the Netherlands and Denmark increases the risk of a biased CER. The transferability of lost productivity might, according to Welte et al,7 be improved by adjustment of the resource valuation.
On the basis of the transferability assessment, both the costs from Becker et al and Hoijenbos et al may with adjustments be transferable, with caution, to a Danish setting. However, the model of Welte does not pay much attention to effectiveness, which raises issues in the assessment of the transferability of these two studies. In Becker et al's study, only incremental effect at follow-up was reported, while in Hoijenbos et al's study no final follow-up effect was reported. This renders adjustment of foreign utilities impossible.
The systematic review identified no more than three studies that evaluated the cost-effectiveness of different strategies to implement guidelines on management of LBP in primary care.12–14 Application of the CHEC list showed that the three studies were of moderate methodological quality while use of Welte's model showed that cost results from two studies13 ,14 could, with adjustments, be transferable to a Danish setting, whereas it was questionable whether the effectiveness results could be transferred as well. In a review, Koes et al19 compared international guidelines on managing LBP and showed that recommendations are generally similar, which underlines the relevance of the present comparison of strategies for implementing LBP guidelines. Despite the resemblance of the implementation strategies, the studies nonetheless reported conflicting conclusions of whether the additional upfront cost of the implementation strategies was counterbalanced by improvements in clinical practice or patient outcomes. Neither Mortimer et al nor Hoeijenbos et al found substantial evidence of cost-effectiveness, while Becker et al reported that the intervention was cost-effective compared with the usual implementation strategy. However, to establish which implementation strategy truly procures most value for money, it is important to transfer both costs and effects to the decision country to ensure a valid comparison. This review underlines the importance of transparent reporting to increase transferability and thus facilitate valid comparisons between competing alternatives.
This systematic review, which is the first to look into the cost-effectiveness of strategies used in the implementation of LBP guidelines in primary care, has shown that only a limited amount of research exists within this field. In addition to evidence synthesis, a transferability appraisal was simultaneously carried out to assess whether the results of the economic evaluations could be transferred to a Danish setting. To ensure that all relevant studies were identified, free text search was used in combination with thesaurus terms. Applying a broad search strategy ensured that no relevant studies were overlooked; however, this also resulted in identification of a large number of irrelevant studies. A limitation of this review is that studies only carrying out an analysis of the effectiveness of the intervention were not included. Furthermore, one author carried out the assessment of transferability. It would have been advantageous if two or more independent assessments had been carried out. Moreover, since there are six other checklists of transferability in addition to Welte's model,22 it is possible that application of any of these would have procured another result. Welte's model was chosen as it has more distinctive criteria for evaluation of transferability; however, there are some concerns in the application of this model.
Welte's model is portrayed as transparent and user-friendly; nonetheless, it is based on subjective judgements with low repeatability/reliability, especially with regard to the 14 specific knockout criteria, where there is no clear distinction of whether or not a study is transferable if it has passed the three general knockout criteria. The great attention to the different aspects of cost estimation and valuation is advantageous. Furthermore, it is specified for each of the 14 criteria how transferability might be improved by adjustment of resource utilisation or valuation; this is, however, under the assumption that all subgroups of costs can be transferred between countries, which might not be the case. For the criterion ‘Productivity and work-loss time’, impact on CER was deemed to be too high or too low as choice of method for assessing lost productivity, human capital versus friction method, greatly affects the cost estimation. Since Welte et al7 state that adjustment of both utilisation and valuation might increase transferability, medium correspondence between countries was chosen due to national differences in choice of method for estimating lost productivity. Another study, however, found that due to large differences between countries, transferring lost productivity is not advisable.23
A common measure of effect is necessary when comparing or pooling different economic evaluations, but effectiveness is not paid much attention in the model of Welte, where only one of the 14 specific criteria addresses this subject. The criterion assumes a preference for a generic measure of effect, for example, EuroQol health status values, but does not discuss how transferability might be improved. A number of studies have shown that transferring utilities between countries might produce biased CERs24–29 why the application of a model that adjusts foreign mean utilities to reflect national preferences of health could increase transferability.30 Moreover, there may be cases where a disease-specific measure of effect would be preferred, but this is not reflected in Welte's model. Within the field of LBP research, a set of core outcomes31–33 has been identified to increase comparability across studies, and perhaps one of these outcomes is more readily transferable between countries.
A study by Knies et al34 further supports more focus on the importance of transferability of the effects, as they found that application of Welte's model yielded better results for cost prediction than for effect prediction. However, no assessment of the homogeneity of the studies was carried out, meaning that both the cost and the effect prediction might have procured another result having only included studies that could be pooled.
Applying a systematic checklist ensures a consistent assessment across studies. However, as Welte et al emphasise that the model is a first attempt, it could be relevant to update this model based on the aforementioned as well as positive features of the other transferability checklists.2 ,35–39 Furthermore, in future research it could be interesting to apply the different checklists to the same studies to compare and contrast transferability results.
A comparison of all relevant strategies for implementing guidelines on LBP management in primary care could provide insight into which strategy procures most value for money. This systematic review, however, showed that transferring the results from the identified studies is not straightforward and underlines the importance of transparent reporting. Moreover, increased focus on transferability of effect is desirable, for example, as a supplement to the model of Welte.
The authors thank Professor Kjeld Møller Pedersen for reviewing and commenting on the manuscript.
Contributors All authors were involved in the design of the study. CEJ designed the literature search and screened the titles and abstracts of the recovered studies in collaboration with AR. CEJ and KDP performed the assessment of quality and CEJ performed the transferability assessment. The critical review and approval of the manuscript involved all authors.
Funding This work was supported by the Health Foundation (Helsefonden) (grant number 2013B133), Trygfonden (grant number ID105189) and the Danish Rheumatism Association (grant numbers R114-A2682 and R102-A2168).
Disclaimer The funders have not played any role in the design or completion of this systematic review. Furthermore, they have not played any role in drafting this paper or submitting it for publication, and hence accept no responsibility for the contents.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement No additional data are available.