Article Text

PDF

Sustainability of professionals’ adherence to clinical practice guidelines in medical care: a systematic review
  1. Stephanie M C Ament1,2,
  2. Jeanny J A de Groot1,3,
  3. José M C Maessen1,4,
  4. Carmen D Dirksen5,
  5. Trudy van der Weijden1,
  6. Jos Kleijnen1,6
  1. 1Department of Family Medicine, School for Public Health and Primary Care (CAPHRI), Maastricht University, Maastricht, The Netherlands
  2. 2School for Oncology and Developmental Biology (GROW), Maastricht University Medical Centre, Maastricht, The Netherlands
  3. 3Department of Obstetrics and Gynaecology, Maastricht University Medical Centre, Maastricht, The Netherlands
  4. 4Department of Patient & Integrated Care, Maastricht University Medical Centre, Maastricht, The Netherlands
  5. 5Department of Clinical Epidemiology and Medical Technology Assessment, Maastricht University Medical Centre, Maastricht, The Netherlands
  6. 6Kleijnen Systematic Reviews Ltd, York, UK
  1. Correspondence to Stephanie MC Ament; stephanie.ament{at}maastrichtuniversity.nl

Abstract

Objectives To evaluate (1) the state of the art in sustainability research and (2) the outcomes of professionals’ adherence to guideline recommendations in medical practice.

Design Systematic review.

Data sources Searches were conducted until August 2015 in MEDLINE, CINAHL, EMBASE, Cochrane Central Register of Controlled Trials (CENTRAL) and the Guidelines International Network (GIN) library. A snowball strategy, in which reference sections of other reviews and of included papers were searched, was used to identify additional papers.

Eligibility criteria Studies needed to be focused on sustainability and on professionals’ adherence to clinical practice guidelines in medical care. Studies had to include at least 2 measurements: 1 before (PRE) or immediately after implementation (EARLY POST) and 1 measurement longer than 1 year after active implementation (LATE POST).

Results The search retrieved 4219 items, of which 14 studies met the inclusion criteria, involving 18 sustainability evaluations. The mean timeframe between the end of active implementation and the sustainability evaluation was 2.6 years (minimum 1.5–maximum 7.0). The studies were heterogeneous with respect to their methodology. Sustainability was considered to be successful if performance in terms of professionals’ adherence was fully maintained in the late postimplementation phase. Long-term sustainability of professionals’ adherence was reported in 7 out of 18 evaluations, adherence was not sustained in 6 evaluations, 4 evaluations showed mixed sustainability results and in 1 evaluation it was unclear whether the professional adherence was sustained.

Conclusions (2) Professionals’ adherence to a clinical practice guideline in medical care decreased after more than 1 year after implementation in about half of the cases. (1) Owing to the limited number of studies, the absence of a uniform definition, the high risk of bias, and the mixed results of studies, no firm conclusion about the sustainability of professionals’ adherence to guidelines in medical practice can be drawn.

Statistics from Altmetric.com

Strengths and limitations of this study

  • This is the first systematic review of the literature that has considered professionals’ adherence to clinical practice guidelines more than 1 year after active implementation. This review shows that in half of the sustainability studies professionals fully sustained in their adherence to a clinical practice guideline.

  • This review showed that sustainability research is a relatively new and underexplored field in healthcare.

  • Sustainability research is not well indexed in electronic databases, and text word searches are prone to high recall and low specificity. However, it is likely that the use of a broad variety of search terms that covered sustainability has downsized the number of relevant studies missed and is a strength of the review.

  • The number of studies and the methodological quality of the studies focusing on the sustainability of professionals’ adherence are limited. This makes it difficult to draw firm conclusions.

Introduction

Quality of care can be improved by decreasing unwarranted practice variation between professionals. One way to reduce practice variation is by transferring evidence-based knowledge into daily practice. To facilitate the translation of the most recent evidence into practice, guidelines are developed and implemented. Following the Institute of Medicine (IOM), clinical practice guidelines are “statements that include recommendations intended to optimize patient care that are informed by a systematic review of evidence and an assessment of the benefit and harms of alternative care options”.1 Guidelines contain practical evidence-based advice for professionals and patients and aim to improve the quality of care.2 In general, uptake of guidelines does not happen spontaneously and often an active implementation approach is required.3 Moreover, once a guideline is successfully implemented in practice, it may be difficult to sustain the quality improvements over a longer period of time. People tend to fall back into old routines4 which may impact long-term adherence to a guideline.

The road towards sustainability of healthcare innovations into practice is suggested to be a dynamic process,5 and sustainable adherence may not be self-evident without continued efforts. Sustainable change of professionals’ behaviour has the potential to result in more optimal healthcare delivery and efficiency. Not sustaining quality improvements can result in nihilistic attitudes towards future innovation. In recent years, sustainability has gained attention in healthcare. Unfortunately, the concept of sustainability is still underdeveloped.6 ,7 Some existing reviews studied sustainability from a wide healthcare perspective, including studies varying from medical care to public health. Results showed that determinants of sustainability varied widely between healthcare areas8 ,9 and suggest that partial sustainability of healthcare innovations is more common than full sustainability.10

In this systematic review, the scope of sustainability research will be narrowed to professionals’ adherence to clinical practice guidelines in medical care. The aim of the current review was to evaluate the state of the art in sustainability research and the level of sustained professionals’ adherence to guideline recommendations in medical practice more than 1 year following the cessation of the implementation project.

Methods

Eligibility criteria

Studies needed to be focused on sustainability and on clinical practice guidelines. Sustainability was described as “Sustainability of change exists when a newly implemented innovation continues to deliver the benefits achieved over a longer period of time, certainly does not return to the usual processes and becomes ‘the way things are done around here’,11 even after the implementation project is no longer actively carried out, until a better innovation comes along”.12 Studies had to include at least two measurements: one before (PRE) or immediately after implementation (EARLY POST) and one measurement longer than 1 year after active implementation (LATE POST). All activities to facilitate the adherence to clinical practice guidelines were labelled as part of the implementation project. Studies needed to be focused on professionals’ adherence to a clinical practice guideline. Studies only using self-reported adherence were excluded to reduce the chance of social desirability bias and an overestimation of results.13 Lastly, studies had to focus on medical care. Participants had to be healthcare professionals who deliver direct patient care. There were no restrictions on study design of the research articles.

Search methods for identification of studies

Electronic searches

We searched MEDLINE (OvidSP; 1946 to February 2014), CINAHL (EBSCO Host; 1982 to February 2014), EMBASE (OvidSP; February 2014), Cochrane Central Register of Controlled Trials (CENTRAL) and the Guidelines International Network (GIN) library for studies. The electronic search strategy was designed to focus on sustainability of guideline recommendations. Free-text terms and MeSH terms regarding sustainability, quality improvement, impact and guideline recommendations were used. An information expert checked the developed search strategies (see online supplementary file 1). Before final analyses, update searches were performed to identify possible additional studies (26 June 2014 and 4 August 2015).

Searching other resources

A snowball strategy was performed, in which the reference sections of reviews6–10 ,14–16 and research papers on sustainability17 ,18 were searched. Also, databases such as PubMed and the Web of Knowledge Science Citation Index were used to locate publications and publications citing the original references. The process was repeated for any new relevant publication found.

Data collection and analysis

Selection of studies

All records were merged into a bibliographic database and screened independently by two reviewers (SMCA and JJAdG) based on title and abstract. Full-text screening was performed by two reviewers (SMCA and JJAdG). Disagreement on selection was resolved in consensus meetings with a third reviewer (TvdW). Reasons for exclusion were documented during the full-text screening. If more clarification or details of a study were needed, an author was contacted. Authors of conference abstracts were emailed and were asked to send the research protocol. Duplicate papers were identified, and all papers published on one study were used for retrieving information.

Data extraction and management

Data of the methodology and results were independently extracted by two reviewers (SMCA, JJAdG), guided by a predefined data extraction form. Effective Practice and Organisation of Care (EPOC) Data Collection Checklist19 items (eg, location of care, type of targeted behaviour, implementation interventions) were integrated in the data extraction form. The data extraction form was developed by the authors and was pilot tested. The following study characteristics were recorded: study design, publication year, whether the study was executed in a single centre or in multiple centres, type of targeted behaviour, location of care, the name of the clinical practice guideline, clinical specialty, the implementation activities used and whether or not the implementation strategy was externally guided. An externally guided implementation strategy is a strategy which is lead and supported by an external expert organisation. With respect to the methodology of the sustainability evaluation, the following data were extracted: the timeframe between the end of the implementation strategy and the sustainability evaluation, the applied definition of sustainability, the data collection method, whether the evaluation was performed on patient, hospital or multiple hospital level and whether the sustainability evaluation was performed on single or multiple centre level. With respect to the outcome measures of the studies, data on the professionals’ adherence rates before, early after implementation and longer than 1 year after implementation, and the authors’ comments with respect to the sustainability of professionals’ adherence were extracted. Adherence was presented in terms of proportion of patients receiving treatment according to the clinical practice guideline recommendations. If sustainability of professionals’ adherence to a clinical practice guideline was evaluated at multiple postimplementation moments, the latest evaluation was selected as LATE POST measurement. The authors (SMCA and JJAdG) checked if updates of the clinical practice guidelines had become available in the postimplementation phase (eg, between the EARLY POST and the LATE POST measurement), which may explain reduced professionals’ adherence. Disagreement on data extraction was resolved in consensus meetings with a third reviewer (TvdW).

Assessment of risk of bias in included studies

Risk of bias assessment was independently conducted by two authors using the Downs and Black checklist for randomised and non-randomised studies.20 This is a checklist which can be used for checking the risk of bias of original research articles of various study designs. Results were interpreted under consideration of risk of bias. The assessments were also used for recommendations for further research by identifying elements of studies that can be improved in future studies. The checklist was adapted to the research question. Risk of bias of the studies was presented on reporting, external validity, internal validity (bias and confounding), power and overall level.

Analysis

The analysis was narrative. This included a summary of the methodological characteristics of the sustainability evaluations, descriptions of the level of sustainability as mentioned by the author, and the level of sustained professionals’ adherence compared with results achieved immediately after implementation. Sustainability was considered to be successful if performance in terms of professionals’ adherence was fully maintained in the late postimplementation phase. A sensitivity analysis was performed by applying a 90% instead of 100% adherence criterion of sustainability.

Results

Description of studies

For this review, 4219 items were retrieved and screened based on title and abstract, and 185 studies were assessed based on full-text reading. Figure 1 shows the study selection process as recommended by the PRISMA statement21 (see online supplementary file 2). Fourteen studies met the inclusion criteria for this review, describing 18 sustainability evaluations.22–35 Table 1 presents the characteristics of the included studies. Two publications were published before and 12 after 2000.23 ,33 In six studies, the targeted behaviour was prescribing,24 ,25 ,28 ,30 ,33 ,35 in four studies procedures,29 ,31 ,32 ,34 in three studies general management of a problem22 ,26 ,27 and in one study23 general management of a problem and prescribing. The location of care was inpatient in five studies,23 ,28 ,29 ,32 ,34 outpatient in four studies24–26 ,35 and mixed in five studies.22 ,27 ,30 ,31 ,33

Table 1

Characteristics of the included studies

Figure 1

PRISMA flow diagram.

The implementation strategy was described in 13 studies (table 2).22–27 ,29–41 According to the EPOC checklist classification, in one study,24 a single element implementation strategy was executed while in the other 12 studies a multifaceted implementation strategy was executed. Implementation activities were professional-targeted interventions (n=12) {Ament, 2014 #20; de Kok, 2010 #33; Ament, 2014 #34; Benenson, 1999 #21; Enriquez-Puga, 2009 #23; Forsner, 2010 #24; Higuchi, 2011 #25; Knops, 2010 #27; Mank, 2003 #35; Knops, 2010 #27; Lozsadi, 2006 #28; Storm-Versloot, 2012 #36; Pantle, 2009 #37; Stephan, 2006 #30; Wakefield, 1998 #31; Williams, 2003 #32; Gerber, 2013 #67; Gerber, 2014 #69}, followed by organisational interventions (n=6)22 ,23 ,26 ,33 ,34 ,36 ,37 ,39 ,40 and financial interventions (n=1).27 In six studies, the implementation strategy was facilitated by external experts.22 ,25–27 ,31 ,35 In one study, it was unclear whether the implementation strategy was externally supported.28

Table 2

Implementation strategies as described by the authors

Characteristics of the sustainability evaluations

The mean timeframe between the end of the implementation strategy and the sustainability evaluation of 13 studies was 2.6 years (minimum 1.5–maximum 7.0). The actual timeframe of one evaluation was unclear, but was at least 2 years.30 Two studies referred to a definition of sustainability.22 ,27 Eight studies used a retrospective data collection method,23–29 ,33 three studies used a prospective data collection method31 ,32 ,35 and three studies used both a prospective and a retrospective data collection method.22 ,30 ,34 Ten papers reported the level of sustained adherence of a single clinical practice guideline,22–24 ,28 ,30–35 while four reported the late postimplementation adherence of two clinical practice guidelines.25–27 ,29 Seven studies had a single-centre design23 ,28–30 ,32–34 and seven studies evaluated sustainability in multiple centres.22 ,24–27 ,31 ,35 Four out of six multiple centre studies evaluated the sustainability on multiple centre level.22 ,26 ,31 ,35 Two out of six multiple centre studies evaluated the sustainability of professionals’ adherence of two guidelines which were implemented in one centre each.24 ,27

Sustainability of changed behaviour

The level of professionals’ adherence was fully sustained in 7 out of 18 evaluations (table 3, see online supplementary file 3). The adherence was not fully sustained in six evaluations, and four evaluations showed mixed sustainability results in the LATE POST measurement compared with the EARLY POST measurement. In one study, the EARLY POST measurement was not executed, while the authors reported sustained results.28 After decreasing the sustainability level of professionals’ adherence to 90% or higher, 9 out of 18 evaluations showed sustained results, 3 evaluations showed no sustained results, 4 evaluations showed mixed results. In two evaluations, it was unclear whether the professionals’ adherence had been sustained at a level 90% or higher.

Table 3

Sustainability of professionals’ adherence to clinical practice guidelines

Five of the 10 papers that reported about a single clinical practice guideline presented sustained professionals’ adherence to clinical practice guidelines in the LATE POST measurement.22–24 ,30 ,34 One of these five papers evaluated the sustainability of a single clinical practice guideline in two centres.24 In both centres, professionals’ adherence had improved in the LATE POST measurement compared with the EARLY POST measurement. The four studies analysing the sustainability of two clinical practice guidelines showed mixed results. Two of these four studies25 ,29 presented the same level or improved adherence to one guideline and decreased adherence to the other guideline in the LATE POST measurement compared with the EARLY POST measurement. The other two of these four studies26 ,27 presented adherence results on guideline recommendation level and did not present overall adherence results on patient level. The adherence to the recommendations of the clinical practice guidelines showed decreased and improved levels in the LATE POST measurement compared with the EARLY POST measurement. In total, eight papers mentioned the term ‘sustainability’ in the conclusion (table 3).22–28 ,32 Five of these studies concluded to have sustained professionals’ adherence in the late postimplementation phase,22–24 ,28 ,32 three out of eight studies described to have a ‘mixed pattern’, ‘small desired’ or ‘almost’ sustained professionals’ adherence.25–27

Risk of bias in included studies

All studies included in the present review had a high risk of bias, following the Downs and Black assessment tool20 (table 4, see online supplementary file 4).

Table 4

Results of the risk of bias assessment

Discussion

Our review focused on the level of sustainability of implementation success in terms of professionals’ adherence. Also, this systematic review described the state of the art in sustainability research. This systematic review identified 14 studies, including 18 evaluations that investigated the sustainability of professionals’ adherence to a clinical practice guideline more than 1 year after the implementation was finished. Of 18 analyses that focused on the extent of sustained professionals’ adherence to a clinical practice guideline, seven analyses revealed fully sustained results. After decreasing the sustainability level of professionals’ adherence to 90% or higher, 9 out of 18 evaluations showed sustained results. The current review showed that the number of sustainability studies is scarce and that the studies are heterogeneous with respect to their methodology. Furthermore, almost no study analysed or reflected on the updates of the guideline in the postimplementation phase. The results of this review suggest that updates of the clinical practice guidelines may have led to a warranted decrease in the adherence to the original clinical practice guideline.

In this systematic review, information was presented about how to search for sustainability evaluations, how sustainability research is defined and about the type and the methodological quality of studies that report on sustainability. As was confirmed in another systematic review,10 the sustainability studies showed to have limited methodological rigour. Two out of 14 studies used an experimental design. The lack of identified studies in the current review suggests that most teams do not focus on the long-term performance effect of quality improvements.64 Owing to the limited number of studies focusing on this subject, the heterogeneity in studies, suboptimal reporting by authors and the revealed methodological weaknesses, no strong conclusions can be drawn based on the presented sustainability results. As also shown in other research, most sustainability studies used a single-case study design by focusing on a single type of programme or performed the evaluation at a single centre level.65 The current review showed that in only two of the studies, a reference for the definition of sustainability was used. Other studies performed a sustainability evaluation without mentioning a definition. This shows the underdeveloped field of sustainability research. Also, a variety of timeframes to study the sustainability of professionals’ adherence to clinical practice guidelines was revealed, varying from 1½ to 7 years following implementation.

Optimal adherence to a clinical practice guideline as determined during implementation is not always desired; for example, clinical experience and evidence may change. This systematic review included all research designs and seems to be the first review with respect to sustainability of professionals’ adherence to clinical practice guidelines to date. Other reviews focused on healthcare from a broad perspective including multiple healthcare fields10 or reviewed studies performed specifically in public health.6 ,9 The sustainability of a health programme in public health may be influenced by other determinants than the sustainability of a clinical practice guideline in medical care. Also, the concept of the sustainability may differ between healthcare fields. For example, in public health sustainability of a health programme may be successfully sustained if health outcomes, for example, changed lifestyle, are maintained and financial support is still available.6 ,65 In medical care, the primary focus is on the quality and safety of care which is supposed to be captured in clinical practice guidelines. Owing to the specific focus on clinical practice guidelines in the current review, mainly other studies were included compared with the existing sustainability reviews.6–10 ,14

Strengths and weaknesses

As yet, the term ‘sustainability’ is not consistently used for this area in the broader medical field, which presents a limitation to the electronic search strategy. The topic is not well indexed in electronic databases, and text word searches are prone to high recall and low specificity. However, it is likely that the use of a broad variety of search terms that covered sustainability has downsized the number of relevant studies missed and is a strength of the review.

In this systematic review, sustainability was assessed as successful if performance in terms of professionals’ adherence was fully maintained in the late postimplementation phase. This definition of sustainability may be too pragmatic as it could be undesirable to fully sustain the professionals’ adherence in the late postimplementation phase. Therefore, a sensitivity analysis was performed to analyse the sustainability at a level of 90% or higher. However, as mentioned before, a limitation of the review is the high risk of bias of all studies included. The majority of the studies used a retrospective data collection method. Nevertheless, results were interpreted under consideration of risk of bias, and the assessments were also used for recommendations for further research by identifying elements of studies that can be improved in new studies. Also, the question is what the best method is for evaluating sustainability. For example, retrospective data may be desired to prevent a Hawthorne effect when studying routine practice.

The results of the current review show more studies with sustained professionals’ adherence than might be expected without continuing efforts and support to promote the level of sustained adherence in the postimplementation phase. Possibly, studies with unfavourable results may not be published or unsuccessful implementation projects may not be evaluated, leading to an under-representation of the true amount of work carried out in the field.66 ,67

Sustainability of professionals’ adherence may be influenced by the perceived quality of the guideline. However, we were not able to analyse the quality of the guidelines given the limited information in the manuscripts and the information on the internet on the specific guidelines. More information about the quality of the guidelines in sustainability evaluations may be helpful to analyse the sustainability of the guideline. Also, the potential effect of the specific implementation strategies was not analysed as part of the systematic review. Professionals’ adherence is an outcome measure used in implementation science, and it captures the behaviour change as a result of implementation strategies. The type of implementation strategy may have had an effect on the sustainability of the implementation results. The studies included used various implementation strategies and implemented different clinical practice guidelines.

Implications for practice

The current review showed that the level of the sustainability of professionals’ adherence to clinical practice guidelines varies on case study level and drops in more than half of the studies. Owing to the lack of sustainability research we think that sustainability failure as presented in this study is an underestimation. Unfortunately, implementation projects are primarily focused on short-term actions and short-term effect.64

Future research

This review complements the existing sustainability research by focusing on sustained professionals’ adherence in medical practice. The current review showed that not many studies reported data on the sustainability of professionals’ adherence to clinical practice guidelines. Also, no strong conclusions can be drawn due to the high risk for bias and the heterogeneity of the studies. As shown in previous research, structural methods for sustainability evaluations are lacking.10 ,68 Furthermore, future implementation and sustainability evaluations may include information about the quality of the clinical practice guideline, such as described in the AGREE instrument.69 More sustainability evaluation research and methodological guidance is needed to make future sustainability research more robust and generalisable and may be helpful in creating a general sustainability language.

Conclusion

This systematic review identified, reported and analysed studies that evaluated the level of sustainability of professionals’ adherence to guideline recommendations in medical practice more than 1 year following the cessation of the implementation project.2 Seven out of 18 evaluations showed sustained professionals’ adherence on average 2.6 years after implementation.1 Owing to the limited number and the lack of methodological quality of the identified studies, no firm conclusion about the sustainability of professionals’ adherence to guideline recommendations in medical practice can be drawn. More sustainability evaluations, methodological sustainability studies and reviews are needed in order to develop a general framework for sustainability measurement and to facilitate uniform language and communication within the sustainability science.

Acknowledgments

The authors would like to thank Karin Vaessen for the assistance in retrieving full-text papers.

References

View Abstract

Related Data

Footnotes

  • Twitter Follow Stephanie Ament at @stephanieament

  • Contributors All authors made substantial contributions to conception and design, analysis and interpretation of data. SMCA and JJAdG were responsible for the data collection, supervised by JMCM, CDD, TvdW and JK. SMCA was responsible for the draft of the manuscript and all authors were also involved in editing the manuscript. All authors have given final approval of the version to be published. All authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. SMCA is guarantor.

  • Funding This work is produced by SMCA under the terms of a doctoral research training fellowship issued by ZonMw, the Netherlands Organisation for Health Research and Development (grant number 171103004).

  • Competing interests All authors have completed the Unified Competing Interests form at http://www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author). SMCA is a health scientist, main investigator and PhD candidate. JJAdG is a medical doctor and PhD candidate. TvdW is professor implementation of evidence in healthcare. SMCA, JJAdG and TvdW are employed by Maastricht University. JMCM is senior research associate at Maastricht University Medical Centre. CDD is professor health technology assessment of clinical interventions at Maastricht University Medical Centre. JK is professor of systematic reviews in healthcare at Maastricht University and founded Kleijnen Systematic Reviews (KSR) Ltd in 2005. KSR is an independent research company that produces and disseminates systematic reviews, cost-effectiveness analyses and health technology assessments of research evidence in healthcare. All authors read and approved the final manuscript.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Data sharing statement Additional data can be accessed via the Dryad data repository at http://datadryad.org/ with the doi:10.5061/dryad.cr020.

Request permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.