Article Text

PDF

Appraising the quality standard underpinning international clinical practice guidelines for the selection and care of vascular access devices: a systematic review of reviews
  1. Ian Blanco-Mavillard1,
  2. Miguel Angel Rodríguez-Calero2,
  3. Enrique Castro-Sánchez3,
  4. Miquel Bennasar-Veny4,
  5. Joan De Pedro-Gómez5
  1. 1 Quality, Teaching and Research Unit, Hospital de Manacor, Manacor, Spain
  2. 2 Health Care Office, Balearic Islands Health Service, Palma, Spain
  3. 3 NIHR Health Protection Research Unit in Healthcare Associated Infection and Antimicrobial Resistance, Imperial College London, London, UK
  4. 4 Department of Nursing and Physiotherapy, Universitat de les Illes Balears, Palma, Spain
  5. 5 Evidence, Lifestyles and Health Research Group, Research Institute of Health Sciences, Universitat de les Illes Balears, Palma, Spain
  1. Correspondence to Dr Miquel Bennasar-Veny; miquel.bennasar{at}uib.es

Abstract

Objective Catheter-related bloodstream infections are one of the most important adverse events for patients. Evidence-based practice embraces interventions to prevent and reduce catheter-related bloodstream infections in patients. At present, a growing number of guidelines exist worldwide. The purpose of the study was to assess clinical practice guidelines for peripheral and central venous access device care and prevention of related complications.

Design Systematic review of clinical practice guidelines: We conducted a search of the literature published from 2005 to 2018 using Medline/PubMed, Embase, CINAHL, Ovid, ScienceDirect, Scopus and Web of Science. We also evaluated grey literature sources and websites of organisations that compiled or produced guidelines. Guideline quality was assessed with the Appraisal of Guidelines for Research and Evaluation, Second Edition tool by three independent reviewers. Cohen’s kappa coefficient was used to evaluate the concordance between reviewers.

Results We included seven guidelines in the evaluation. The concordance between observers was substantial, K=0.6364 (95% CI 0.0247 to 1.2259). We identified seven international guidelines, which scored poorly on crucial domains such as applicability (medium 39%), stakeholder involvement (medium 65%) and methodological rigour (medium 67%). Guidelines by Spanish Health Ministry and UK National Institute for Health and Care Excellence presented the highest quality.

Conclusions It is crucial to critically evaluate the validity and reliability of clinical practice guidelines so the best, most context-specific document is selected. Such choice is a necessary prior step to encourage and support health organisations to transfer research results to clinical practice. The gaps identified in our study may explain the suboptimal clinical impact of guidelines. Such low adoption may be mitigated with the use of implementation guides accompanying clinical documents.

  • clinical practice guideline
  • implementation
  • evidence-based nursing
  • adverse events
  • vascular access
  • quality appraisal

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

Statistics from Altmetric.com

Strengths and limitations of this study

  • Methodological quality evaluation of vascular access devices clinical guidelines is lacking.

  • Despite recommendations being based on subjective assessments, evaluators’ concordance using the Appraisal of Guidelines for Research and Evaluation, Second Edition tool was substantial.

  • We identified low scores on crucial guideline domains such as applicability, rigour and stakeholder involvement.

  • The gaps identified may explain the suboptimal impact of guidelines in practice and the need for implementation guides accompanying the clinical documents.

Introduction 

Vascular access devices (VADs) are the most commonly used invasive devices in hospitals worldwide.1 These devices expose patients to multiple complications related to their insertion, maintenance and management.1 2 Catheter-related bloodstream infections (CRBSIs) are one of the adverse events with worst impact for patients3 in terms of morbidity and mortality.4 CRBSIs account for 40.3% of all bloodstream infections, of which are associated with the use of central venous catheters (CVCs) and peripheral venous catheters (PVCs).5 The incidence of central line-associated bloodstream infections is 5.7 per 1000 catheter-days,6 and 0.1%, 0.5 per 1000 catheter-days in PVCs.7 CRBSIs can prolong length of hospital stay2 5 and present an attributable mortality rate of up to 25%.8 9 The approximate average cost per episode is US$45 000 and resulting in US$2.3 billion of unnecessary expenditure per year in the USA.3

Nowadays, health organisations benefit from the advances in knowledge about service delivery generated by researchers resulting in optimal care for patients and citizens. The aim of such evidence-based practice is to offer care experiences that are informed by the best available scientific knowledge, clinical expertise and user preferences.10 11 To assist the decision-making of health professionals and patients about appropriate healthcare interventions in specific clinical circumstances,12 clinical practice guidelines (CPGs) are developed. These CPGs are based on empirical evidence, explicit in their methods of development, critically reviewed by experts, free of conflicts of interest and with specific and unambiguous recommendations.

CPGs are not however exempt from problems.13 In recent years, the number of these CPGs has grown dramatically and thus the volume of evidence proven to be unmanageable and of variable quality.14 Another concern may refer to the inclusion of studies with statistical yet marginal or insignificant clinical significance.15 Additionally, there is frequent tardiness in the implementation of the recommendations within the CPGs, probably fuelled by perceptions of clinical judgement as the main element in clinical decision-making.16 These facts can ultimately weaken the credibility of CPGs and therefore increase the difficulty of their implementation.

To date, few systematic attempts have been made to compare the quality of CPGs that provide recommendations for the care and prevention of adverse events associated with vascular catheter in adults.17 Therefore, the purpose of this study was to assess the quality of such CPGs using the Appraisal of Guidelines Research and Evaluation, Second Edition (AGREE II) tool18 and analyse methodological factors related to the process of CPGs development on effective knowledge mobilisation.

Methods

Search strategy

Two researchers (IB-M, MAR-C) conducted the search from March to April 2018. Searches were constructed using relevant medical subject headings (MeSH): ‘Complications,’ ‘Catheter-Related Infections,’ ‘Catheter obstruction,’ ‘Phlebitis,’ ‘Extravasation of Diagnostic and Therapeutic Materials,’ ‘Peripheral Catheterization,’ ‘Central Venous Catheterization,’ ‘Nursing care,’ ‘Guideline’ and ‘Evidence-Based Practice’ with variations of keywords terms ‘Adverse events’ and ‘Implementation.’ The following databases and bibliographic resources were searched: Medline/PubMed, Embase, CINAHL, Ovid, ScienceDirect, Scopus and Web of Science. We also explored the websites of organisations that compile or produce guidelines such as Joanna Briggs Institute (JBI, Australia), National Health and Medical Research Council (NHMRC, Australia), Registered Nurses Association of Ontario (RNAO, Canada), The National Institute for Health and Care Excellence (NICE, United Kingdom), National Guideline Clearinghouse (NGC, US), US Centers for Disease Control and Prevention (CDC, US), Scottish Intercollegiate Guidelines Network (SIGN, Scotland) and Library Clinical Practice Guidelines of the National Health Services (GuiaSalud, Spain). In addition, we evaluated grey literature available in professional societies and groups such as the Infusion Nurses Society. The search considered evidence from January 2005 to April 2018 to avoid the inclusion of CPGs with outdated recommendations online supplementary appendix 1.

Supplementary file 1

Eligibility criteria

We used the definition of CPG developed by the Institute of Medicine.12 We selected CPG with recommendations developed systematically and including a grading system related to the quality of evidence or the strength of the recommendations to assist healthcare professionals and patients in making decisions about appropriate healthcare, at the prevention of complications, management and care on CRBSIs associated with VAD in hospital adults. Expert consensus statements and specifics CPGs of other pathologies, such as cancer, radiology or renal were excluded, as its target population was very specific.

Data management and extraction

IB-M and MAR-C independently conducted the selection of studies, with disagreements resolved after discussion with a third researcher (JDP-G). The titles and abstracts obtained were then scrutinised for selection of CPGs. Only full-text studies in English or Spanish were included in the review.

IB-M and MAR-C independently extracted the following information: lead author, developing organisation, country, year of publication, recommendations, quality of evidence and strength of recommendations. IB-M developed a data extraction form based on the domains specified in the AGREE II instrument for the evaluation of CPGs online supplementary appendix 2.18 The extracted data were synthesised and similarities and differences compared.

Supplementary file 2

We reported our findings in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-analyses guidelines (PRISMA).19 The documents to be evaluated were read randomly by the reviewers, to avoid biasing the scores due to familiarity with the previously evaluated CPGs.20

Data analysis

Three reviewers independently assessed the CPGs using the AGREE II instrument.18 This tool use methodological quality criteria to examine the variability of quality, rigour and transparency used during the development of CPGs. IB-M calculated the final quality score according to the protocol of the instrument and the scores provided by the three reviewers for the items within each domain. In addition, a median score and rank was obtained for each domain of the instrument. Cohen’s kappa coefficient was used to evaluate the concordance between reviewers on the item ‘global guide evaluation’. The following labels will be assigned to the corresponding ranges of kappa: Poor (<0.00), Slight (0.00–0.20), Fair (0.21–0.40), Moderate (0.41–0.60), Substantial (0.61–0.80) and almost Perfect (0.81–1.00).21 The Epidat V.4.1 software was used for statistical analysis.22

Patient and public involvement

No patients or members of the public were involved in the development of the research question, nor the design or the conduct of the study, nor the interpretation and writing up of results.

Results

A total of 1447 citations and abstracts were identified through the database search, with eight further CPGs obtained from guideline development organisations. Duplicates were deleted and excluded. The reviewers also discarded six summaries of CPG; a standard of practice from the Infusion Nurses Society; a Best Practice Sheet of Joanna Briggs Institute; two CPGs for the maintenance of intravascular catheters in children; four oncology CPGs; one Anaesthesiology CPG; two infusion CPGs; one radiology CPG; two nutrition CPGs and two renal CPGs; and one older version online supplementary appendix 3. The selection process is summarised in the PRISMA flow diagram in figure 1.

Supplementary file 3

Figure 1

The Preferred Reporting for Systematic Reviews and Meta-analyses diagram representing the systematic literature search. AGREE II, Appraisal of Guidelines for Research and Evaluation, Second Edition.

Seven CPGs were finally included in the synthesis of evidence and evaluated with the AGREE II instrument. The characteristics of the selected CPGs are presented in table 1. The CPGs were developed by National Institute for Health and Care Excellence,23 Registered Nurses Association of Ontario,24 25 Ministry of Health26 (Spain), Centres for Disease Control and Prevention/Healthcare Infection Control Practices Advisory Committee,27 The Society for Healthcare Epidemiology of America/Infectious Diseases Society of America28 and Department of Health of Queensland Government29 (Australia).

Table 1

Characteristics of clinical practice guidelines (CPG)

The CPGs were published between 2008 and 2014: three from 2014,23 26 28 one from 2013,29 one from 201127 and two from 2008.24 25 There was heterogeneity in the objectives of the different documents. For example, the epic323 aimed to describe clinically effective measures to be used by health professionals in hospital infection prevention; however, in three24 25 27 the specified objective was the provision of recommendations to professionals to help in decision-making about peripheral and central venous access. Recommendations for CVC alone were detailed in one paper,28 with another focusing only on peripherally inserted central catheters.29

The scores for each domain evaluated with the AGREE II tool are presented in table 2. Domain 1 (‘Scope and purpose’) relates to the general scope and purpose of the CPGs, specific clinical questions to be addressed and target population. The median score was 74% (range 37%–100%), with five documents scoring above 70%.

Table 2

Results obtained from clinical practice guideline (CPG) using the Appraisal of Guidelines for Research and Evaluation, Second Edition instrument

The domain 2 (‘Stakeholder involvement’) reflected whether the guideline development process included input and involvement from appropriate stakeholders. The median score obtained was 65% (range 25%–91%), with three guidelines scoring above 70%. Additionally, the item 5 (‘The views and preferences of the target population have been sought’) in the AGREE II instrument received the lowest score.23–26

In terms of ‘Rigour of development’ (domain 3), the median score across the category was 67% (range 9%–87%), with two guidelines scoring above 70%. This section focused on the methods in place for gathering and synthesising evidence during the development or updating of the document. Most guidelines described the process of development, but they did so in varying levels of detail. We found poor description of the process followed to search and review evidence, and only one document described the approach to follow regarding updates of the information.

The domain 4 (‘Clarity of presentation’) centred on how specific and unambiguous the recommendations were, including clearly presented options for the management of the health condition and with easily identifiable key recommendations. This domain was the best evaluated, obtaining a median score of 84% (range 81%–91%), and with all CPGs exceeding the 70% score threshold.

‘Applicability criteria’ (domain 5) referred to facilitators and barriers to the implementation of recommendations within the guidelines, the strategies for implementation of such recommendations and the likely resource implications associated. The median score obtained by the CPGs on this domain was 39% (range 14%–88%) with three above 70%. The CPGs were rated negatively as they did not take into account possible enabling factors for implementation, strategies to improve adoption, resource considerations or availability of monitoring indicators.

Finally, the domain 6 (‘Editorial independence’) aimed to ensure a lack of bias in the development of the guideline. The domain was relatively well scored, with a median across CPGs of 78% (range 19%–88%) and five above 70%. Most CPGs did provide statements regarding funding, potential influence on the recommendations endorsed or relevant conflicts of interest.

Overall assessment

The average score across the CPGs was moderate (median 62%), scores ranged from 38%29 to 90%.26 The best results following the global evaluation of the CPGs and their recommendations for practice of care were obtained by CPGs of the Ministry of Health in Spain,26 and epic3,23 being strongly recommended. In addition, three CPGs24 25 28 have been recommended, which would needed modifications on the domains of rigour and applicability28; process of actualisation24 25 and two CPGs27 29 have not been recommended for use of clinical practice (table 2).

Agreement among reviewers

The agreement among the three independent reviewers is presented in table 3. The overall agreement was substantial (K=0.6364; 95% CI 0.0247 to 1.2259), with perfect intracategory concordance (K=1.0) for guidelines classified as ‘recommendable’. The evaluators demonstrated a moderate agreement (K=0.5556) for those guides classified as ‘recommendable with modifications’ coinciding with a moderate quality of evaluation. However, the agreement was small (K=0.2) when the quality of the documents was low and their use in clinical practice was not recommended.

Table 3

Kappa concordance index among observers for recommendation of CPGs

Discussion

Our study focused on international CPGs aimed at facilitating the management and care of VAD and preventing complications associated on CRBSIs with their use in adults. We compared the quality of existing guides using the AGREE II tool and analysed relevant methodological factors related to the process of guideline development. Many of the CPGs included in this study scored highly in some, but not all, domains. Overall, the quality of the CPGs examined was moderate. More than 70% of these presented specific and well-described recommendations including different options for clinical management, specific objectives, health aspects to be addressed and the target population. The best scores were received by the domains ‘Clarity of presentation’ and ‘Scope and objective’.

However, we identified low scores on other equally crucial domains for effective implementation including ‘stakeholder involvement’, ‘methodological rigour’ and applicability’. These findings are consistent with other studies of similar methodology.20 30 Regarding stakeholder involvement, the guidelines did not report on whether the views of patients or users were sought during the process of guideline development. Such absence may be due to a number of factors including lack of recognition and value of the perceptions of the main actors targeted by the guides.13 CPGs tend to centralise knowledge from experts without the consideration of patient or user involvement in guideline development, assuming that the patients’ care may have a less active role.31–33

In terms of methodological rigour, the evidence suggests that the quality and accuracy of CPG elaboration can be highly variable,34 even when dealing with the same subject. These low scores may be due to excessively rigorous criteria for the inclusion of clinical trials based on organisational or behavioural interventions. CPGs evaluated offer little or no information on the strengths and limitations of the evidence used, or the approach and timing of updates or consideration of emerging evidence.

Finally, a lack of consideration towards applicability criteria may hamper the implementation of guideline recommendations in clinical practice, this aspect is as important as the methodological rigour and stakeholder involvement in effective knowledge mobilisation. Currently, there is a fundamental problem that may be influencing the difficulty of implementing recommendations. CPGs only include studies that provide results about what needs to be done and are collected in the best available evidence. However, the use of realist revisions on what has worked, and for whom could help in optimising the implementation process.35 In the last decade, we have observed the growth in the volume of evidence to be appraised to a point of unmanageability, especially clinical guidelines, obfuscating evidence integration.14 Some CPGs include studies with statistically significant benefits, but which may be marginal in clinical practice.13 Ultimately, the low scores in the domains referred along with the issues cited above may explain the low applicability of guidelines and the reported low adherence of recommendations across many clinical areas and settings.36

Clinical implementation is a complex, multifaceted phenomenon37 which requires a deep understanding of decision-making processes and active strategies within organisations.38 Such strategies ought to incorporate mechanisms to influence how tacit and explicit knowledge is constructed and internalised in routine practice.39 Decision-making is not only achieved through a careful selection of information based on a defined evaluation of possible outcomes40 but depends as much on multiple human factors. Scientific evidence should receive significant, but not necessarily predominant, attention since optimal decisions would require the integration of such best evidence with clinical expertise together with the preferences of users,10 41 leading to the development and use of ‘mindlines’.16 39 42 Further, knowledge transfer must be guided by conceptual models that articulate proposed strategies to effect behavioural change and the factors that may influence adoption of best clinical practice. At present, models of proven efficiency43–47 have been instrumental to translate the results produced by evidence syntheses onto the necessary adherence by professionals, thus facilitating dynamics of knowledge transfer and mobilisation.48 We endorse the need to supplement CPG with implementation guides that facilitate and ensure effective implementation process to fidelity the best available evidence. We endorse the need to supplement CPG with implementation guides that facilitate and ensure effective implementation process to fidelity the best available evidence, including result indicators associated with prevention and control of infections related to VAD.49 50 Likewise, the identification of barriers and constraints at institutional and individual level should be the first step,51 for the inclusion of strategies that promote fidelity52 to recommendations through multicomponent and multimodal interventions53 54 formed by facilitation of evidence,46 55 the use of e-learning in health professionals56 and health information of the users.57 These multimodal interventions may lead to notable improvements in the clinical outcomes of users.2 58

Limitations

This review presents limitations, chiefly related to the search process and inclusion criteria of guidelines. For example, we excluded guidelines not published in English and Spanish. Regarding the quality evaluation of the guidelines, the AGREE II instrument, despite its validity, does not offer guidance to interpret the results. Our criteria for endorsing some guidelines for practice of care were based on the global score across all domains, being a subjective perception to each evaluator. However, we performed a concordance analysis among the evaluators’ findings for the recommendation of the guidelines. Further, AGREE II only assesses the quality of guidelines’ structure and content but does not delve into the practical implications or quality of the recommendations.

Conclusions

Our findings indicate that quality of CPGs reviewed for the prevention of complications, management and care associated with VAD was moderate, being substantial the overall agreement among reviewers. There is a need to incorporate mechanisms of critical evaluation about the validity and reliability of selected guidelines within environments of practice, as a prior and essential requisite to knowledge mobilisation. The gaps identified with low CPG scores in critical domains for knowledge transfer may explain the suboptimal clinical impact of guidelines on healthcare practice. This is evidenced by the partly adherence of healthcare professionals to recommendations.36 59 We endorse the need to supplement clinical practice documents with implementation guides that ensure effective implementation.

References

  1. 1.
  2. 2.
  3. 3.
  4. 4.
  5. 5.
  6. 6.
  7. 7.
  8. 8.
  9. 9.
  10. 10.
  11. 11.
  12. 12.
  13. 13.
  14. 14.
  15. 15.
  16. 16.
  17. 17.
  18. 18.
  19. 19.
  20. 20.
  21. 21.
  22. 22.
  23. 23.
  24. 24.
  25. 25.
  26. 26.
  27. 27.
  28. 28.
  29. 29.
  30. 30.
  31. 31.
  32. 32.
  33. 33.
  34. 34.
  35. 35.
  36. 36.
  37. 37.
  38. 38.
  39. 39.
  40. 40.
  41. 41.
  42. 42.
  43. 43.
  44. 44.
  45. 45.
  46. 46.
  47. 47.
  48. 48.
  49. 49.
  50. 50.
  51. 51.
  52. 52.
  53. 53.
  54. 54.
  55. 55.
  56. 56.
  57. 57.
  58. 58.
  59. 59.
View Abstract

Footnotes

  • Contributors All authors made significant contributions to the manuscript. IB-M and MAR-C collected the data. IB-M, MAR-C, JDP-G and MB-V analysed the data. IB-M, MAR-C, EC-S, MB-V and JDP-G drafted and critically revised the manuscript for important intellectual content. All authors read and gave final approval of the version of the manuscript submitted for publication.

  • Funding Enrique Castro-Sánchez is affiliated with the National Institute for Health Research Health Protection Research Unit in Healthcare Associated Infections and Antimicrobial Resistance at Imperial College London in collaboration with Public Health England and Imperial College Healthcare NHS Trust. EC-S has received an Early Career Research Fellowship from the Antimicrobial Research Collaborative at Imperial College London and recognises the support of the Florence Nightingale Foundation as well as the NIHR Imperial Patient Safety Translational Research Center.

  • Competing interests The opinions and points of view included in the article only represent the opinion of the authors and not the NHS, the NIHR, the Department of Health or Public Health England.

  • Patient consent Not required.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Data sharing statement The manuscript was a systematic review of published studies; the full references of these studies have been provided in the reference list.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.