Statistics from Altmetric.com
Strengths and limitations of this study
This is the first attempt to evaluate the methodological quality of clinical practice guidelines (CPGs) for the management of hypertension in China using the Appraisal of Guidelines, REsearch and Evaluation II (AGREE II) instrument.
The results highlight the strengths and weaknesses of the existing CPGs, providing suggestions on how improvements can be made in the future.
The main limitation of this study is that the methodological quality assessment was based on the information about the assessment items the individual CPGs reported, which may not actually reflect the construction process.
Hypertension is an important public health challenge, and is the leading cause of death and disability in China. It has been estimated that one in six Chinese adults have hypertension but that only one-quarter are aware of their condition.1 Hypertension is the most powerful risk factor for cardiovascular disease.2 In addition, increased blood pressure is the leading preventable risk factor for premature mortality in the general population in China.3 Despite increased rates of blood pressure-lowering treatment, only a few patients have their hypertension effectively controlled.4 The economic burden of hypertension is heavy due to both direct healthcare costs and indirect costs from the cardiocerebral vascular complications.
Clinical practice guidelines (CPGs) for hypertension have been developed to help optimise the management of the condition and are thought to be capable of improving the quality, appropriateness and cost-effectiveness of patient care.5 The intention of CPGs is to reduce the gap between research and practice, and to provide professionals with recommendations based on the best currently available evidence on how to manage health conditions.6 ,7 The use of CPGs is recognised as an important component of hypertension management, including support for the use of interventions that are of proven benefit and enhanced awareness of ineffective methods. However, a large number of practice guidelines have been produced by numerous organisations in China over the past 10 years, and the increasing interest in the development of CGPs for hypertension has been accompanied by growing concerns about the variations in the recommendations and qualities among the guidelines.8–10 The methodological quality is of great importance, and it is thus critical to assess CPGs for hypertension using a well-defined process. A high methodological quality CPG development process is more likely to yield a CPG that contains relevant and appropriate recommendations.11
Several guideline appraisal tools have been developed to assess the methodological quality of guidelines.12 The Appraisal of Guidelines for REsearch & Evaluation (AGREE) Instrument, developed by an international group of researchers from 13 countries, has been validated and accepted to address the issue of variability and to appraise the methodological quality of CPGs.13–15 The original AGREE Instrument, which was released in 2003, has been refined to improve the usability and methodological properties, which resulted in the AGREE II reported in 2009.16 The AGREE II instrument is valid and reliable, with a 23-item tool comprising six quality domains, followed by 2 final overall assessment items that require the appraiser to make overall judgments of the CPGs and to reflect on the development process. The aim of the present systematic review was to evaluate the methodological quality of CPGs for hypertension, developed in China, using the AGREE II instrument.
According to a prespecified search strategy, relevant guidelines were identified through searches of the CBM (Chinese Biomedical Literature Database), Wanfang Database, CNKI (Chinese National Knowledge Infrastructure) and CGC (China Guideline Clearinghouse) websites up to March 2014. Reference lists of all relevant guidelines were manually scanned and, in addition, Google Scholar was searched for additional information.
We employed a combined search of MeSH terms and free-text words, and the following search terms, “hypertension” or “high blood pressure” and “guideline” or “recommendation” or “consensus”, were used. Two investigators independently screened the titles and abstracts of the literature for potentially relevant guidelines.
Inclusion and exclusion criteria
All CPGs developed in China about the management of hypertension in adult patients were eligible for inclusion in the present study. Single-author overviews, secondary or multiple publications, editorials, translations and short summaries, were specifically excluded. The latest version of the guideline was identified for assessment and, if several publication forms of one guideline existed, only the form that included the greatest detail on the methodology used for the guideline development was assessed.
The methodological quality of existing CPGs for hypertension was evaluated using the AGREE II instrument, which consists of 23 items organised within six domains: scope and purpose, stakeholder involvement, rigor of development, clarity of presentation, applicability and editorial independence.
When using this instrument, each item is ranked on a 7-point scale ranging from 1 (strongly disagree) to 7 (strongly agree). A score of 1 is given when there is no information on that item or if it is poorly reported. A score of 7 is given if the quality of reporting is excellent and when full criteria have been met.
The score for each domain is obtained by adding together all of the scores of the individual items in a domain and then standardising them as follows:
Although using a single quality score is not encouraged by the AGREE II, an overall assessment is included. The final component of the AGREE II instrument involves a recommendation regarding the use of the guidelines in practice as “Yes, to recommend this guideline (Y)”; “Yes, to recommend but with modification (Y, but)” and “No, not to recommend (N)”. For each guideline that was given a recommendation of (Y), the overall domain scores were ≥60% for all six domains. For guidelines that were given a recommendation of (N), the overall domain scores were <30% for at least three domains. For the guidelines that were given a recommendation of (Y, but), the overall domain scores were ≥30% for at least three domains, while at least one domain had a score of <60%.17 ,18
Two investigators with a full understanding of the AGREE II manual independently assessed the identified CPGs. Prior to evaluating the guidelines included in the review, each investigator rated a superseded guideline, and then the rates were compared among reviewers, discrepancies were discussed, and a consensus was reached about the interpretation of each question.19 The κ score was adopted as a measure of the agreement between the two appraisers’ ordinal item assessment (strongly agree/agree vs strongly disagree/disagree). Although there are no absolute cut-offs for interpreting κ coefficients, a κ>0.8 is generally considered to be acceptable.20 ,21
Search and description of studies
A total of 7845 articles were identified, 101 of which were considered to be potentially relevant and 17 of which proved eligible for final evaluation, as summarised in figure 1. The general characteristics of the assessed guidelines are shown in table 1.22–38 The publication dates ranged from 2001 to 2011. The number of references cited ranged from 0 to 218, with five guidelines not reporting any references,26 ,31 ,34 ,35 ,38 four of which were developed on the Chinese mainland.31 ,34 ,35 ,38 The majority of the reviewed CPGs (13 guidelines) were published in Chinese, while four guidelines were published in English,26–28 ,37 two of which were developed in Hong Kong26 ,37 and two in Taiwan.27 ,28 A total of four CPGs23 ,29 ,33 ,36 stated that they received drug company sponsorship, but only one of these guidelines declared that the views of the funding bodies did not influence the recommendations, and a declaration of the conflicts of interest of the guideline developers was not provided for 13 CPGs.
Six of the 17 CPGs assessed covered the management of hypertension in general,23 ,25–28 ,37 with 1 focused on prevention28 and 1 focused on emergency hypertension.25 The others were specific to the pharmaceutical treatment of hypertension, one of which concerned Traditional Chinese medicine,32 one ethnomedicine,31 one the elderly,33 and one, home medication.38
The majority of CPGs did not describe the level of evidence they cited or grade the recommendations. Two guidelines developed on the Chinese mainland coded the level of evidence (I, IIa/IIb, III) on the basis of the study design, without linking these codes to the recommendations (A, B, C).29 ,36 Only one guideline developed in Hong Kong classified the level of evidence and explicitly linked the evidence to the recommendations using the Scottish Intercollegiate Guidelines Network (SIGN) classification.26
Guideline quality scores
The scaled AGREE II domain percentages for all 17 CPGs are presented in table 2. Concerning the ‘scope and purpose’ and ‘clarity of presentation’ of the guidelines, the mean score for both was 41%, with a range of 6–78% and 6–67%, respectively. These two domains were the most adequately addressed, with scores above 60% in 4 of the 17 clinical guidelines.
Less well addressed was the ‘stakeholder involvement’ domain, for which the average domain score was 28%, with the extremes being 3–67%, and only 1 of the 17 guidelines scored above 60%. CPGs performed similarly in the domain of ‘applicability’, with one scoring above 60%, while the average score was 20% and varied from 0% to 69%, including one CPG that scored 0%.
The last two domains were the least well addressed. On average, the score for the ‘rigor of development’ domain was 18%, with scores ranging between 1% and 36%. Finally, the scores for the ‘editorial independence’ domain ranged from 0% to 46%, with an average domain score of 16%, including five CPGs that scored 0%, making this domain the least well addressed.
The final evaluation of the guidelines summarises the overall opinion of whether or not the guideline should be recommended for clinical use. Eight out of the 17 CPGs can be recommended, with modification, for use based on the AGREE II instrument.
There has been an increase in the development of CPGs in the field of hypertension; however, to the best of the authors’ knowledge, the present systematic review is the first attempt to evaluate the methodological quality of CPGs for the management of hypertension in China using the AGREE II instrument. It should be noted that the AGREE II instrument reflects how well the development process is reported but does not assess the content of the CPG or the quality of evidence used to formulate the recommendations, which is a common deficit of all existing appraisal tools.12
We found that there was considerable variation in the assessed CPGs in terms of the number of pages, authors and references. It was found that some CPGs comprised just one page, without any references or authors’ names, and many CPGs did not provide a statement about any potential conflicts of interest. It has been stated that if a conflict of interest is unavoidable, it should be disclosed. For these guidelines with no declaration of the conflicts of interest, there may have been no conflicts, but because the authors did not state that there was ‘no conflict of interest’, the potential for conflict is unclear. A few CPGs described the level of evidence, but without linking it to the recommendations. The differences and shortcomings in the classification and grading system were confusing.
Our results demonstrated that the methodological quality of CPGs for hypertension is undesirable in general and varies from guideline to guideline according to the different domains, with the ‘clarity of presentation’ and ‘scope and purpose’ receiving the highest scores, the ‘rigor of development’ and ‘editorial independence’ scoring the lowest, and ‘stakeholder involvement’ and ‘applicability’ being between these domains. The assessed guidelines performed best in the domains of ‘clarity of presentation’ and ‘scope and purpose’, which was consistent with the findings of a systematic review for recent hypertension CPGs,39 five priority diseases’ CPGs within Southern Africa40 and endocrine CPGs in North America.41 Despite the fact that these two domains scored highest among the six, the scores were still much lower than the global average scores, and great improvements that include clearly summarising and focusing on the issues that are most related to the patients and physicians are needed in the future in China.42
The CPGs for hypertension developed by the Writing Group of 2010, Chinese Guidelines for the Management of Hypertension in 2010, received the highest scores on average, but these were generally still less than 60%. Further assessment indicated that it failed to provide sufficient details about how the final recommendations were formulated.
The most poorly undertaken domain, ‘editorial independence’, is meant to assess the possible conflicts of interest of the author(s) and whether the guideline was developed independently of the funding source. European guidelines on a range of topics published from 2000 to 2007 similarly found that most guidelines scored low on ‘editorial independence’,43 while hypertension CPGs published from 2006 to 2011 confirmed that great progress had been made in the ‘editorial independence’ domain.39 Poor scores for hypertension CPGs developed in China highlight the need to improve the development process under an explicit conflict of interest. Insight into the ‘editorial independence’ increases the transparency of the guideline development process. The cited reasons for the low scores included poor reporting on whether a conflict of interest was assessed during the process of development of the guidelines or, if it was assessed, how it was addressed.44 However, low scores can also be explained by poor reporting. When no information is provided about the assessment item, the corresponding score will be low, which may not mean that the guideline was inappropriately developed, but that there was inadequate information provided.
All of the guidelines performed poorly with respect to ‘rigor of development’, which is considered to be crucial for the guideline quality by ensuring that a rigorous process was used to judge the underlying quality of evidence on which the guidelines were developed. A further analysis found that seven CPGs with the name of ‘expert consensus’ scored 12% (ranging from 5% to 20%), which is less than the average level, indicating that the scores for the domain of ‘rigor of development’ could be considered a useful reference to differentiate consensus from guideline. This step can play a key role in determining whether the recommendations are truly based on the evidence and in understanding how the evidence is synthesised. There is an explicit link between the best evidence available and the recommendations made, including clarifying which systems were used to evaluate the quality of the evidence and to grade recommendations.45
The less well addressed domain, ‘applicability’, produced disagreement, with the findings of the most recent published studies reporting that ‘applicability’ domain scored (42%) lower than the other five AGREE domains among 137 guidelines developed in USA, Canada, UK and an international group.46 Coming to the hypertension CPGs, ‘applicability’ scored low (38%) relative to all domains but ‘rigor of development’ (30%).39 It was shown that the type of developer (disease-specific foundation, non-profit healthcare system) was associated with applicability score, and the majority of hypertension CPGs included in the present study was developed by hypertension-specific association and non-profit healthcare system, which may be explained by the applicability scores achieved. Although the reported ‘applicability’ domain performed poorly, the scores are much higher than the score in the present study. The poor ‘applicability’ scores reflected that the implementation of guidelines, such as organisation barriers, the cost and the criteria used to monitor the local implementation or adaption, were not clearly addressed.47 ,48 It is worth considering that the process of defining facilitators and barriers to application should be integrated early in the guideline development process, including professional implementation strategies.49
The assessed guidelines achieved a moderate mean score in the ‘stakeholder involvement’ domain, which assessed the degree to which the guideline represents the views of its intended audience.50 It could be noted that patient preferences and experiences should be factored into decisions regarding clinical care, especially hypertension, the management of which can impact quality of life. The guidelines we reviewed would have benefited from ensuring that all guideline committees had patient representatives and that literature reviews specifically addressed the quality of life (when available).
The main limitation of this study is that the methodological quality assessment was based on what the CPGs developers actually reported, which might not truly reflect the construction process. Although two independent trained appraisers conducted the assessment using a standardised instrument, the involvement of subjective bias in the evaluation was unavoidable. The AGREE collaboration recommends that each guideline be assessed by at least two appraisers, but that without proper training, adding appraisers may increase the rating bias.51
The overall methodological quality of the CPGs for hypertension in China was generally low throughout the appraisal process as determined using the AGREE II instrument. Considerable variability remained between guidelines, and strategies should be implemented to enhance the clarification of the subject of CPGs, the application of evidence-based systematic methods, and the transparency of CPGs. In addition, multidisciplinary groups associated with professional organisations in China, especially the methodological experts in the field for which the CPG is being drawn up, should be consulted.
The authors would like to thank Boxi Yan at West China Medical School, West China Hospital of Sichuan University, for his kind and insightful suggestions in the review.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.