GRADE guidelines: 20. Assessing the certainty of evidence in the importance of outcomes or values and preferences—inconsistency, imprecision, and other domains

Abstract

Objective

To provide Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) guidance for assessing inconsistency, imprecision, and other domains for the certainty of evidence about the relative importance of outcomes.

Study Design and Setting

We applied the GRADE domains to rate the certainty of evidence in the importance of outcomes to several systematic reviews, iteratively reviewed draft guidance, and consulted GRADE members and other stakeholders for feedback.

Results

We describe the rationale for considering the remaining GRADE domains when rating the certainty in a body of evidence for the relative importance of outcomes. As meta-analyses are not common in this context, inconsistency and imprecision assessments are challenging. Furthermore, confusion exists about inconsistency, imprecision, and true variability in the relative importance of outcomes. To clarify this issue, we suggest that the true variability is neither equivalent to inconsistency nor imprecision. Specifically, inconsistency arises from population, intervention, comparison and outcome and methodological elements that should be explored and, if possible, explained. The width of the confidence interval and sample size inform judgments about imprecision. We also provide suggestions on how to detect publication bias and discuss the domains to rate up the certainty.

Conclusion

We provide guidance and examples for rating inconsistency, imprecision, and other domains for a body of evidence describing the relative importance of outcomes.

Introduction

The Grading of Recommendations, Assessment, Development and Evaluation (GRADE) working group developed a widely accepted approach to rate the certainty of a body of evidence in the contexts of systematic reviews, health technology assessment, health-care recommendations, and decision support [1], [2], [3], [4]. This is the 20th in the ongoing series of articles describing the GRADE approach in the Journal of Clinical Epidemiology and complements articles in this and other journals. We previously described the reasons for decreasing and increasing the certainty of a body of evidence; how an overall rating of the evidence is performed [5]; how to create GRADE evidence profiles and summary of findings tables [6], [7], [8], [9]; how evidence is used to move to recommendations and decisions [10], [11], [12], [13], [14], [15]; how evidence is dealt with in particular circumstances of diagnostic, prognostic, equity-related, multiple treatment comparison, environmental and public health questions [16], [17], [18], [19], [20], [21]; how GRADE applies to rapid advice [22]; how GRADE deals with new risk of bias tools [23]; and when there is missing outcome data [24].

Decisions in health care require an assessment of the certainty of how much people value the importance of the outcomes that researchers and practitioners intend to affect [10], [11], [12], [13], [14], [15]. Indeed, a variety of instruments are in use to elicit the relative importance of outcomes, including health state value or utility, or willingness to pay instruments. Our previous article in this series introduced the topic and the typical instruments in some detail [26]. For example, patients with severe or very severe chronic obstructive pulmonary disease (COPD) are willing to pay $13.46 to avoid mild side effects and $67.51 to gain symptom relief [25]. This suggests patients place more value on avoiding no symptom relief than avoiding mild side effects. We also described the terminology regarding the relative importance of outcomes and how it relates to the concept of values and preferences. The term outcome includes “health state” and non-health states that are related to the interventions under consideration, a broad set of the outcomes directly and indirectly related to health or a disease or non–health consequences. We then introduced the GRADE approach to rate the certainty of research evidence that focuses on this gap area, the rating of the certainty of a body of evidence about the relative importance of outcomes with a focus on risk of bias and indirectness [26]. We also introduced our rationale and explanations about the terminology of outcome importance [26]. In this article, we will provide guidance on rating of a body of evidence about the relative importance of outcomes dealing with the GRADE domains' inconsistency and imprecision and describe other concepts related to publication bias, rating up the certainty of evidence and variability in estimates. This guidance informs the certainty in the evidence about the relative importance of the outcome (values) and, thus, the evidence to decision framework. However, it will also affect the assessment of the balance between desirable and undesirable health outcomes and eventually affect the balance of the overall desirable and undesirable consequences (including the consequences on other criteria in the evidence to decision framework, for example, ethical consequences or consequences on health equity). In addition, our approach will be of use for those using utilities and values and preferences apart from GRADE evidence to decision frameworks; in particular, those conducting health technology assessments and decision modeling.