Article Text
PDF
Abstract
Objectives To map the available methodological guidelines and documents for conducting and reporting benefit–risk assessment (BRA) during health technologies’ life cycle; and to identify methodological guidelines for BRA that could serve as the basis for the development of a BRA guideline for the context of health technology assessment (HTA) in Brazil.
Design Scoping review.
Methods Searches were conducted in three main sources up to March 2023: (1) electronic databases; (2) grey literature (48 HTA and regulatory organisations) and (3) manual search and contacting experts. We included methodological guidelines or publications presenting methods for conducting or reporting BRA of any type of health technologies in any context of the technology’s life cycle. Selection process and data charting were conducted by independent reviewers. We provided a structured narrative synthesis of the findings.
Results From the 83 eligible documents, six were produced in the HTA context, 30 in the regulatory and 35 involved guidance for BRA throughout the technology’s life cycle. We identified 129 methodological approaches for BRA in the documents. The most commonly referred to descriptive frameworks were the Problem, Objectives, Alternatives, Consequences, Trade-offs, Uncertainty, Risk and Linked decisions and the Benefit–Risk Action Team. Multicriteria decision analysis was the most commonly cited quantitative framework. We also identified the most cited metric indices, estimation and utility survey techniques that could be used for BRA.
Conclusions Methods for BRA in HTA are less established. The findings of this review, however, will support and inform the elaboration of the Brazilian methodological guideline on BRA for HTA.
Trial registration number https://doi.org/10.17605/OSF.IO/69T3V.
- public health
- health policy
- statistics & research methods
Data availability statement
All data relevant to the study are included in the article or uploaded as supplementary information.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
STRENGTHS AND LIMITATIONS OF THIS STUDY
This is the first scoping review aiming at mapping methodological guidelines and publications on methods of benefit–risk assessment, especially in the context of health technology assessment (HTA).
We used the framework proposed by Arksey and O’Malley and the refinements made by the Joanna Briggs Institute.
We performed an electronic search on the main databases as well as manual searches on a vast source of grey literature, including 38 HTA bodies and networks, and 12 regulatory agencies’ websites.
Despite our attempt to conduct a comprehensive search, we may have missed documents reporting methodological recommendations or guidelines for benefit–risk assessment methods used in practice which were not publicly available online.
The extraction of data may have been impacted by the great variety and inconsistent classification of benefit–risk assessment frameworks and methods, and data synthesis may have been impacted by some overlapping and redundant content from documents produced by the same organisations.
Introduction
Benefit–risk assessment (BRA), also referred to as risk–benefit or benefit–harm assessment, is an important component in decision-making throughout the life cycle of a health technology, from its development, regulatory approval, postmarketing surveillance, decisions about incorporation and reimbursement in health technology assessment (HTA), decision-making in clinical practice, to its obsolescence.1–3
BRA of comparative technologies is usually carried out informally, without following a systematic and reproducible process,3 which can lead to inappropriate or intransparent decisions. During the last two decades, efforts have been observed to apply more structured, objective and transparent approaches, aiming at better communication and decision-making.1 3–5 For this purpose, several frameworks have been proposed to guide BRA.1
Structured approaches for BRA have been used, in particular for regulatory decisions and postmarketing surveillance. The European Medicines Agency (EMA) has been making recommendations on BRA structured methods for new drug applications since 2007.6 The Pharmacoepidemiological Research on Outcomes of Therapeutics (PROTECT) initiative was established by a European Consortium aiming to support the monitoring of BRA of medicines in Europe and to provide recommendations to various stakeholders, particularly regulators.7 With the same attention, in 2009 the Food and Drug Administration (FDA), in the USA, initiated an effort to explore more systematic approaches for BRA as part of the drug review process and proposed its benefit–risk framework (FDA BRF).8
Concerning the HTA context, to the best of our knowledge, the efforts for using formal approaches for BRA are in a preliminary phase compared with the regulatory setting. In Brazil, the content of the HTA dossier submissions to the National HTA body, the Comissão Nacional de Incorporação de Tecnologias (CONITEC), should include the description of the clinical evidence of the technology of interest (ie, efficacy, effectiveness, accuracy and safety) compared with the technology already available in the public health system.9 However, currently no recommendations regarding the scope, methods and reporting of BRA are provided by CONITEC.
This review represents the first phase in a larger project to improve the application of BRA in the context of the Brazilian HTA bodies. A partnership with Rede Brasileira de Avaliação de Tecnologias em Saúde (REBRATS), a strategic network to facilitate the elaboration of priority HTA studies for the Brazilian health system,10 will provide methodological and training support to increase the use of BRA methods in the reports under deliberative decision-making processes. Furthermore, findings from this scoping review will inform the development of a methodological guideline on BRA for the CONITEC.
Therefore, the objectives of this scoping review were: (1) to map the available methodological guidelines and documents for conducting and reporting BRA during health technologies’ life cycle—within this objective, we pursue identifying the definitions of BRA, the approaches for conducting BRA and the visual tools for reporting BRA results that have been used; and (2) to identify methodological guidelines for BRA, which could be used as the basis for the development of a BRA guideline for the Brazilian HTA context.
Methods
Study design
This scoping review was based on the framework proposed by Arksey and O’Malley11 and the updated guidelines by the Joanna Briggs Institute.12 13 The review protocol is published in BMJ Open14 and is registered in the Open Science Framework (https://doi.org/10.17605/OSF.IO/69T3V). The reporting of this review follows the Preferred Reporting Items for Systematic reviews and Meta-Analyses Extension for Scoping Reviews (PRISMA-ScR) recommendations (online supplemental table 1).15
Supplemental material
Research question and eligibility criteria
Our research question was ‘What are the methods for BRA in the context of a health technology’s life cycle?’. The relationship between our objectives, research question and eligibility criteria is depicted in figure 1.
Relationship between the objectives, research question and eligibility criteria for the scoping review. BRA, benefit–risk assessment; HTA, health technology assessment.
Our eligibility criteria followed the Population, Concept and Context mnemonic framework.12 13 We included (1) methodological documents concerning BRA involving any types of health technologies and populations (Population); (2) presenting methods for conducting or reporting BRA (Concept) and (3) developed in any context of the health technology’s life cycle (Context). The types of publications included were full-text methodological guidelines, recommendations, standards, consensus, methodological reports, methodological reviews, methodological studies, research reports addressing specific methods for BRA and reporting guidelines. We excluded editorials, comments, studies using qualitative methods, conference abstracts, studies reporting methods exclusively for either the assessment of benefit or risk/harm (ie, not reporting BRA trade-off or balance) and publications focusing only on the description of a specific methodological approach that could be used for BRA but did not present the application for BRA.
In case an eligible document stated methods for BRA balance and, in addition, other metric indices for quantifying either benefits or harms that fall into the categorisation proposed by Mt-Isa et al,3 which was followed for data charting (see section ‘Charting the data, summarising and reporting the results’), such indices were also reported in our review.
Information sources and search strategy
We performed a comprehensive search on three main sources: (1) biomedical electronic databases (electronic databases); (2) websites of key HTA and drug regulatory organisations (grey literature) and (3) manual search and contacting experts in the field (manual search).
The search strategy in the: (1) electronic databases (EMBASE via OVID and MEDLINE via PubMed) followed a three-step approach,12 13 using indexed and free-text terms, validated filters16 17 and no language or publication date restrictions. The strategy was validated by an experienced research librarian and peer-reviewed using the Peer Review of Electronic Search Strategies (PRESS) checklist.18 These searches were completed in October 2022. The complete search strategy is presented in online supplemental tables 2 and 3. (2) Grey literature consisted of searching the websites of 36 HTA bodies and global HTA networks, and 12 key regulatory authorities using free-text terms such as ‘benefits AND (risks OR harms) AND methods’ or adaptations made accordingly, and no publication date restrictions. The search was performed using the language of origin of the evidence source, or when more than one language was available, preference was given to English, Spanish and Portuguese. These searches were conducted from October 2022 to January 2023. The list of websites is presented in online supplemental table 4. (3) Manual search consisted of hand-searching the reference lists of all relevant documents identified in the two previous sources, and contacting experts by email. We performed the manual searches and contacted the experts from February to March 2023.
Selection process
We conducted a pilot test for the selection process. All reviewers working in independent pairs screened the titles and abstracts of a random sample of 100 titles/abstracts,13 using the pre-specified eligibility criteria. In case of disagreements higher than 15% within the pair, another random sample was screened. The pairs started screening the documents when 85% (or greater) agreement was achieved.
Three pairs of independent reviewers screened all references retrieved from the electronic database search by reading titles and abstracts. One pair of independent reviewers screened the titles and abstracts identified from grey literature. The full-text of potentially eligible documents identified during the screening as well as those identified via manual search were retrieved and assessed by three pairs of reviewers according to the eligibility criteria. Any disagreements during the selection process were solved through a consensus within the team of reviewers, or by a third reviewer.
Charting the data, summarising and reporting the results
A charting form to guide the data extraction was developed using the classification of BRA methods and visual tools proposed by Mt-Isa et al3 and Hallgreen et al.19 The definitions of benefit, risk/harm and BRA provided by the documents were grouped through analysis content.20 The charting form was validated by the reviewers conducting the extraction from five eligible documents. Data extraction was performed by three pairs of independent reviewers. Any disagreements were solved through a consensus within the team of reviewers, or by a third reviewer.
We conducted a structured narrative synthesis and reported the results in evidence tables and figures along with descriptive statistics to identify common characteristics and map the evidence.
Patient and public involvement
Although is important to have the involvement of patients and the general public in the HTA decision-making processes, such stakeholders are usually not involved in methodological reviews as the present one. Therefore, patients and/or the general public were not involved in the design, conduct or reporting of this study. The findings of this review will provide an important basis for the elaboration of the Brazilian methodological guideline on BRA for HTA and we intend to collaborate with HTA experts, REBRATS and CONITEC to promote the use of BRA methods in the context of HTA.
Results
Figure 2 provides an overview of the selection process. Of 12 915 references retrieved from electronic databases, 11 761 were screened, the full-text of 66 were assessed, 34 were excluded (reasons are presented in online supplemental table 5) and 32 publications were included.1–3 5 19 21–47 Regarding grey literature, our search resulted in the identification of 160 documents. Among these, 25 were included.6 8 48–70 Additionally, we retrieved 26 methodological documents through manual searches.4 71–95 In total, our scoping review encompassed 83 documents or publications that met the eligibility criteria.
Overview of the selection process. HTA, health technology assessment.
Characteristics of the included documents
Table 1 presents a condensed summary of the characteristics of the included documents. The majority were published by institutions in the USA, followed by the European Union or partnerships between European institutions. Accordingly, most of the documents originated within the regulatory framework of the USA and Europe, with seven from the EMA6 48–53 and six from the FDA.8 56–60 Figure 3 depicts the geographic area in which the documents were developed, highlighting the countries that produced documents for the HTA context.
Number of documents published per geographic area. Transcontinental stands for ≥2 countries from different continents. Continent (Europe, North America) stands for ≥2 countries within the same continent. HTA, health technology assessment.
Characteristics of included documents
Most of the documents were published during the years 2013–2017. The first document was published in 1998 by the Council for International Organisations of Medical Sciences (CIOMS) Working Group.90 The first publication involving the HTA context was a review on multiple criteria decision analysis funded by the NICE Decision Support Unit, UK, published in 2011.70 The two most recent documents within the HTA context, published in 2022, were the General Methods from the German Institute for Quality and Efficiency in Healthcare (IQWiG),65 and a Methodological Handbook for the evaluations of clinical effectiveness, safety and diagnostic validity of health technologies from a Spanish HTA body.54
More than 37% of the documents comprised literature reviews. Eighteen documents consisted of methodological reports (21.7%), informing and/or describing methods of BRA.6 31 41 48–55 64 74 75 78 79 87 90 Thirteen (15.7%) were methodological guidelines providing recommendations for conducting BRA, mostly developed by regulatory agencies.8 56–60 62 65–68 85 86 Furthermore, 14 (16.8%) methodological papers have proposed new methods of BRA or their application, all of which were published in peer-reviewed journals.24 25 30 33 47 61 63 69 73 77 81 84 94 95 We also identified six systematic reviews on BRA methods,1 3 19 32 37 43 and one reporting guideline developed specifically for reporting of BRA of vaccines.44
Medicinal products (including the former (n=13),4 6 24 28 36 38 48–53 80 pharmaceutical drugs (n=33),2 3 5 21–23 31–35 37 39 40 42 45 58 61 66 71–73 75 77 78 81–83 85–87 89 90 vaccines (n=5)43 44 88 94 95 or a combination of the previous with biologics and radiopharmaceuticals (n=6))1 8 56 60 62 84 were the type of technology most addressed in the documents (n=57; 68.8%). Medical devices (including the former (n=6),57 59 63 67 68 79 diagnostic tests (n=1)25 or a combination of the previous with equipments (n=2))46 47 were addressed in nine documents (10.8%). The remaining documents stated that the BRA methods could be applied to all types of technologies (general; n=17, 20.4%).19 26 27 29 30 41 54 55 64 65 69 70 74 76 91–93
Five of the documents produced for the HTA context were developed by HTA bodies.54 55 65–67 One document was developed by an academic institution with the funding of an HTA body.70 One document, although produced by a member of the International Network of Agencies for Health Technology Assessment (INAHTA), aimed to describe methods for BRA in systematic reviews.41 Most of the documents, including all produced for the HTA context, were funded by institutions not involved in for-profit activities.
A possible conflict of interest, identified in about half of the documents, was present if the individuals involved in the document development received support or employment, any stocks or shares and any consultation fees or other forms of remuneration from the industry.
Definitions of benefit, risk and BRA
In total, 31 documents provided the definition of ‘benefits’. We classified the definitions as ‘favourable effects’ (n=11; 13.3%),19 26 40 45 50 60 62 63 65 79 85 ‘positive results for an individual or a population, and the probability of achieving such results’ (n=11; 13.3%),5 6 32 39 48 49 68 78 90 91 93 ‘a potential effect that moves the condition of the patient from disease towards health’ (n=6)27 42 80 89 94 95 and ‘results that influence the overall benefit–risk balance in a clinically meaningful way and that provide evidence supporting the product approval’ (n=3).23 72 75
Risks or harms were defined in 31 documents. Definitions that emerged were ‘unfavourable effects’ (n=15; 18.1%),26 27 40 45 50 60 63 65 75 79 80 85 91 94 95 ‘negative results for an individual or a population, and the probability that a negative event will happen’ (n=10; 12.0%),6 19 32 39 48 49 68 78 87 93 ‘a potential effect that moves the condition of the patient from health towards disease’ (n=4)42 62 89 90 and ‘results that influence the overall benefit–risk balance in a clinically meaningful way and that provide evidence not supporting the product approval’ (n=2).23 72
BRA was defined in 28 documents, and most of them stated that BRA ‘involves balancing between benefit and risk, however, it does not specify whether the assessment is quantitative, qualitative or both’ (n=20; 24.1%).1 6 8 23 48–50 60 62 64–66 68 76 84–86 89 92 93 In contrast, other definitions emerged as ‘a quantitative or qualitative evaluation of medical product, incorporating explicit outcome weighting within a formal analysis taking both benefits and risks of the product into account’ (n=5),43 57 59 78 87 and ‘a quantitative evaluation of medical product, incorporating explicit outcome weighting within a formal analysis taking both benefits and risks of the product into account’ (n=3).35 74 90
Approaches for BRA
We identified 129 methodological approaches or elements of BRA, including frameworks, metric indices, estimation and utility survey techniques. Figure 4 presents the approaches cited in at least five of the 83 eligible documents (see online supplemental table 6 for the complete list of approaches). In 14 (16.9%), only descriptive (ie, qualitative) frameworks for BRA were reported,8 22 24 26 57–59 67 71 81–83 85 86 18 (21.7%) cited only quantitative frameworks,25 27 30 37 44 46 47 61 64 69 70 77 80 84 89 92 94 95 and 58% reported both descriptive and quantitative frameworks. The descriptive frameworks most frequently cited in the documents were the Problem, Objectives, Alternatives, Consequences, Trade-offs, Uncertainty, Risk and Linked decisions (PrOACT-URL),1 3–5 19 22 26 28 32 34 36 38 40 42 43 45 50–52 68 71–75 78 79 87 91 and the Benefit–Risk Action Team (BRAT) framework,1–5 19 21 22 26 28 31 32 34 36 40 42 43 50 72–75 78 79 82 83 87 88 91 cited in 29 documents (34.9%) each. The descriptive frameworks most recommended were the PrOACT-URL1 42 51 52 75 78 87 and the FDA BRF,1 8 56–60 recommended in seven documents each (8.4%). Among the quantitative frameworks, multicriteria decision analysis (MCDA) was the most frequently cited (n=52; 62.7%)1–6 19 21 23 28 29 31–34 36–43 45–52 56 64 65 68–70 72–79 87–89 91 92 94 95 and recommended (n=13; 15.7%).38 39 42 47 50 52 69 73 74 77 78 87 95 Other frequent quantitative frameworks cited were Markov decision processes (n=18; 21.7%)1 3 25 32 33 39 40 42–45 50–52 78 87 91 94 and decision trees (n=16; 19.3%).1 3 25 29 32 40 42 43 46 47 50–52 75 78 87
BRA methodological approaches identified in the included documents and publications. Adapted from PROTECT.7 AE-NNT, adverse event-adjusted number needed to treat; AHP, analytic hierarchy process; ASF, Ashby and Smith framework; Beckmann, Beckmann model (evidence-based model); BLRA, benefit-less-risk analysis; BRAT, Benefit–Risk Action Team; BRR, benefit–risk ratio; CA, conjoint analysis; CDS, cross-design synthesis; CUI, Clinical Utility Index; CMR-CASS, CMR Health Canada, Australia’s Therapeutic Goods Administration, SwissMedic and Singapore Health Science Authority; CPM, confidence profile method; COBRA, consortium on benefit–risk assessment; CV, contingent valuation; DAG, directed acyclic graphs; DALY, disability-adjusted life years; DAM, decision analytic model (no specific designation was given to the model); DCE, discrete choice experiment; DI, Desirability Index; FDA BRF, FDA benefit–risk framework; GBR, global benefit–risk; HALE, health-adjusted life years; INHB, incremental net health benefit; ITC, indirect treatment comparison; KM, Kaplan-Meier; MAR, maximum acceptable risk; MCDA, multicriteria decision analysis; MCE, minimum clinical efficacy; MDP, Markov decision process; MTC, mixed treatment comparison; NCB, net clinical benefit; NEAR, net efficacy adjusted for risk; NNH, number needed to harm; NNT, number needed to treat; PBRER, periodic benefit–risk evaluation report; Principle of 3’s, principle of threes; PrOACT-URL, Problem, Objectives, Alternatives, Consequences, Trade-offs, Uncertainty, Risk and Linked decisions; PROTECT, Pharmacoepidemiological Research on Outcomes of Therapeutics; PSM, probabilistic simulation method; QALY, quality-adjusted life years; Q-TWiST, quality-adjusted time without symptoms and toxicity; QFRBA, quantitative framework for risk and benefit assessment; RBAT, risk–benefit acceptability threshold; RBC, risk–benefit contour; RBP, risk–benefit plane; RV-MCE, relative value-adjusted minimum clinical efficacy; RV-NNH, relative value-adjusted number needed to (treat to) harm; RV-NNT, relative value-adjusted number needed to treat; SABRE, Southeast Asia benefit–risk evaluation; SBRAM, Sarac’s benefit–risk assessment; SG, standard gamble; SMAA, Stochastic multicriteria acceptability analysis; SPM, stated preference method; SW, swing weighting; TTO, time trade-off; TURBO, transparent uniform risk–benefit overview; UMBRA, unified methodologies for benefit–risk assessment; UT-NNT, utility-adjusted and time-adjusted number needed to treat. *General: no specific designation was given to the descriptive framework. #Other: approaches cited in <5 of the included documents (see online supplemental table 6 for the complete list of approaches). Numbers are presented (as number of documents citing the approach; proportion of the total 83 documents).
Metric indices were reported in 59 (71.1%) documents. The most cited threshold indices were the number needed to harm (n=38; 45.8%)1–6 19 21 25 29 31 32 34 35 37 40–44 47–50 54 55 61 66 76–78 80 81 87 89 91–93 and the number needed to treat (n=37; 44.6%),1–6 19 21 25 29 31 32 34 35 37 39–42 47–50 54 55 61 66 76–78 80 81 87 89 91–93 which were also recommended in four documents.2 66 78 87 The quality-adjusted life years (QALY) emerged as the most cited (n=37; 44.6%)1–3 5 19 25 29 32 34 37 39–42 44 46–52 55 61 65 67 70 75 76 78–80 84 87 89–91 and recommended health index (n=3),25 78 87 followed by the quality-adjusted time without symptoms and toxicity,1 3 5 19 29 32 37 39–41 47 50 76 78 87 89 92 reported in 17 documents.78 87 As for trade-off indices, the incremental net health benefit was the index mostly cited (n=32; 38.6%)1–3 5 19 21 25 29 32 34 37 40–43 46 47 49 50 55 61 64 67 68 76 78–80 87 89 91 95 and recommended (n=6; 7.2%);47 61 64 78 87 95 followed by the transparent uniform risk–benefit overview1 3 6 29 32 41 42 48–50 76–78 80 87 90 and the benefit–risk ratio,2–5 19 32 37 40 43 44 78 84 87 90 92 95 each cited in 16 documents (19.3%).
Estimation techniques were reported in 41 documents. The probabilistic simulation method (PSM) was the most cited approach (n=32; 38.6%).1 3–5 19 25 29 32 33 37 38 40–47 50–52 68–70 74–76 78 87 91 94 Indirect treatment comparison and mixed treatment comparison (MTC) were cited 18 (21.7%)1 3–5 32 33 40–42 47 54 55 64 65 67 74 78 87 and 17 (20.5%)1 3–5 19 32 33 40–42 54 64–66 74 78 87 times, respectively. PSM38 78 87 and MTC1 78 87 were recommended in three documents.
Concerning utility survey assessment techniques, reported in 48 documents, the dominating approaches were the discrete choice experiment (DCE) and the conjoint analysis, mentioned in 26 (31.3%)1 3–5 19 30 32 34 40 41 44 46 47 61 63 65 67 70 74 76 78 79 87 93–95 and 22 (26.5%)1 3 21 30 32 40–42 46 47 50–52 63 65 75 78 79 87 91 93 94 of the documents. DCE was recommended in three documents,65 78 87 followed by swing weighting, recommended in two.69 77
Visual tools to present BRA results
Tools for visual representation of BRA results were used or cited in 75 documents. Summary tables were the most common tool, present in almost 60% of the documents. Tree diagrams and value trees were present in 34 (41%) documents, followed by bar charts (33.7%), dot charts (33.7%), lines (31.3%) and area graphs (28.9%). ‘Non-conventional’ visual tools that emerged were pictograms (n=5)19 22 54 79 91 and suggestions for using interactive visual displays to enable active participation of the audience (n=4).31 48 49 87 The complete list of visual tools to present the results of BRA is depicted in table 2.
Types of identified visual tools to present results of BRA (total of publications n=83; publications that reported or presented visual tools n=74)
Methodological documents produced in the HTA context
Among the 83 eligible, six methodological documents (7.2%) were produced in the context of HTA,54 55 65–67 70 although 35 (42.2%) explicitly stated that the BRA would be applicable throughout the life cycle of the technology, which implies the context of HTA among others.
The guide for the elaboration of evaluation reports of medicines published by the Agencia de Evaluación de Tecnologías Sanitarias de Andalucía (AETSA), in Spain, suggests that the drug evaluation reports should present in the discussion section a comparison of the safety and efficacy results to obtain an overall assessment of the intervention,66 but no structure, framework or quantitative approach was recommended.
The EUnetHTA HTA Core Model for Rapid Relative Effectiveness, V.4.2 published in 2015, states that both relative benefits and harmful effects of a technology are essential in quantifying the net benefit of an intervention and are essential for being able to form a balanced view of the overall value of a technology.55
The methodological manual for the elaboration of evaluations of clinical effectiveness, safety and diagnostic validity of health technologies published by the Colombian Instituto de Evaluación Tecnologica en Salud (IETS) provides overall guidance for conducting HTA reports. Concerning BRA, the manual states that effectiveness and safety outcomes should be included in the report to allow the benefit–risk balance. However, although the conclusion of the HTA report must state whether the technologies of interest have less, similar or greater effectiveness and safety compared with their alternatives, the manual does not provide recommendations on how to evaluate the balance between them.54
In 2011, Thokala published a report about the applicability of MCDA for HTA. The author compared the MCDA process and the NICE technology appraisal process and described the general practical issues that might arise from using an MCDA approach in the HTA process.70
The General Methods, V.6.1 of 2022, a comprehensive methodological guideline published by the German IQWiG states that each predefined patient-relevant outcome (both beneficial and harmful aspects) is initially assessed on an outcome-specific basis and then presented along with the respective certainty of the evidence for each outcome. Within the overall weighing of benefits and harms, these individual outcomes are then summarised into a global conclusion on the extent of added benefit. If needed, a joint combined measure of benefit–harm such as QALY can be used.65
The Australian Medical Services Advisory Committee (MSAC) guidelines suggest constructing an assessment framework or logic diagram to illustrate the necessary steps that link the use of a technology in the target population and the consequences on outcomes.67 A guidance on formal BRA quantitative framework is not provided.67
Discussion
We have conducted a scoping review of available methodological guidelines and documents for conducting and reporting BRA during health technologies’ life cycle. We identified 129 approaches for conducting BRA and 37 visual tools for reporting BRA results. This is the first review stratifying the findings based on the HTA context. Confirming our previous perception about decision support in HTA, the efforts for using formal structured approaches for BRA have been more modest in that context. Only six documents produced by HTA bodies were identified;54 55 65–67 70 however, they do not provide detailed guidance on how to select the best approach and how to conduct BRA in the HTA context.
Mt-Isa et al identified 49 approaches for BRA and classified them into four main categories which were followed in our review: frameworks, metrics, estimation techniques and utility survey techniques.3 Frameworks, which can be subdivided into descriptive and quantitative, provide a structure that guides the assessment to support decision-making.3 They do not provide mathematical algorithms that result in automated decisions.40 Descriptive frameworks provide qualitative instructions, while quantitative frameworks additionally can provide formal quantitative methods to assess the balance between benefits and risks or provide tools to evaluate long-term benefits and risks/harms.3 Metrics are systems of measurement and can be subdivided into threshold indices (they handle either benefit or risk but not both), health indices (which include validated and standardised quality-of-life indicators) and trade-off indices (which integrate benefits and risks into a single metric representing the value of the trade-off for direct interpretation). Estimation techniques include generic statistical techniques, and they are applicable in combination with other methods. Utility survey techniques include methods to elicit and collect health utilities and value preferences and they also can be applied in combination with other methods.3
The assessed documents agree that some decisions are straightforward, but others need more objective criteria. In cases where a new technology increases benefits and decreases risks, or when the benefits clearly outweigh the risks, a formal quantitative BRA may not be essential. On the other hand, when the benefit–risk balance is not so clear and/or stakeholders preferences influence this balance, the additional use of quantitative BRA methods can be advantageous, if not crucial for decision-making.91 In all cases, at least a structured descriptive framework is recommended to transparently present the rationale to support decision-making and ensure that key aspects of the assessment process are not overlooked.87 As a second step, the explicit and quantitative assessment of benefit–risk balance may be added in situations where the trade-off is more difficult to judge.96
Strengths of our scoping review include a comprehensive search strategy resulting in more than 12 000 retrieved references from electronic databases and 160 full-text documents from 48 HTA and regulatory organisations. The selection and data charting processes were piloted to ensure concordance between the reviewers. Perhaps most importantly, this is the first scoping review aiming at mapping methodological guidelines on methods of BRA highlighting the findings for the HTA context.
As all scoping reviews, ours has several limitations. First, despite our comprehensive search, we may have missed eligible documents for BRA used by HTA and/or regulatory organisations not publicly available online or not searched by our group. Second, we included many documents produced by the same organisations. Therefore, although the documents were unique, they may present some overlapping and redundant content biasing our descriptive percentage results. Third, we identified the most cited BRA approaches, however, this does not mean that such approaches are the most used to support decision-making. Fourth, some of the methodological approaches might have been cited in the literature under different names and definitions, although they would fall into the same technical category. We have made efforts to collect the different spelling and wording approaches into the same technical nomenclature; however, we may have missed some specific approaches. Finally, appraising the features of the BRA approaches identified was beyond the scope of our review. This would require an extensive assessment from different health decision science perspectives and a full appraisal of all statistical and modelling methods. Such an assessment would result into a lengths report and be extremely laborious, precluding the timely conclusion of this review.
Although our goal was not to appraise the operational characteristics of each identified approach, we will test and explore the potential of at least the two most cited descriptive and quantitative frameworks in case studies in the context of HTA before making formal recommendations. We are aware that no best approach fits the multitude of populations, diseases, health technologies and their clinical applications, and therefore, our intention is not to prescribe or recommend any ‘one size fits all’ BRA approach, but to highlight the uses, advantages, disadvantages, human resources training/skills and computational requirements to support the selection of the methodologies to be used in future BRA in HTA dossier submissions to the Brazilian CONITEC.
We will also face the challenge of making recommendations on the source of data to conduct BRA in the context of HTA, which might consider a broader spectrum of sources compared with BRA for regulatory marketing authorisations, as well as periodicity of BRA for monitoring technologies incorporated in the Brazilian public health system. In addition, the election of the method to be applied for BRA also have to consider the need for rapid evaluation, especially in case of a public health crisis.97 These are topics not discussed in the documents identified in our review. Such aspects must be assessed and discussed in the future steps, and our intention is that the results and conclusions from this review will provide an important basis for these next steps towards a more explicit and transparent BRA in the context of HTA in Brazil.
Conclusions
Our review identified 129 methodological approaches for BRA, including descriptive and quantitative frameworks, metric indices, estimation and utility survey techniques, in 83 methodological guidelines and documents for conducting and reporting BRA in the different phases of the life cycle of health technologies. Among the documents assessed, we identified only six methodological documents produced in the context of HTA. We will test and explore the potential of the two most cited descriptive and quantitative frameworks in case studies in the context of HTA to evaluate their performance. The findings of this review will support these steps, and finally, inform the elaboration of the Brazilian methodological guideline on BRA for HTA.
Data availability statement
All data relevant to the study are included in the article or uploaded as supplementary information.
Ethics statements
Patient consent for publication
Acknowledgments
We thank Rachel Couban, Health Research Impact Librarian, Faculty of Health Sciences, McMaster University, Hamilton, ON, Canada, for her contribution on the development of the electronic search strategy for this review.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
EAS and BdOA contributed equally.
Contributors BdOA and PCDS conceived the main idea behind the study. EAS, BdOA and PCDS wrote the manuscript. EAS, BdOA, FHdAM, AFRB, NSF, FCG and PCDS extracted data. EAS, SMD and PCDS analysed, summarised and interpreted the findings. EAS, BJ, US and PCDS critically revised the manuscript and made important intellectual contributions to its development. All authors read and approved the final version of the manuscript. EAS, BdOA and PCDS; Guarantor.
Funding This work was supported by the Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq) grant number 400224/2022-4, with no role in the design of the study and collection, analysis and interpretation of data and in writing the manuscript. EAS, BOA, FHAM, SMD, NSF and FCG, and received CNPq schorlarships.
Map disclaimer The inclusion of any map (including the depiction of any boundaries therein), or of any geographic or locational reference, does not imply the expression of any opinion whatsoever on the part of BMJ concerning the legal status of any country, territory, jurisdiction or area or of its authorities. Any such expression remains solely that of the relevant source and is not endorsed by BMJ. Maps are provided without any warranty of any kind, either express or implied.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.