Article Text
PDF
Abstract
Objectives As part of the National Institute for Health and Clinical Excellence (NICE) Single Technology Appraisal (STA) process, manufacturers present submissions outlining the clinical and cost-effectiveness of new technologies. These submissions are critically appraised by Evidence Review Groups (ERGs), who produce a report, which forms part of the evidence considered by the NICE Appraisal Committees. The purpose of this research was first to identify common issues and concerns identified by the ERGs in their analyses of manufacturers' submissions (MS). The aim was then to use these as a basis to develop feedback for manufacturers.
Design A qualitative study using a content analysis approach to examine two sources of evidence, the first 30 ERG reports and 21 clarification letters associated with these STAs.
Setting UK HTA programme.
Primary and secondary outcome measures Common issues and concerns in MS.
Results There were positive comments regarding the quality of the MS, many of which were clearly written. The majority, however, were generally of poor quality and issues and concerns identified across the ERG reports and clarification letters included: criticisms related to the data being used especially data employed in the cost-effectiveness model, failure to perform a necessary analysis and poor reporting of processes used in the MS. Aspects of the decision problem were also often poorly or inadequately addressed by manufacturers. The majority of points raised for clarification related to the economic data analysis. Internal inconsistencies between the clinical and economic sections of the submission were frequently highlighted. These were used as the basis for the development of 12 suggestions for manufacturers.
Conclusions Much can be done to improve the quality of MS in the NICE STA process. Suggestions include the need for clear and transparent reporting of methods and analyses.
This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/2.0/ and http://creativecommons.org/licenses/by-nc/2.0/legalcode.
Statistics from Altmetric.com
Article summary
Article focus
As part of the NICE STA process, ERGs critically appraise manufacturer submissions in their reports and write clarification letters requesting more information from manufacturers. Two sources of evidence, ERG reports and the clarification letters associated with these STAs were analysed using a content analysis approach. The aim of this study was to identify issues and concerns identified by the ERGs in their analyses of submissions and use these to develop feedback to manufacturers.
Key messages
A qualitative analysis of 30 ERG reports and 21 clarification letters was undertaken to identify recurring issues and concerns in manufacturers' submissions. The issues and concerns identified in this analysis were used to inform the development of a set of 12 suggestions to manufacturers to improve the quality of future submissions to NICE.
Strengths and limitations of this study
The research study applied validated methods and used multiple reviewers to check and verify data and analyses. Only the first 30 completed ERG reports and 21 associated clarification letters were examined in this analysis. There may be some misinterpretations in these analyses as documentary analysis was the only method used.
Introduction
The National Institute for Health and Clinical Excellence (NICE) produces technology appraisals using clearly defined processes, which lead to recommendations on the use of new and existing medicines and treatments within the NHS.1 This guidance is currently mandatory in the NHS in England and Wales, giving NICE the potential to decrease variation in the provision of healthcare. The Single Technology Appraisal (STA) process was introduced in 2005 and was specifically designed to appraise a new technology for a single indication and produce guidance as close to market authorisation as possible. The pressures that initiated this imperative are many and come from government, patients, patient organisations, clinicians and as would be expected the pharmaceutical manufacturers.
In the STA process, the manufacturers' submission (MS) to NICE provides the clinical and cost-effectiveness data from which the appraisal proceeds. NICE has provided a standard report template and extensive guidance for manufacturers to assist them in the preparation of their submissions.2 The MS is expected to include a systematic review of the clinical and cost-effectiveness evidence for the technology under consideration as well as a cost-effectiveness analysis, which, as standard, includes the submission of an economic model demonstrating their view of the cost-effectiveness of the technology. The introduction of the STA process initially raised concerns that basing decisions on submissions from the manufacturers had the potential to provide a less robust process for producing guidance than the original NICE multiple technology appraisal (MTA) process. For MTAs, the evidence is identified and appraised by independent academic groups.3 Issues continue to be raised regarding the rigour and timeliness of the STA process.4 5 A recent analysis of the timings and outcomes of the STA process showed that it is not as rapid as originally anticipated.6
For STAs, external independent Evidence Review Groups (ERGs) are charged with the task of critically appraising the MS by identifying strengths, weaknesses and gaps in the evidence presented. The resultant ERG reports then form a part of the evidence considered by the NICE Appraisal Committees (ACs). As part of the process, the ERG has the opportunity to request more information from the manufacturer in the form of a clarification letter sent 2 weeks after the submission has been received. ERGs are expected to appraise critically the MS to determine whether the clinical and economic evidence presented is:
relevant to the issue under consideration in terms of patient groups, comparators, perspective and outcomes;
complete (all relevant evidence identified);
inclusive of all study design information (including the type of study, the circumstances of its undertaking and the selection of outcomes and costs) and inclusive of all intention-to-treat patients;
fit for purpose (contributing to an overall assessment of the clinical benefit and quality of life, preferably in such units that allow comparison of the benefits from different technologies and between different patient groups);
methodologically sound in that the presented analyses and modelling are replicable, have face validity and are open to external scrutiny.
Common issues associated with the MS for the NICE MTA process were explored by Burls and Sandercock7 in their tongue-in-cheek paper in the BMJ's Christmas issue in 2003 (eg, choice of comparators, length of follow-up and study selection). Miners et al8 also highlighted the significantly lower estimated cost-effectiveness ratios submitted by manufacturers compared to those submitted by the independent assessment groups in the NICE MTA process. Although the STA process is different, and extensive scoping is carried out to provide a comprehensive decision problem, which guides the STA process, many of the same approaches continue to be used by manufacturers in their STA submissions to NICE.
There is currently uncertainty regarding the future of the NICE technology appraisal process.9 However, these issues will still be relevant in any process where submissions from manufacturers are provided as part of the evidence used to make decisions regarding new treatments. The purpose of this research study was to identify common issues and concerns identified by the ERGs in their analyses of MS and to use these as a basis to provide feedback to manufacturers to assist them in future submission development as well as recommendations to the ERGs and NICE. The suggestions for manufacturers are presented in this paper.
Methods
Two sources of information were used to identify common issues and concerns with MS: the first 30 completed ERG reports and 21 associated clarification letters. These are presented in detail in Kaltenthaler et al10 and summarised briefly here. The 30 ERG reports and their related letters were anonymised, and none is referred to explicitly in this paper.
Twenty-one clarification letters were available, relating to the first 30 STAs. The use of clarification letters was not part of the original STA process so a number of the early reports did not have letters. The clarification letters were examined, and data were extracted using a set of open codes with definitions to categorise data. Initial data coding was conducted by one reviewer (RD) and included clinical and economic issues, indirect comparisons, licensing, systematic review methods and report quality. After initial coding, the reviewer (RD) examined and categorised the clinical and report quality points into three groups: reporting of systematic review methods, clinical data and report quality. Two reviewers (PF and AB) regrouped and categorised the economic data into three categories: clarification/explanation, data/methods and further analyses.
A documentary analysis of the first 30 ERG reports was undertaken. For this research, we were exclusively interested in the content, rather than the context of the reports. Extraction of relevant data from the ERG reports was conducted by three team members (AB, CC and PF) using forms developed for this project and piloted on two ERG reports by all three reviewers. The aim of the extraction was to retrieve data on key elements of the processes undertaken by ERGs and the strengths and weaknesses of the MS. Much of these data consisted of text, that is, statements or summaries by the ERG. Thematic analysis11 was the method chosen to analyse these data as it is grounded in the data and therefore permits the generation of a novel thematic framework reflecting the ERGs' assessments of the strengths and weaknesses of MS to the STA process. This interpretive process was initially performed by one reviewer (CC) on the extractions from a random sample of 10 ERG reports. If themes identified in this way were considered to be related, then they were placed under a broader theme that captured them all. Definitions were then developed for each theme in order to produce greater reliability in the coding of data. Two members of the project team (RD and EK) then independently assessed whether these thematic interpretations of the data were both credible and appropriate and whether the themes identified reflected the data therein. This led to a small number of revisions and some further clarification of the themes' definitions. The extracted textual data from the remaining 20 ERG reports were then coded using these agreed themes following a process akin to that outlined in framework analysis.12 This was performed by two reviewers (CC and EK) each working on the data extracted from half of the remaining 20 ERG reports.
From these two sources, common issues and concerns were identified and these formed the basis for the development of feedback for manufacturers to improve the quality and content of their MS.
Results
Clarification letter analysis
Over 400 points from the clarification letters were analysed and presented in four main categories: ‘report quality’, ‘systematic review methods’, ‘clinical data’ and ‘economic data analysis’.
The ‘report quality’ category covered general issues related to the overall quality of the MS as well as errors and inconsistencies. Also included in this category were issues related to the presentation of data and absence of information. In the ‘systematic review methods’ category, there were issues related to searching and study selection, decisions regarding inclusion criteria and general methods issues. The comments indicated that manufacturers often failed to report the use of robust methods for the conduct of their systematic reviews. The ‘clinical data’ category included points related to specific clinical questions, requests for additional clinical data, explanations or further analyses and provision of rationale for the analyses that had been undertaken. Requests included a number of issues such as additional subgroup analysis, use of end points and outcomes and, most frequently, queries around the use of indirect and mixed treatment comparisons.
The majority of clarification points related to the ‘economic data analysis’ category and covered issues such as clarification of data sources and choices, queries about modelling decisions and requests for additional analyses. Internal inconsistencies between the clinical and economic sections of the submission and inconsistencies within the economic section of the MS were identified as particular problems highlighted in the clarification letters.
ERG report analysis
There were many positive comments in the ERG reports regarding the quality and reporting of the MS. However, for the purpose of this analysis, it is the main shortcomings and/or inconsistencies that are presented. The thematic analysis generated a large number of themes and from these five broader themes emerged. These five themes related to ‘processes’, ‘reporting’, ‘satisfaction of objectives’, ‘reliability and validity’ and ‘content’.
The ‘processes’ theme related to interpretations of ERG comments on the conduct or performance of the systematic review and cost-effectiveness modelling within the MS, in particular the conduct of analyses. More than half (17/30) of the ERG reports explicitly criticised the conduct of the systematic review within the MS, for example, the quality of the searching, screening or quality assessment processes, the definition or application of inclusion criteria, the presence of errors or failure to perform meta-analyses or even the complete absence of a formal systematic review. Criticisms related to the manufacturers' analyses covered both failure to perform a necessary analysis (12/30 reports) and the performance of an inappropriate analysis (17/30 reports). Two-thirds (20/30) of ERG reports contained criticisms related to the data being used, especially data employed in the cost-effectiveness model.
The ‘reporting’ theme referred to ERG comments on the quality of the manufacturers' description of the conduct of both the reviewing and modelling. The vast majority (27/30) of ERG reports identified inadequate reporting of processes in the MS. These included limited descriptions of searching, prohibiting replication and lack of transparency in the description of the processes used in the clinical review process generally for both direct and indirect comparisons. The manufacturers' limited reporting of the cost-effectiveness model was the most common criticism (11/30 reports) and included failure to describe adequately either the parameters or assumptions behind the model, the generation or source of various values or the impact of bias from missing data.
The four categories identified in the clarification letters translate into these two themes of ‘processes’ and ‘reporting’ derived from the overall ERG analysis. This is because the letters concern the choice and/or provision of data (processes) and clarifications on what the manufacturer had actually done (reporting). The following three themes reflect more the overall submission.
The ‘satisfaction of objectives’ theme referred to issues related to the decision problem (population, intervention, comparator, outcomes and NICE base case) and whether or not these were satisfactorily addressed in the MS. Twenty of the 30 ERG reports raised issues with the trial populations presented in the MS and 18/30 raised various issues with the comparators. By contrast, few ERGs reported that the MS did not satisfy the intervention or outcome elements of the decision problem. The majority of submissions appear to have adhered to the NICE base-case scenario and prompted no criticism.
The ‘reliability and validity’ theme was related to issues on the robustness or limitations of the MS findings. A total of 27/30 ERG reports stressed the presence of uncertainty within the analyses, thus highlighting the lack of certainty surrounding the results presented in the MS. The principal causes of uncertainty identified by the ERGs concerned the exaggerated effect of the technology from the analysis, especially the relative efficacy of the technology versus relevant comparators; the safety of the technology and uncertain levels of risk for different populations.
The ‘content’ theme referred to the amount and quality of trial evidence presented in the MS. This attracted comments from the ERGs when only one or two randomised controlled trials were evaluated in the MS; where there were few head-to-head trials available for inclusion and/or the quality of the included trials was poor.
Suggestions for manufacturers
From the thematic analysis of the ERG reports and clarification letter analysis, it was possible for the team to identify key suggestions for ways to improve the quality of evidence submitted to NICE by manufacturers. These were developed further at a workshop held in April 2010 with 50 participants from 10 ERG groups, the National Institute for Health Research (NIHR) Evaluation, Trials and Studies Coordinating Centre (NETSCC) and NICE. These suggestions are shown in box 1.
Suggestions for manufacturers for the preparation of submissions to the NICE STA process
Manufacturer submissions should be comprehensive, clearly written, appropriately copy edited and internally consistent.
Definitions for all key terms and abbreviations in the manufacturers' submissions (MS) should be provided.
There should be transparency in the reporting of methods and analyses.
Where applicable, reviews should adhere to internationally accepted standards for conducting and reporting reviews.
Where there is a single clinical study, the study report and protocol should be an addendum to the submission.
There should be clear reporting of methods and results used for indirect comparisons.
The submission should provide relevant and sufficiently detailed data related to clinical progression, outcomes and adverse events.
There should be clear and concise rationale for the synthesis of clinical data.
Clear rationale should be provided for the types of analyses chosen for use in the submission.
There should be clear and concise rationale for the development of economic models and the assumptions used to develop models need to be provided.
A systematic identification and selection of utility values should be included where appropriate in the MS.
Reviewing of model parameter values should be comprehensive and transparent. How comprehensive depends on how critical the parameter is to the appraisal.
Discussion
This research used ERG reports and clarification letters as sources of information to identify common issues and concerns with evidence submissions from manufacturers as part of the NICE STA process. It is worth emphasising that the ERGs made many positive comments about the submissions included in this study. The analysis of the ERG reports did, however, identify criticisms of the conduct of the systematic review within the MS as well as criticisms of the data being used, especially data employed in cost-effectiveness models. Major issues regarding poor reporting of processes used in the MS were also identified. The majority of clarification points related to economic data analysis and covered issues such as inconsistencies between the clinical and economic sections of the submission, queries regarding sources of data and their use in the economic analysis, questions about modelling decisions, data queries and requests for additional analysis. These issues covered both what evidence was presented by manufacturers and how it was presented. They highlight the frequent lack of clarity regarding methods of review synthesis and methods used for indirect comparisons. The inconsistencies within the MS reflect that several groups within the pharmaceutical company may contribute to each MS.
The analysis of the ERG reports applied validated methods and used multiple reviewers to check and verify data and analyses. All data used in this analysis were available on the NICE website (http://www.nice.org.uk) and therefore in the public domain. Only the first 30 completed ERG reports were included due to time and resource constraints. These were the first 30 STAs and some of the critical appraisal methods currently used by the ERG teams have been developed over time and may not be accounted for in these analyses. At the time of writing (December 2011), the ERG reports for 94 STAs had been completed so the analysis presented here includes approximately a third of completed STAs to date. There may be some misinterpretations in these analyses as documentary analysis was the only method used. Analysis was limited to what was reported in the ERG reports. The research would have been strengthened by interviewing manufacturers, ERGs and NICE to gain a more in-depth understanding of the issues. This research dealt with the ERGs' interpretations of manufacturers' submissions. It was beyond the remit of this work to explore the actual submissions themselves or the relationship between the quality of submissions and subsequent decisions as well as changes in submissions over time. These are potential areas of further research. Only 21 clarification letters were examined in this analysis as some early STAs did not have clarification letters and others were unavailable. The approach provides an important overview of the commonalities of the queries put forward in the clarification letters. Clarification letters are currently being developed using a more structured approach. Recently, NICE has made changes to the MS template through a consultation process. All these changes may impact on the issues presented here, although the team involved in this research continues to be involved in the NICE STA process and feels that the issues will remain the same in the near future.
Many of the issues identified in this analysis are similar to those identified by Hill et al13 in their review of the economic analyses submitted as part of the appraisal process in Australia. Of the 249 problems identified in their analysis, 154 (62%) related to uncertainty in the estimates of comparative clinical efficacy and 71 (28.5%) related to modelling issues, which included clinical assumptions or cost estimates, used in the construction of the economic models.
There has been much speculation regarding the future of NICE and the STA process.14–16 Although the future of the NICE STA process is unclear, these suggestions are relevant to any subsequent process, which involves the production of a submission of evidence by manufacturers. NICE is set to retain its role in recommending drugs for use in the NHS within the new value-based pricing approach,17 and the same sorts of information will be needed to inform the decision-making process. Claxton et al18 discussed some of the pros and cons of value-based pricing several years ago and argued that the cost-effectiveness threshold has a key role and that its assessment should be transparent and based on independent scientific analysis. As NICE moves towards the use of value-based pricing in January 2014, in spite of misgivings reported during the consultation process,19 issues about the definition of ‘value’ in medical terms will be central and have implications for other countries.20 For many drugs going through the STA process, available clinical data (relevant population and/or comparator) do not meet the decision problem or do not represent the UK population, and standard treatment will vary between countries. This makes it crucial that any decision-making process that relies on the submission of evidence by manufacturers will require a clearly written and internally consistent submission with justification for the use of data sources.
The relationship between the quality of the MS and the effect this has on the ability of the AC to make an informed decision on a technology has not yet been clearly investigated and this warrants further research. This work should include evaluation and monitoring of submissions to determine whether or not the suggestions set out here have been heeded by manufacturers. Manufacturers should be offered training and support to ensure that submissions meet the needs of the AC. Further research should explore with manufacturers how they have changed their submissions based on past experience with the STA process and the types of issues that have been raised in clarification letters. Improved dialogue with manufacturers will ensure the development of a better understanding of the need for clear submissions. It is hoped that uptake of these suggestions by manufacturers will result in more transparent and internally consistent submissions and will improve the efficiency of the current NICE STA process and any subsequent process. The poor quality of submissions can cause delays to the process as there may be the need for further data requests from manufacturers before the AC can make a decision. The poor quality of submissions does also mean that the key issues may not be explored fully by the ERGs and therefore by the AC, potentially having an impact on the decisions that are made. The appraisal and decision-making process would be enhanced and facilitated by the improvements suggested here.
References
Footnotes
To cite: Kaltenthaler EC, Dickson R, Boland A, et al. A qualitative study of manufacturers' submissions to the UK NICE single technology appraisal process. BMJ Open 2012;2:e000562. doi:10.1136/bmjopen-2011-000562
Contributors ECK and RD conceived the study. CC, ECK, PF, RD and AB were involved in data extraction. CC and RD did the qualitative analysis. ECK wrote the first draft of the paper. All authors contributed to the writing and revising of the manuscript. ECK and RD are the guarantors.
Funding This project was funded by the NIHR Health Technology Assessment programme and parts are published in full in the Health Technology Assessment journal series. Visit the HTA programme website for more details http://www.hta.ac.uk/project/1921.asp. The views and opinions expressed therein are those of the authors and do not necessarily reflect those of the Department of Health.
Competing interests None.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement More information on the analyses presented in this paper is available in Kaltenthaler et al (2011) and by contacting the corresponding author at e.kaltenthaler{at}sheffield.ac.uk.