Article Text
PDF
Abstract
Objectives To assess the extent and type of data redaction in all active technology appraisals (TA) and highly specialised technology (HST) evaluations issued by the National Institute for Health and Care Excellence (NICE) from its conception of the institute to September 2019. To propose policy recommendations for transparency.
Methods Structured audit to establish extent of data redaction—proportion of appraisals and specific data categories and assess redaction by: indication, appraisal process, manufacturer, type of data—price, adverse events (AEs), clinical (excluding AEs), incremental quality-adjusted life-years. Longitudinal analysis over 20 years.
Results All TAs with available documentation and active recommendations (n=408) and HSTs (n=10) published from March 2000 to 11 September 2019 have been assessed for data redaction. Overall, 333 TAs (81.6%) have data redaction, 86 (25.8%) of them are heavily redacted. Clinical data (excluding AEs) are redacted in 268 (65.7%) appraisals, AE data in 128 (31.4%), price in 238 (58.3%). In total, 87% of oncology appraisals have redacted data vs 78% of non-oncology appraisals. 91% of single TAs have redacted data vs 59% of multiple TAs. 25% of final guidance documents (e.g. Final Appraisal Determination - FAD) do not report one or more instance of clinical data. Data redaction increased substantially over time, and is currently at its highest level with 100% of TAs having at least some data redaction in 2019/2020, 96% of appraisals in 2018/2019% and 94% of appraisals in 2017/2018. All 10 HST evaluations have redacted data, with 4 of them being heavily redacted.
Conclusions Documents supporting NICE TA and HST recommendations are significantly redacted, thereby concealing clinical and economic data of importance to patients, clinicians and researchers. Documents remain redacted on the NICE website for years. Policy change is required to ensure transparency of data underpinning NICE’s decisions.
- health policy
- quality in health care
- public health
- ethics (see medical ethics)
Data availability statement
Data are available in a public, open access repository. Data are available in a public, open access repository and can be used by anyone with no restriction-free access.Osipenko, Leeza. (2021). Ledger of data redactions in NICE Technology Appraisals (Data set). Zenodo. https://zenodo.org/record/5236080.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
This is the first comprehensive audit of data redaction practices in the technology appraisal programme at National Institute for Health and Care Excellence (NICE) over the 20-year period.
This study establishes the extent of data redaction, type of redacted data and documents across different NICE processes.
The findings can be externally validated by checking the original data made available by the author in the open-source repository.
No systematic investigation has been performed to identify if academic-in-confidence data redacted from the NICE appraisal documentation have been published at a later date or are available through other sources, such as clinical study reports.
Introduction
National Institute for Health and Care Excellence (NICE) was established in 1999. One of the key functions of NICE is to produce guidance on medicines, devices and other interventions for their use in the National Health Service (NHS). Before NICE issues any recommendations via the health technology assessment (HTA) process, a medicine or a device must obtain a marketing authorisation from a regulatory agency: the European Medicines Agency (EMA), the UK’s Medicines and Healthcare Regulatory Authority or accredited organisations issuing CE marks. The regulators review submissions by companies, which include laboratory methods, manufacturing specifications, preclinical and clinical data in order to make decisions on safety and efficacy of products. NICE accepts the decisions of the regulators and then evaluates further clinical and economic aspects of these products to ensure that all patients in the NHS have equitable access to the most clinically and cost-effective treatments that are available.1
Technology appraisals (TA) is the core HTA programme that has been conducting multiple TAs (MTAs) since the inception of NICE. The single TA (STA) process was introduced in 2006 and gradually overtook MTAs. TAs predominantly focus on pharmaceuticals. Over time, other HTA programmes were introduced to evaluate interventional procedures, devices, diagnostic tests and medicines for ultrarare diseases—the programme for highly specialised technologies (HST). Each appraisal process is a rigorous exercise, which includes the evaluation of the manufacturer’s submission by independent academic groups, public consultation and decision making by independent committees (not employed by NICE), whose conflicts of interest are openly managed.1
The public, including doctors, patients and researches, has many potential routes to obtain detailed scientific information about new medicinal products and devices: regulatory agencies, HTA agencies, company websites, commercial and public data repositories, journal publications, etc. However, the availability, quality and visibility of such data is not guaranteed. Despite successes in trial reporting,2 3 publication of clinical study reports (CSRs) by regulators and industry,4 5 and data-sharing initiatives,6 access to data remains a problem due to resource constraints of the providers,4 7 commercial interests4 8 and publication bias.9
Industry-sponsored publications in scientific journals may not reflect issues and warnings issued by the regulators.10 Due to their format, many publications lack much of the detail available to the agencies11 and are at a high risk of interpretation bias.9 The EMA started putting CSRs in the public domain in 20155 and then stopped in 2018 due to workload constraints and lack of resources.12 It remains suspended due to ongoing business continuity linked to the COVID-19 pandemic. The EMA is publishing clinical data for COVID-19 medicines in line with its exceptional transparency measures for treatments and vaccines for COVID-1913 Health Canada (HC) started to release CSRs in March 2019 and plans to make data available within 120 days after the agency’s decision.4 The number of available reports remains small. The Food and Drug Administration initiated a pilot programme in 2018 to improve transparency to drugs evaluation and make CSRs available online.14 15 This initiative did not challenge the status quo. It has been decided that international harmonisation on data provision is required and until then data can be made available via individual requests.16 By contrast, NICE has and continues to enable access to all background documentation for currently active guidance. The processes and methods that NICE employs are structured and transparent. Every appraisal (unless replaced, terminated or withdrawn) has an online history page, which contains a comprehensive list of background documents on the technology and the process. These usually comprise of the following: the scoping document, list of consultees, manufacturer submission(s), consultees comments, assessment report(s) from the academic group, appeal documents, equality impact assessment, committee slides, appraisal consultation document and final appraisal determination (FAD). FAD is the output of the decision issued by NICE, which interprets the evidence and explains recommendations reached by the committee. This document is written in a clear language for the public, clinicians and patients to understand the underlying rationale for the decision. The final guidance is issued on the webpage for each TA summarising the recommendation, the technology, the committee’s discussion, research recommendations (if any), composition of the committee, implementation and additional information for the public.
Unfortunately, access to the documentation does not guarantee the visibility of data due to redaction practices. For example, GlaxoSmithKline has released CSRs for over 2500 studies; however, data redaction (especially relating to adverse events (AEs)) in these reports make them only moderately useful for researchers.4 Publication of heavily redacted clinical trial protocols remains a problem,17 18 in part because the level of redaction in CSRs submitted to the regulators is unknown, which has been criticised in the past.19 Bullement et al have shown that clinical data redaction in NICE STAs, which have confidential discounts (on price) via a patient access scheme (PAS)20 is extremely high.21
Data redaction at NICE is sanctioned either as commercial-in-confidence (CiC) or academic-in-confidence (AiC). The NICE process guide defines CiC as ‘information, the disclosure of which in public could have a significant impact on the commercial interests of a particular company22 ’ and AiC as ‘information, the disclosure of which in public would seriously jeopardise the ability of the data owner to publish the information in a scientific paper’.22 EMA has a policy on data redaction, according to which, clinical data are not considered to be commercially confidential information23 besides a few exceptions limited to novel trial designs, and exploratory endpoints and biomarkers. NICE has an agreement with the Association of the British Pharmaceutical Industry (ABPI) on ‘guidelines for the release of company data into the public domain during a health technology appraisal.’24 The title of the document implies that this agreement is applicable only to the actual appraisal process. However, the influence of this document spreads far beyond this remit, as many NICE documents remain redacted after the issuance of the guidance.22 The NICE TA process guide is reflective of this agreement and does not provide further clarity on data transparency once the appraisal process is completed and the recommendation is made public. It actually reinforces that ‘a version (of manufacturer submission) for public release after the committee has met, in which all the confidential information is redacted, should be provided by the company’ and that ‘the data owner retains the right to make a final decision about the release of confidential information into the public domain.’23
Data held and released by the agencies differ in quantity and content. Regulatory agencies hold the most comprehensive sets of data for each product. Usually, these are highly technical and voluminous documents, and clinicians and patients may not have sufficient resources and knowledge to interpret these documents effectively.25 Considering the rigour and transparency of the processes at NICE, the NICE website is an important source of evidence for the general public because: (1) background documents are available for each active guidance, (2) summaries of clinical and safety data focus on clinically relevant issues and (3) documents are written in an accessible language. Thus, NICE can play a critical role for evidence provision to the public in England and Wales. On a positive recommendation from NICE, medicines and devices become available to patients in the NHS. It would seem logical that at this stage (usually within 90 days), that all clinical data concerning the product are made publicly available.
Back in 2012, Strech and Littmann drew attention to the fact that the EMA and NICE restrict full access to unpublished evidence,26 implying that the decision-making practices of the agencies have failed to account for different specifications of public interest. The authors requested the EMA and NICE to revise their policies on data transparency suggesting that the default restriction to data should be changed to a default access to data.26 No action has been taken by NICE. Considering worrying findings on data redaction practices at NICE,22 this study aimed to conduct a complete audit of data redaction across all active NICE TAs and HSTs issued since the inception of the Institute 20 years ago, and establish the actual level and nature of data redaction practices at NICE.
Methods
Creation of TA/HST ledger
An Excel workbook (see data repository27) was used to create a ledger of all TAs and HSTs conducted by NICE between March 2000 (publication of TA1) and 11 September 2019 (publication of TA600). The status of each appraisal/evaluation was established from reviewing them individually on the NICE website (nice.org.uk). In the ledger, all TAs are arranged by the year of publication and classified by the status of the TA as of September 2019: recommended, in research, not recommended, replaced, terminated. The list of HSTs is presented separately.
Search strategy
All background documents for active (recommended, not recommended and in-research) TAs and HSTs were identified on the NICE website. These documents over the years follow varying formats but in general, have a consistent structure presenting information according to the NICE’s process22 and methods28 guides (which had a few editions since the establishment of the Institute) and evidence submission templates. For each appraisal, the following documents were searched for data redaction: FAD, assessment group (AG) or evidence review group (ERG) report (including erratum corrected documents and online supplemental analyses post consultation or submission of additional data by the company), manufacturer’ submission (MS), and committee slides (if available). FADs were searched for the terms CiC and AiC. All reviewed documents were visually inspected for black boxes concealing text, tables and graphs as a means of redaction. In addition, word search for CiC and AiC was performed in each document.
Data extraction
Data extraction was performed by one person (the author) into a template.27 For each appraisal, data were extracted on the basis of the following: TA number, sponsor company, year, process (STA)/MTA), technology name, technology type (pharmaceutical/non-pharmaceutical), indication, AiC and CiC (unrelated to price) data from FAD, redacted items (by type) from ERG/AG document, committee slides and MS. Appraisals from 2020/2021 were not included in the audit to extract specific data, however all of them were checked by the author for presence of data redaction to see if data redaction practice remains in place to date.
Assessing extent of redactions
In the NICE documentation data redaction can take two forms—either blacked out sections of the documents concealing the data (pertinent to background documents) or an explanation that data cannot be presented due to AiC or CiC designation (pertinent to FAD documents). To facilitate quantitative assessment of redactions in NICE appraisals, the following scoring system was adopted. Each appraisal was assessed for data redaction on: price, clinical data (excluding AEs), incremental quality adjusted life years (QALYs) and AEs. The score of zero was awarded to each item if no redactions were identified. The score of 1 was awarded to each item if (A) the appraisal referred in the FAD to confidential agreements and price discounts or if prices were redacted in any of the background documents, while the company did not have a commercial agreement or a PAS; (B) if any clinical data were redacted (eg, outcomes, results of indirect comparison analyses, utilities, baseline characteristics of patients in trials, time to treatment discontinuation, number of responders, modelled outcomes data); (C) if QALY gains were redacted and (D) if any data in the AE tables or paragraphs discussing AEs were redacted.
The aggregate score (0–3) was developed to quantify the level of redactions for the entire appraisal. Zero was awarded to appraisals with no redactions. Appraisals where redactions were minor—one paragraph or one item—were noted (online supplemental table 12, please see reference27) but no additional points for redaction were awarded. The score of 1 was awarded to those appraisals where either price or clinical data were redacted but not both. The score of 2 was awarded to appraisals: (A) where both price and clinical data were redacted; (B) where price were available, but clinical data redactions were substantial and/or FADs withheld clinical data under AiC or CiC. The score of 3 was awarded to heavily redacted appraisals (many items and pages redacted leaving little or no visible clinical data).
Data analysis
Data analysis (descriptive statistics in Excel) was performed by one researcher, the author. Data were analysed by the process type (STA, MTA, HST), by recommendation status (positive, negative and in research), by technology type (pharmaceutical, non-pharmaceutical), by company (including only companies which products had 10 or more currently active STA recommendations). The proportion of FADs was calculated where clinical data were not reported due to CiC and AiC data non-disclosure. Longitudinal analysis was performed to assess the pattern of data redaction over the 20-year period. A separate analysis was undertaken to explore in more detail clinical and economic data redaction patterns. TAs and HSTs were identified that redacted the following data: (A) background characteristics of patients, (B) indirect comparisons (analysis and/or results), (C) economic model (D) clinical or modelled outcomes data, which is designated as CiC (rather than AiC), (E) ERG/AG comments/criticism and (F) clinical data redaction when price information is available.
Qualitative evaluation
A non-systematic approach was taken to present illustrative examples of data redaction practices from selected appraisals (n=25, online supplemental table 13, please see reference27). EMA and HC CSRs were reviewed to assess data availability for the four appraisals, where NICE redacted data.
Validation
Validation of data extraction and analysis was performed by the researcher during the review of the data repository27 to identify appraisals for the qualitative analysis. Each entry was rechecked for contents and completion of records. Additional searchers were performed where data were missing, and any errors and inconsistencies were corrected. All calculations were internally validated to ensure that the total number of appraisals in the analysed data set remained constant.
Patient and public involvement statement
Not applicable for this type of research.
Results
As figure 1 shows, 191 (32%) appraisals were not reviewed because they were replaced (n=125) or terminated (n=66) and their documents were not available on the NICE website. A total of 409 (68%) TAs were reviewed, and TA77 (MTA) was excluded from the analysis because all documents pertinent to this appraisal were missing on the NICE website. Out of the remaining 408 reviewed TAs, 38 (9%) were negative (that is, they did not include any positive recommendations) and 370 (91%) were positive (or included at least one positive recommendation).
Flow diagram of technology appraisals (TAs) and highly specialised technology evaluations (HSTs), which has been adapted from PRISMA to depict selection of TAs and HSTs for analysis. Briefly, a total of 600 TAs and 10 HSTs)were identified in the original data set. Records were screened to determine active appraisals and after removing terminated, missing and replaced appraisals, 408 TAs and 10 HSTs were included in the analysis. MTA, multiple technology appraisal; PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses; STA, single technology appraisal.
Across appraisals, reviewed manufacturer submissions range between 100 and 450 pages, ERG reports between 50 and 300 pages and AG reports between 80 and 450 pages. Out of all available appraisals/HST evaluations (n=418), 343 (82%) redacted data. Specifically, 238 (58.3%) TAs and 9 (90%) HSTs redacted price data, 268 (65.7%) TAs and all HSTs redacted at least some clinical data, 141 (34.6%) TA and 1 (10%) HST redacted incremental QALYs, 128 (31.4%) appraisals and 6 (60%) HSTs redacted AEs (figure 2). Overall, 86 (21.1%) TAs and 4 (40%) HSTs were heavily redacted (score of 3). For example, TA596 committee papers (page count—296, out of which over 60 pages are slides and organisational pages) have 89 pages with data redaction, many of these pages are blacked out completely.29
Presence and extent of data redaction in technology appraisals (TAs) (n=408) and highly specialised technology evaluations (HSTs) (n=10) over 80% of appraisals had some data redaction. A total of 90% of STAs, 60% of MTAs and 100% of HSTs had some data redacted. In TAs at least some clinical data were redacted in over 65% of appraisals, QALYs in 35%, adverse events in over 30% and price in nearly 60%. A similar trend is shown for HSTs. AEs, adverse events; MTAs, multiple technology appraisals; QALY, quality-adjusted life-year.
The total number of positive oncology appraisals reviewed was 167. Of these, 146 (87%) had data redacted compared with 159 (78%) of non-oncology appraisals. In total, 59% of MTAs had data redactions compared with 91% of STAs. Six percent of MTAs were heavily redacted compared with 28% of STAs.
Data redaction increased dramatically over time (figures 3 and 4). At present, data redaction in TAs is at its highest level with 100% of TAs having at least some data redacted in 2019, 2020 and 2021. Ninety-six per cent of appraisals redacted data in 2018/2019 and 94% in 2017/2018. Over the past 10 years, STAs almost completely replaced the MTA process and since data redaction increased over the years this is likely to explain why STAs are more redacted than MTAs.
Redaction of data over time in NICE technology appraisals (n=408). This figure shows how data redaction has increased steadily and considerably over the past 8–10 years. In particular, price data have been redacted since 2008, in spikes that reflect over 50% of TAs by 2012. Accompanying this trend, quality-adjusted life-year (QALY) and adverse events data redaction have risen over the past decade. Importantly, at least some clinical data have been redacted in more than 50% of appraisals as early as 2005. NICE, National Institute for Health and Care Excellence; TAs, technology appraisals.
The pattern of data redaction in technology appraisals (n=408) at NICE. Over time, the amount and type of data redacted over the past 20 years has increased substantially. Initially, there were no data reactions in 2000/2001. In the first 6–7 years, only minimal price and clinical data were redacted. However, since 2008/2009 onwards, the amount of price and clinical data redaction has increased. The volume of data redaction has increased substantially since 2012. NICE, National Institute for Health and Care Excellence.
Figure 5 shows the extent of data redaction in the documents of the 11 companies that have ten or more currently active STA recommendations.
Data redaction in National Institute for Health and Care Excellence (NICE) technology appraisals (TAs) documents by 11 companies with 10 or more active single technology appraisals (STAs) recommendations (2006–2019). This figure shows data redaction practices for 11 companies which underwent many appraisals at NICE and had 10 or more active STAs at the time of the analysis. In almost all cases, price and some clinical data were redacted, and heavy data redactions are notable for all presented companies. Price and/or clinical data redactions were common for all companies. BMS, Bristol Myers Squibb.
In non-recommended TAs (n=38), 28 (74%) had redacted data. Specifically, price was redacted in 16 (42%), clinical data in 22 (58%), QALYs in 11 (29%) and AEs in 9 (23.7%) appraisals. The total number of non-pharmaceutical recommended TAs reviewed was 25 (6.1% of total) and 14 (56%) of them had redacted data. Specifically, price was redacted in 7 (28%), at least some clinical data in 12 (48%), QALYs in 2 (8%) and AEs in 1 (4%) appraisal.
The total number of in-research appraisals reviewed was 34, where 32 appraisals were for medicines in the Cancer Drugs Fund (CDF) and 2 MTAs with no redacted data. All CDF appraisals had data redactions. Specifically, price was redacted in all CDF appraisals, at least some clinical data in 27 (84%), QALYs in 16 (50%) and AEs in 17 (53%) appraisals.
No black box redactions are used in any of the FADs; however, these documents refer to CiC and AiC statements to conceal information. Details on discounts and commercial agreements are CiC and are not reported in the FADs (besides a few exceptions of the complex schemes).20 Ninety-four (25%) of all FADs withheld at least some clinical data as CiC or AiC. Final recommendations presented on the webpage are usually reflective of the FADs, and for the FADs not reporting data under CiC or AiC no discrepancies were identified with the final recommendations.
Table 1 presents additional insights into data redaction practices across 408 TAs and 10 HSTs. The actual appraisal numbers for each category can be found in the data repository (please see reference27).
Additional insights into data redaction practices in TAs (n=408) and HSTs (n=10)
Indirect comparisons were redacted in 88 (21.6%) appraisals. Baseline characteristics of patients in the trials were redacted in 51 (12.5%) appraisals, 28 (6.9%) appraisals redacted critical comments from ERG/AG and 41 (10%) appraisals redacted at least some clinical data as CiC rather than AiC. Out of 170 TAs that did not have price data redacted, 93 (54.7%) had clinical data redaction, 46 of these appraisals received aggregate scores of 2 or 3 indicating heavy redaction of clinical data.
Minimal redactions (eg, one paragraph) were identified in 34 (8.3%) appraisals.
Qualitative
To illustrate further the approach and extent of data redaction, a more detailed analysis was undertaken of selected appraisals published across different years. For example, TA199 redacted all R programming for the York economic analysis in addition to some clinical outcomes from the Impact-3 trial (change in Health Assessment Questionnaire Disability Index scores and stage 2 AEs).30 TA303 has heavy data redaction across MS and ERG documents, but also the most redacted FAD with nine instances (in addition to price data) of undisclosed clinical data.31 Manufacturer submission for TA170 features redactions on 46 pages out of 136 pages.32 TA355 not only redacted clinical data and indirect comparison analysis and outputs, but also featured heavy redactions of the ERG critique of network meta-analysis, indirect comparisons, and summary of the trial characteristics.33 AG report for TA59 has 136 instances of AiC data redaction.34 TA161 has significant clinical data redaction, but also relies (along with TA160) on the fracture risks calculation by the AG, which uses the WHO algorithm that is AiC. Owing to this situation, the economic model for this appraisal was never released to consultees and commentators during this appraisal.35 Besides TA303, none of these appraisals have a PAS and no logical explanation can be given to explain the egregious redaction of clinical data in TA303 to protect the confidentiality of the discount.
AEs and selective nature of redactions
The NICE website provides a high-level summary or some specific examples of adverse effects for each product. AEs data presented in the NICE background documents is taken from CSRs and usually provides complete summaries of AEs of all grades from the trials. This study finds that in 31.4% of TAs and 60% of HSTs, these data are being redacted fully or partially. For example, TA485 MS provides detailed AEs data from the MOBILITY trial (different doses of sarilumab vs placebo), the MONARCH trial (adalimumab vs sarilumab) and the TARGET trial (different doses of sarilumab vs placebo), but all AEs from the ASCERTAIN trial (tocilizumab vs sarilumab) on pages 177–178 are redacted.36 The ASCERTAIN trial reported safety and tolerability of subcutaneous sarilumab compared with intravenous tocilizumab in patients with rheumatoid arthritis was published in May 2019,37 and reported no clinically meaningful differences in AEs between these two drugs. NHS patients and clinicians had no access to this information for 19 months and it is still not available on the NICE website. No CSRs for sarilumab are posted on the EMA or HC websites.38 39 There are many treatment options for patients with rheumatoid arthritis, however, patients for whom existing treatments do not work and who are seeking to switch to other products that may have a better AE profile, provision of such information could be critical.
TA292 (aripiprazole for bipolar disorder in adolescents) has a lot of clinical data redacted in background documents, but in the FAD, data on the incidence of clinically significant weight gain at 30 weeks was considered to be AiC by the manufacturer and was not presented, as well as data on the proportion of participants with a body mass index on or above the 95th percentile at week 30.40 This information would have been critical for clinicians and patients to know. This product has no PAS, but these data have never been made available on the NICE website. The CSRs for aripiprazole are not available on the EMA or HC websites.38 39
The pattern of data redaction in the NICE documents is highly variable. From black boxes concealing the data, it is impossible to establish if redactions are AiC or CiC. For example, in TA462 the ERG report actually states on p400 of the committee papers: ‘The ERG has taken a conservative approach and marked up, as AIC or CIC, any unmarked data whenever we were aware it was marked as AIC or CIC elsewhere in the submitted evidence.’41 Under AiC, presumably the entire data set from a study (all utility values or all AEs or all outcomes data) would not be made available; however, this is not necessarily the case. Data redaction can focus on specific parameters. For example, in TA395 (in table 34 MS, p76, please see reference42) has half of the data in the table available on AEs and the other half the data are redacted; table 35 AE data are fully redacted (please see reference42) but in table 36 (please see reference42), two values of AEs are available and 11 are redacted.42 Furthermore, another three tables documenting AEs are fully redacted.42 These are the values from the same clinical trials. Another example is TA405 presenting selective redaction of AEs43 while these data are available on the EMA website38 (figure 6).
Trifluridine–tipiracil for previously treated metastatic colorectal cancer. Manufacturer submission to National Institute for Health and Care Excellence (NICE) versus European Medicines Agency (EMA) example showing redacted adverse events data from a phase II trial, from the manufacturer submission to NICE as part of TA405. This information is in stark contrast to the unredacted identical data available from the clinical study report submitted by the company to the EMA. CSR, clinical study reports; TA, technology appraisal.
Unredaction practices
Establishing whether data unredaction takes place was not possible because no versions of documents stating: ‘unredacted version’ were identified. Possibly, unredacted data are presented in the TA163 ERG document in tables highlighted in yellow (pp16-17, 53–54)44 and in TA426 (reconsideration of TA251) in different colour (pages 10–17, 28, 33 of the committee papers for TA251)45; however, this TA features redacted data on indirect comparisons and AEs.46
Some appraisals made attempts to indicate data availability, but no action was taken to make them available. The ERG report for TA298 states: ‘please note that the ERG report was prepared before publication of the RADIANCE and REPAIR trials, therefore, some of the redacted information in this report is no longer confidential’.47 This ERG document remains heavily redacted 8 years later. TA324 (2014) is a partial update of TA88, and UKPACE trial data are not presented and cited as CiC on pages 85–86,48 despite the UKPACE trial being published in 2005.49 Perhaps information for TA324 was copied from TA88 (2005). Data in relation to UKPACE trial in TA88 remain redacted (416 instances of CiC removed data).50 In addition to specific data related to the trial, the documents are redacted for the AG comments on detection bias, on attrition bias, and on the statistical power of the UKPACE study. Such data should have been made available to researchers and clinicians helping them critically understand the quality of the UKPACE trial.
There are examples of data redaction that clearly illustrate the lack of due diligence by NICE. TA450, where the committee papers51 are heavily redacted, was published on the NICE website on 28 June 2017. The TOWER trial results were published 3 months before on 2 March 2017 in the NEJM52 with all the supplemental material (detailed protocol, data on outcomes and AEs tables). The background documents for this appraisal remain redacted. Moreover, since many journal publications are behind publisher paywalls, access to the peer-review published literature is not freely available to patients and the public.
A heavily redacted TA592 was published on 9 August 2019.53 All the redacted information related to clinical efficacy and the safety of cemiplimab is publicly available on the website of the Canadian regulator HC (figure 7). Documents related to this product on the HC website are redacted only for the personal patient data.39 Canada released all the data on the 23 August 2019, which is only 2 weeks later than NICE. Another example (figure 8) relates to unredacted data on safety and efficacy of abemaciclib, which was posted on the HC website on 12 August 2019,38 while a heavily redacted appraisal (TA579) by NICE was published on 9 May 2019.54 For both appraisals, documents on the NICE website remain redacted.
Cemiplimab for treating metastatic or locally advanced cutaneous squamous cell carcinoma. Manufacturer submissions presented on the National Institute for Health and Care Excellence (NICE) website and on the Health Canada website (as of October 2019). Example of an entire table redaction of clinical tumour response outcomes from a manufacturer’s submission to NICE as part of TA592. In contrast, the identical information is clearly presented in full (unredacted) in the clinical study report submitted by the manufacturer to Health Canada.
Abemaciclib with fulvestrant for treating hormone receptor-positive, HER2-negative advanced breast cancer after endocrine therapy: Manufacturer submissions posted on the National Institute for Health and Care Excellence (NICE) website and on the Health Canada website (as of October 2019). Example of adverse events data redaction from a manufacturer’s submission to NICE as part of TA579. In contrast, the identical information is clearly presented (unredacted) in the clinical study report submitted by the manufacturer to Health Canada.
CiC redaction of clinical data
In contrast to AiC, CiC data carries no obligation for even a theoretical future transparency. As this study shows, at least 41 (10%) FADs features CiC clinical data redaction (table 1) and only 41% of these appraisals, concealing clinical data as CiC, have commercial agreements with NICE. For example, TA34 HTA report has 67 ‘Commercial in confidence information removed’ statements and 25 CiC redactions in tables.55 TA337 background documents and the FAD56 are full of CiC concealment of clinical data, which range from baseline characteristics of patients, to trial outcomes, to the treatment-related AEs, severe AEs and AEs that resulted in people stopping the study. In TA86, survival data are redacted (20 instances) with the reference to the unpublished study by Goss et al listing it as CiC.57 Review documents from 2010 do not refer to the ‘unpublished’ Goss study. Further search for this publication did not yield any results, meaning that it has never been published. It is unclear why an unpublished study was classified as CiC and not AiC in the first instance. These appraisals have no PAS.
Discussion
The NICE processes have been rightly praised for their high level of transparency.58 The agency details names of the reviewers and conflicts of interest, types of evidence reviewed and evaluation steps and processes, meetings are held in public, and documentation is made available online. However, over the years, NICE has taken a lenient position on data redaction practices.
In total, 82% of NICE’s documents pertinent to TAs and HSTs (100%) have varied levels of data redactions. Over the past 20 years, data redaction has increased substantially. The volume of clinical data redaction is egregious. This study reveals that NICE does not unredact data even when availability of data in the public domain is obvious.
Policy implications
In Germany, it is not possible to redact data in the HTA process.58 Both German agencies responsible for HTA, the IQWiG and G-BA, who are involved in the assessment report and appraisal documentation, respectively, have a strong legal basis for this transparency.58 There is a law in place in Germany whereby the assessment of all new drugs that enter the German market (AMNOG, 2011) requires that a submission of complete data from the manufacturer and the information relating to the underlying decision making are in the public domain.59 60 In addition, full transparency of all information and data are required in the IQWiG’s assessment report to enable full commenting procedure and decision-making transparent.59 Importantly, strict penalties are issued in instances where incomplete data are submitted. Thus, any redaction by default would be considered incomplete data. This enhanced transparency in Germany has resulted in substantially more information being available on new drugs at the point of market entry.60 The German example clearly illustrates that industry complies with transparency if public agencies enforce it. Germany is a bigger pharmaceutical market than the UK and traditionally characterised by an earlier launch of pharmaceutical products compared with the UK thus NICE can hardly argue that protection of industry’s commercial interests jeopardises market access. In the current political climate in the UK, there are strong forces to make the system responsive to investment and the innovation agenda. The government’s zeal to protect the industry’s interests during the uncertainty posed by Brexit remains strong as corporations continue to pressure the UK government with threats of moving business outside the UK.61 In May 2019, the UK government has refused to sign up to a global resolution on greater transparency for drug pricing.62 The resolution urges governments and others buying health products to share information on actual prices paid, and pushes for greater transparency on patents, clinical trial results and other factors affecting pricing from laboratories to patients. The position of NICE (supported by the current legal framework) on redacting price data on drugs that have commercial agreements has been strongly favouring interests of the pharmaceutical industry since the establishment of the PAS in 2007.20
Considering that in the NHS, medicines and most devices are provided free of charge, it could be argued that redaction of price data is irrelevant to the patients and taxpayers in England and Wales, but this is a weak argument for a number of reasons. First, in contrast to the French agency Haute Autorité de Santé and the Federal Joint Committee (G-BA) in Germany, NICE, does not explicitly state the judgement with regard to added therapeutic benefit (eg, major, minor, nothing new) of new medicines. This means that we have no way of seeing how the potential benefit has been quantified (some TAs may provide these data in technical documents but require training in health economics to interpret) and compared with existing medications in the same indication. Thus, if a confidentially agreed discount is high enough for the product to be deemed cost-effective according to the NICE methodology, regardless of the actual clinical benefit of this product, it will enter the NHS. This scenario represents a gaping missed opportunity to improve on the level of evidence relating to clinical benefit, which is especially warranted as much of the preclinical research leading to clinical trials is publicly funded.63 Second, as Bullement et al illustrate, NICE’s methodology actually requires redaction of some data in order to keep the price discounts confidential because the release of such data will allow the back-calculation of the price agreed.21 This methodological interdependence results in significant data redaction, especially in recent years where almost every recommended product has a commercial agreement. Third, the lack of pricing transparency in England has implications within and outside the country. Organisations such as ABPI propose that NICE is providing value for money to the NHS. In reality, this is a theoretical assumption rather than a fact64 and taxpayers do not know whether they are paying more or less for medications than other European countries, how big the difference is, what services are being displaced to fund new medicines and devices, and whether modelled assumptions of clinical benefit on which NICE committees make decisions ever materialise in real life. In the meantime, the pharmaceutical industry continues to post top profitability margins,65 and the clinical benefits of the growing number of pharmaceuticals entering the market remain questionable66 67 and might later be disappointing,68 and healthcare system budgets are not coping with the pressure. International efforts should continue to bring price transparency on pharmaceuticals and devices across the globe, and the UK government can help lead this change.
There is no authority to hold the agency accountable to transparency. The legislative support of price and clinical data transparency at the national level will eliminate data redaction practices by NICE, which will serve not only NHS patients and clinicians, but a much wider clinical and research community around the world.
Context of other research
Research and inquiry into data redaction practices is limited. Goldacre in his book ‘Bad Pharma’ brings an example of data redaction in the NICE appraisal of ranibizumab stating that ‘this level of censorship is not an everyday phenomenon’.19 As this study shows, substantial clinical data redaction is indeed an everyday practice, and it is the unredacted data that are becoming a transparency phenomenon. Recommendations that Strech and Littmann made to NICE on data transparency have not been adopted to date.26 Since their publication, data redaction practices have become more prevalent. Bullement et al found that the volume of data censored as AiC in NICE appraisals was extremely high and called for the revision of the current practices. They also concluded that censoring appears to be performed on an ad hoc basis with no consistent pattern in the information censored.21 The findings of this study agree with those by Bullement et al, however, the authors are supportive of the AiC concept (with timely unredaction) while this work finds no grounds on which the AiC clinical data redaction should be practised. Panteli et al found the lack of transparency within HTA bodies on processes and consideration of data, suggesting data sharing between the agencies to improve the situation.69 This research identified asymmetry in data visibility between regulatory agencies and NICE, and reiterates the need for interagency collaboration to maximise data transparency. Finally, EMA starting the release of CSRs in 2014, stated that it is guided by the conviction that public health interests must outweigh any private intellectual or commercial interest.5 Unfortunately, NICE is not following the German example and the call for implementation of similar legal requirements in other countries remains unanswered.59 60
Limitations of the study
First, this study was performed by a single researcher, and while the results were internally validated by the researcher, external validation by an independent source was not possible. While this is a limitation of the study, all information has been extracted from resources in the public domain and is included in the online data repository,27 which allows for external validation and re-assessment. Second, a subjective approach was used to determine an aggregate scoring system to assign levels of the extent and type of data redaction (0–3). However, the pragmatic methodology used to score redaction practices allows for replication. Finally, each manufacturer submission to NICE is accompanied by a checklist of confidential information, summarising the type and location of data redacted, but these checklists are not in the public domain and could not be used as an additional validation tool in this study.
Future research
Future work can analyse data redaction practices in regulatory and HTA agencies to enact policy change towards transparency and joint action to make data available. To unredact all NICE documents, publications that include data marked as AiC and CiC should be identified through searches or direct contact with the sponsors.
Conclusion
Data redaction in NICE appraisals is alarming in volume and context. Transparency of data is not regulated or sanctioned, and it undermines the role of NICE in protecting the health of patients and informing the public while safeguarding the interests of industry. When clinical data are redacted by an HTA agency, patients, clinicians and researchers are deprived of information that might be crucial for their decision making with regard to a specific intervention. Policy change is required to ensure that all clinical data are fully available in the NICE documents for all the products for which NICE has issued guidance. All documents underpinning past decisions on the NICE website should be unredacted to make clinical data visible. International and harmonised policy changes are needed to enable global price transparency.
Data availability statement
Data are available in a public, open access repository. Data are available in a public, open access repository and can be used by anyone with no restriction-free access.Osipenko, Leeza. (2021). Ledger of data redactions in NICE Technology Appraisals (Data set). Zenodo. https://zenodo.org/record/5236080.
Ethics statements
Patient consent for publication
Ethics approval
Not required. This secondary desk research did not involve animals or human participants.
Acknowledgments
LO thanks Dr Lisa Hutchinson for reviewing the manuscript; Dr Till Bruckner and Debra Winberg for filing Freedom of Information requests to NICE.
References
Footnotes
Contributors LO is the sole author of this project. LO came up with the original idea and methodology, conducted all data compilation, extraction, analysis and validation. She prepared all drafts of the manuscript and revisions.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.