Article Text

Download PDFPDF

A qualitative study of manufacturers' submissions to the UK NICE single technology appraisal process
  1. Eva C Kaltenthaler1,
  2. Rumona Dickson2,
  3. Angela Boland2,
  4. Christopher Carroll1,
  5. Patrick Fitzgerald1,
  6. Diana Papaioannou1,
  7. Ronald Akehurst1
  1. 1School of Health and Related Research (ScHARR), University of Sheffield, Sheffield, UK
  2. 2Liverpool Reviews and Information Group (LRiG), University of Liverpool, Liverpool, UK
  1. Correspondence to Dr Eva C Kaltenthaler; e.kaltenthaler{at}


Objectives As part of the National Institute for Health and Clinical Excellence (NICE) Single Technology Appraisal (STA) process, manufacturers present submissions outlining the clinical and cost-effectiveness of new technologies. These submissions are critically appraised by Evidence Review Groups (ERGs), who produce a report, which forms part of the evidence considered by the NICE Appraisal Committees. The purpose of this research was first to identify common issues and concerns identified by the ERGs in their analyses of manufacturers' submissions (MS). The aim was then to use these as a basis to develop feedback for manufacturers.

Design A qualitative study using a content analysis approach to examine two sources of evidence, the first 30 ERG reports and 21 clarification letters associated with these STAs.

Setting UK HTA programme.

Primary and secondary outcome measures Common issues and concerns in MS.

Results There were positive comments regarding the quality of the MS, many of which were clearly written. The majority, however, were generally of poor quality and issues and concerns identified across the ERG reports and clarification letters included: criticisms related to the data being used especially data employed in the cost-effectiveness model, failure to perform a necessary analysis and poor reporting of processes used in the MS. Aspects of the decision problem were also often poorly or inadequately addressed by manufacturers. The majority of points raised for clarification related to the economic data analysis. Internal inconsistencies between the clinical and economic sections of the submission were frequently highlighted. These were used as the basis for the development of 12 suggestions for manufacturers.

Conclusions Much can be done to improve the quality of MS in the NICE STA process. Suggestions include the need for clear and transparent reporting of methods and analyses.

This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: and

Statistics from


  • To cite: Kaltenthaler EC, Dickson R, Boland A, et al. A qualitative study of manufacturers' submissions to the UK NICE single technology appraisal process. BMJ Open 2012;2:e000562. doi:10.1136/bmjopen-2011-000562

  • Contributors ECK and RD conceived the study. CC, ECK, PF, RD and AB were involved in data extraction. CC and RD did the qualitative analysis. ECK wrote the first draft of the paper. All authors contributed to the writing and revising of the manuscript. ECK and RD are the guarantors.

  • Funding This project was funded by the NIHR Health Technology Assessment programme and parts are published in full in the Health Technology Assessment journal series. Visit the HTA programme website for more details The views and opinions expressed therein are those of the authors and do not necessarily reflect those of the Department of Health.

  • Competing interests None.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Data sharing statement More information on the analyses presented in this paper is available in Kaltenthaler et al (2011) and by contacting the corresponding author at e.kaltenthaler{at}

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.