Evidence review group approaches to the criticalappraisal of manufacturer submissions for the NICE STA process: A mapping study and thematic analysis

Abstract

Background: The National Institute for Health and Clinical Excellence (NICE) singletechnology appraisal (STA) process was set up as a rapid way to appraise newtechnologies for use within the NHS in England and Wales and has been in placesince 2005.Objectives: This study had five primary objectives: (1) to provide a map of the STA processto date; (2) to identify current approaches to the critical appraisal of manufacturers'submissions (MSs) by Evidence Review Groups (ERGs); (3) to identify recurring themesin clarification letters sent to manufacturers; (4) to provide recommendations for possiblealternative approaches to be used in the critical appraisal process; and (5) to revise thecurrent ERG report template.Data sources: Data for the mapping of the STA process were obtained from the NICEwebsite (www.nice.org.uk). Data for the analyses of the ERG reports and clarification letterswere taken from the reports and letters themselves.Review methods: For the mapping, a spreadsheet was developed to collect data on 22predefined variables related to timings and outcomes. Simple descriptive statistics wereused to analyse the data. For the thematic analysis, a documentary analysis of 30 ERGreports was undertaken. Data on key elements of the MSs, the processes undertakenby ERGs and the strengths and weaknesses of MSs were extracted. A framework of apriori themes was developed. Data were extracted, coded and analysed according toa framework approach. Twenty-one clarification letters were examined and data wereextracted using a set of codes to cover report quality, systematic review methods andclinical/economic issues. The current ERG report template was modified and sent to thecurrent ERG teams for comment. All comments were considered and formed the basis forfurther revisions to the template.Results: Ninety-five STAs were included in the mapping exercise. Many STAs were subjectto delay or cancellation for a variety of reasons. The ERG reports highlighted the strengthsand weaknesses of MSs to the STA process. Thematic analysis of these data offereda means of clarifying and describing these aspects of the submissions. This analysisgenerated five themes: process, reporting, satisfaction of objectives, reliability and validityof findings, and content. Points from clarification letters were analysed and presented infour main categories: report quality, systematic review methods, clinical data analysis and economic data analysis.Limitations: Nearly all data were obtained from the NICE website; therefore, any errors inthe data on the website will be reflected in the mapping analysis presented in this report.Missing data for the mapping exercise limit the generalisability of the findings. Analyseswere limited to what was reported in the ERG reports and the clarification letters.Conclusions: Guidance suggested for manufacturers will help to ensure that moreappropriate submissions are received in the future while recommendations provided forERG teams will help to guide teams to ensure that reporting methods are transparent.Funding: The National Institute for Health Research Health Technology Assessment programme. © Queen's Printer and Controller of HMSO 2011.