NHS Diabetes Prevention Programme in England: formative evaluation of the programme in early phase implementation

Objectives Evaluation of the demonstrator phase and first wave roll-out of the National Health Service (NHS) Diabetes Prevention Programme (DPP) in England. To examine: (1) intervention design, provision and fidelity assessment procedures; (2) risk assessment and recruitment pathways and (3) data collection for monitoring and evaluation. To provide recommendations informing decision makers on programme quality, improvements and future evaluation. Design We reviewed programme documents, mapping against the NHS DPP specification and National Institute for Health and Care Excellence (NICE) public health guideline: Type 2 diabetes (T2D) prevention in people at high risk (PH38), conducted qualitative research using individual interviews and focus group discussions with stakeholders and examined recruitment, fidelity and data collection procedures. Setting Seven NHS DPP demonstrator sites and, subsequently, 27 first wave areas across England. Interventions Intensive behavioural intervention with weight loss, diet and physical activity goals. The national programme specifies at least 13 sessions over 9 months, delivered face to face to groups of 15–20 adults with non-diabetic hyperglycaemia, mainly recruited from primary care and NHS Health Checks. Participants Participants for qualitative research were purposively sampled to provide a spread of stakeholder experience. Documents for review were provided via the NHS DPP Management Group. Findings The NHS DPP specification reflected current evidence with a clear framework for service provision. Providers, with national capacity to deliver, supplied intervention plans compliant with this framework. Stakeholders highlighted limitations in fidelity assessment and recruitment and retention challenges, especially in reach and equity, that could adversely impact on implementation. Risk assessment for first wave eligibility differed from NICE guidance. Conclusions The NHS DPP provides an evidence-based behavioural intervention for prevention of T2D in adults at high risk, with capacity to deliver nationally. Framework specification allows for balance between consistency and contextual variation in intervention delivery, with session details devolved to providers. Limitations in fidelity assurance, data collection procedures and recruitment issues could adversely impact on intervention effectiveness and restrict evaluation.

Specially, when it is pretty likely to be variable among sites and contexts, raising issues related to fidelity-flexibility, not only on how the clinical intervention has been organized and carried out, but on the implementation strategies used.
Within the different types of program evaluation (process, outcome, impact, etc.), the study actually describes factors (organized by subject) that have facilitated / prevented the 3 aspects of evaluation related to clinical intervention based on review of documentation and comparison to recommended program, and key agent opinions and perceptions extracted from interviews. As a consequence, the true contribution of the study, taking into account its strengths and weaknesses, should be more clearly specified in both the title and the body of the manuscript.
The comparison of how the programs were organized / implemented with respect to the recommendations is limited, describing in a table the operational organization of the program according to 5 sites and explaining some discrepancies in the use of BCTs and fidelity aspects. An exhaustive or at least better organized description of how the clinical intervention program has been organized / implemented at each site, and its deviation from what is recommended, would help, especially those different ways of putting it into practice. But also, it would be very interesting to know which of these procedures are successful or not, based on not only perceptions of the agents but on numbers (quantitatively). The NICE baseline measurement tool can be a possible guide on how to structure a table for that description.
Though it is a "qualitative" evaluation, and not a formal program process evaluation, further quantitative information will help to understand the magnitude of the study (ie., reach of the different screening procedures, intervention delivery procedures…by each site). This information will also help to describe which procedures and strategies are more successful.
An implementation framework or model that guides the design or at least the analysis, synthesis, and interpretation of these determinants of implementation (ie, CFIR framework) is missed. This would help not only to understand which determinants are key and why, but also to identify and / or design strategies to address such determinants in future program roll-out. Linked to this previous point, and within the parts describing the framework and methods that have guided the evaluation, the authors refer to the mapping of the retrieved information from providers against the NICE recommendations, equality indicators, fidelity, data collection and quality assurance procedures. But none of these matched or mapped elements are clearly displayed or presented within the manuscript. They also refer to the development of some tools (page 9, last paragraph) for use in intervention feasibility, acceptability and fidelity assessment. An example of these tools, due to the help that may offer to the reading public, is highly recommendable.
Minor points Abstract-findings: more precise or concrete results summarizing actual obtained results A Figure Flow-diagram is highly recommendable to fully describe the structure and process (phases, sites, centers, professionals, patients referred, etc) of the study.

GENERAL COMMENTS
Page 8, paragraph 1, Introduction section: what was the goal of this study? This should be clearly stated clearly. This need not to be a hypothesis to prove or disprove, but if it is a study to ensure the quality of implementation, the goals of the program's implementation should and should be stated clearly. As currently written, the introduction section seems to simply say that the program was mandated and that various aspects of the program were examined.
Page 8, paragraph 3, methods section: A preposition seems absent. sentence should read "we agreed upon research governance and reporting strategies".
Page 8, paragraph four, "methods to address objectives": The first sentence describes "systematically mapping information supplied by providers". It's not clear to me what this means, and I suspect that the investigators used written materials as the input.
Page 13, paragraph one: I don't think the "details of themes" is detailed enough. For example, what does "social factors, such as group support" mean? There is one example of a quote in the theme of group support in Box 2. Should the reader interpret this as meaning most or all of the comments on this topic were positive?
Page 22, paragraph three and page 23, paragraph 1: these two paragraphs are informative but don't seem to be necessary to the discussion section. I would prefer that the discussion section instead interprets the findings and talks more about how these findings will affect for their implementation efforts.

VERSION 1 -AUTHOR RESPONSE
Reviewer: 1 Reviewer Name: Alvaro Sánchez Pérez Institution and Country: Primary Care Research Unit of Bizkaia-Osakidetza, SPAIN Competing Interests: None declared This is a very relevant manuscript for those involved in the scale-up of proven clinical interventions. We thank Alvaro Sánchez Pérez for this appreciation of our work Specifically, the objective of the study is to evaluate the demonstrator phase and first wave roll-out of the NHS diabetes prevention program (DPP) in England, to inform subsequent phase. To do so, they have examined: (i) intervention design and provision, in relation to the evidence base, and procedures to assess intervention fidelity; (ii) risk assessment procedures and recruitment pathways and (iii) data collection, monitoring and evaluation.
The evaluation methods included structured document review, comparing the program with the recommendations in NICE PH38, and qualitative research involving individual interviews and interactive focus group discussions with key stakeholders, including service users Due to the relevance at public health level of the general NHS Diabetes Prevention Program in England and the intended goals of the present study (informing the rule-out of future phases in a more efficient way, for both implementation and evaluation), the present manuscript has several core elements in which it might be improved.
Major points First, it is not clear what and how within the MRC framework the present evaluation fits. What is the specific contribution or the specific question that the study is trying to answer within such framework? Please comment on this, especially, when the study seems to be the Phase IV translation stage.
Response: We thank the reviewer for this suggestion, which has helped to clarify an important goal of our study.
We have clarified the specific contribution in terms of providing recommendations based on findings to inform decision makers (in both the abstract and the methods). This is in line with the implementation phase requirements for evaluations, as described in the MRC 2008 framework for development and evaluation of complex interventions to improve health.
These changes are tracked in the document and copied below with changes highlighted for convenience.
Evaluation of the demonstrator phase and first wave roll-out of the NHS diabetes prevention programme (DPP) in England, to inform subsequent phases. To examine: (i) intervention design, provision and fidelity assessment procedures; (ii) risk assessment and recruitment pathways and (iii) data collection for monitoring and evaluation and to provide recommendations based on findings to inform decision makers on programme quality, improvements and future evaluation In planning the evaluation we drew on the Medical Research Council (MRC) guidance for development and evaluation of complex interventions to improve health12 and the MRC guidance for process evaluation of complex interventions to improve health.13 Comment: Second, and linked to the previous point, the present evaluation only considers the clinical intervention level, not informing about the most important aspect in my opinion when addressing the transfer/dissemination/scale-up of clinical interventions: the implementation strategy used to enhance the adoption and execution of the clinical intervention in the studied sites/contexts. Specially, when it is pretty likely to be variable among sites and contexts, raising issues related to fidelity-flexibility, not only on how the clinical intervention has been organized and carried out, but on the implementation strategies used.
Within the different types of program evaluation (process, outcome, impact, etc.), the study actually describes factors (organized by subject) that have facilitated / prevented the 3 aspects of evaluation related to clinical intervention based on review of documentation and comparison to recommended program, and key agent opinions and perceptions extracted from interviews. As a consequence, the true contribution of the study, taking into account its strengths and weaknesses, should be more clearly specified in both the title and the body of the manuscript.
Response: We thank the reviewer for drawing our attention to this need for clarification of the scope of our evaluation. We have suggested an addition to the title that might clarify. However, if the reviewer thinks that this still doesn't quite hit the spot, and they are prepared to offer a better suggestion we would be appreciative.
Our evaluation was conducted at an early stage in the roll out of the NHS DPP. The scope of our evaluation was to focus on the nationally commissioned programme and implementation strategies, and support for its future evaluation, rather than on details of site-specific organisational strategies to facilitate uptake of the programme in different localities. We made important, specific recommendations to the NHS DPP management group that were based on our findings, (such as to improve recruitment data collection -please see highlight in the manuscript). These recommendations, if actioned by the management, would support the assessment of implementation strategy in future evaluations. We also reported on the early views of local commissioners in relation to their experiences of implementation, but we were aware that these views were recorded at an early stage in the process.
We have added to the discussion of strengths and limitations of our study, (p22) as suggested, to clarify the focus of this early phase study and the need to further consider local implementation strategies as the programme is embedded and rolled out across the country.
The comparison of how the programs were organized / implemented with respect to the recommendations is limited, describing in a table the operational organization of the program according to 5 sites and explaining some discrepancies in the use of BCTs and fidelity aspects. An exhaustive or at least better organized description of how the clinical intervention program has been organized / implemented at each site, and its deviation from what is recommended, would help, especially those different ways of putting it into practice. But also, it would be very interesting to know which of these procedures are successful or not, based on not only perceptions of the agents but on numbers (quantitatively). The NICE baseline measurement tool can be a possible guide on how to structure a table for that description.
We have provided a table comparing intervention components across first wave provider organisations as appendix A (supplementary file). Detailed evaluation of the ways in which first wave sites operationalise the programme was beyond the scope of this early phase evaluation, which was designed to inform the national programme strategy and future implementation and evaluations. In our reports we made recommendations on the need for further evaluation of site specific operational variation, with specific details of data requirements (eg. the collection of data to provide detail on routes to recruitment) to facilitate such evaluations.
We agree that quantitative data would be of interest. However, these data were not available to us for this early stage evaluation. Part of our remit was to make recommendations on data collection that would support future evaluation. We made data collection recommendations, but action on these recommendations was a management decision.
Comment: Though it is a "qualitative" evaluation, and not a formal program process evaluation, further quantitative information will help to understand the magnitude of the study (ie., reach of the different screening procedures, intervention delivery procedures…by each site). This information will also help to describe which procedures and strategies are more successful.
Response: As above the lack of quantitative data at this early stage (other than the collated management report data, which is included in our paper) is regrettable. We hope that the data collection requests that we made will ensure these data are suitable and available for further evaluation at later times. We have added to the strengths and limitations to position the recommendations within the MRC framework. We have also explained the value of eliciting formal response to our recommendations, from the management group, to facilitate impact evaluation, which would be of implementation interest. (page 22) Comment: An implementation framework or model that guides the design or at least the analysis, synthesis, and interpretation of these determinants of implementation (ie, CFIR framework) is missed. This would help not only to understand which determinants are key and why, but also to identify and / or design strategies to address such determinants in future program roll-out.
Response: We thank the reviewer for suggesting the CFIR framework. We have not used this as part of our evaluation as the focus at this early stage was on evaluation of the national programme and roll-out procedures rather than local operational strategies for the implementation of this programme. The local site-specific implementation of a nationally commissioned and funded programme, which is delivered by four dedicated provider organisations is an interesting implementation model. We do have a rich set of qualitative data, but there is a concern that these data are collected very early in the roll-out.
Comment: Linked to this previous point, and within the parts describing the framework and methods that have guided the evaluation, the authors refer to the mapping of the retrieved information from providers against the NICE recommendations, equality indicators, fidelity, data collection and quality assurance procedures. But none of these matched or mapped elements are clearly displayed or presented within the manuscript.
Response: The mapping for first wave programmes used the recommendations in NICE guidance and PROGRESS indicators. The NHS DPP national service specification and the baseline provider documents that the provider organisations submitted as part of their procurement process were mapped against these. These provider organisations are commercial enterprises and as such these detailed documents were made available to the evaluation team with confidentiality understanding in place. We are able to publish summaries of the findings from this mapping procedure. We are not able to make these mapping tables open access at this time, due commercial interests and confidentiality arrangements, which were agreed with the management group. However, if a research team were to request access to these data (tables) for specific research use we would enquire if these might be made available for defined research purposes.
Comment: They also refer to the development of some tools (page 9, last paragraph) for use in intervention feasibility, acceptability and fidelity assessment. An example of these tools, due to the help that may offer to the reading public, is highly recommendable.
Response: At present these tools are in a draft stage and as such they are untested (other than through limited PPI processes). We hope to be able to assess the utility of the tools and then, if they can be tested and found to be fit for purpose, we would be pleased to then make them available more generally. In the meantime we would be pleased to hear from others who are involved with interventions for prevention of type 2 diabetes in people at high risk and who would like to work with us to assess the utility of these tools.
Minor points Abstract-findings: more precise or concrete results summarizing actual obtained results We have added to the abstract to clarify as suggested.
Comment: A Figure Flow-diagram is highly recommendable to fully describe the structure and process (phases, sites, centers, professionals, patients referred, etc) of the study.