Article Text
PDF
Abstract
Objectives Innovations resulting from research have both national and global impact, so selecting the most promising research studies to fund is crucial. Peer review of research funding applications is part of the selection process, and requires considerable resources. This study aimed to elicit stakeholder opinions about which factors contribute to and influence effective peer review of funding applications to the UK National Institute for Health Research (NIHR), and to identify possible minor improvements to current processes and any major changes or potential innovations to achieve a more efficient peer review process.
Design Qualitative interviews with 30 stakeholders involved in the peer review process.
Participants Participants were drawn from three NIHR coordinating centres and represented four types of stakeholders: board members with responsibility for making funding decisions, applicants, external peer reviewers and NIHR staff.
Methods All interviews were conducted by telephone apart from three that were face to face with NIHR staff. Data were analysed using a thematic template method.
Results The responses from NIHR staff, board members and reviewers differed from those received from applicants. The first three groups focused on how well the process of peer review did or did not function. The applicants mentioned these points but in addition often reflected on how their personal application was assessed. Process improvements suggested included: developing a more proportionate review process; providing greater guidance, feedback, training, acknowledgement or incentives for peer reviewers; reducing the time commitment and amount of paperwork; and asking reviewers to comment on the importance, strengths and weaknesses of applications and flaws which are potentially ‘fixable’.
Conclusions Overall, participants were supportive of the need for peer review in evaluating applications for research funding. This study revealed which parts of the process are working well and are valued, and barriers, difficulties and potential areas for improvement and development.
- peer review
- research funding
- process evaluation
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
A strength of this study was our use of indepth interviews and thematic analysis to reveal the views of a range of stakeholder groups regarding the peer review process, and on elements which could potentially be changed or developed.
This indepth investigation of the peer review process for funding applications elucidated ways in which resource may be saved.
A weakness of the study was the fact that we only recruited participants within one—although large and well established—funding body, so that our findings may not be generalisable.
Background
The National Institute for Health Research (NIHR)1 is the UK’s largest health research funder and one of the largest in the world. It is a long established national funder of applied health research, covering the journey from ‘bench to bedside’. However, we know that on average it takes 17 years from a promising development in a laboratory to research findings which translate into better treatments and services for patients, with 10 years within the NIHR research pathway.2 The NIHR is committed to improving the efficiency of this process, the overall aim being to simplify the research pathway and reduce the time taken to make that journey. This study is part of a wider programme of work undertaken by the NIHR Push the Pace project.2 This wider work aims to simplify the research pathway, the time taken to complete that journey and improve transparency, so making a real difference to patients’ lives.
Innovations resulting from health research have both national and global impact, so selecting the most promising studies to fund is crucial. Peer review is widely used as a method for evaluating grant applications in health research3 and there is much in the literature about the relative merits and limitations of different peer review systems and processes.4–8 Applications for funding are often evaluated by internal and external reviewers,9 the internal reviewers commonly being members of an independent decision making board or committee, and the external peer reviewers being external to that committee and funding organisation, with comments fed in to aid discussion and the decision making process by the committee. This is exactly the process used across the NIHR.
Currently, most applications go through a two stage process, where at the first stage the outline applications submitted are discussed by a board. Those applicants successful at this first stage are invited to submit a full application. The full applications are sent out for external peer review prior to being discussed for a second time by a board where funding decisions are made. External peer reviewers cover a range of expertise, and non-UK based reviewers may be consulted, depending on the topic being considered. Applicants are permitted to suggest individuals who may be suitable peer reviewers. The same application form is used irrespective of the amount of funding sought.
The process of external peer review requires considerable time and resource from a variety of stakeholders4 10: the applicants preparing the proposal,11 12 the peer reviewers reviewing the proposal,13 the staff at the funding institution administering the process14 15 and members of the funding board. External peer review of funding applications is an integral part of the NIHR research funding process. If that process could be simplified or streamlined, that would reduce the amount of resource required and contribute to time savings in the overall research pathway.
The aim of this study was to elicit stakeholder views regarding their experiences of the NIHR peer review process for evaluating funding applications; their views regarding which parts of the process are working well and are valued; and those aspects which are not so valued or viewed as barriers or difficulties. We also sought views on potential changes or innovations in peer review to help inform decisions about what an improved or alternative NIHR peer review system might look like. This study was part of a wider piece of work and has been submitted together with another article, ‘The influence of external peer reviewer scores for funding applications on funding board decisions: a retrospective analysis of 1561 reviews’.16
Methods
The NIHR1 is a large organisation comprising several managing centres. A total of 30 participants was sought from four different stakeholder groups and from three NIHR coordinating centres in order to gain a varied sample that was in line with, and indicative of, those engaged in the peer review process. The three NIHR coordinating centres involved in this study were: the NIHR Evaluation, Trials and Studies Coordinating Centre (NETSCC), the Central Commissioning Facility (CCF) and the Trainees’ Coordinating Centre (TCC). They manage the following research funding programmes with remits defined by the subject matter:
NETSCC: efficacy and mechanism evaluation (EME), health services and delivery research (HS&DR), health technology assessment (HTA), public health research (PHR) and systematic reviews (SR).
CCF: invention for innovation (i4i), programme grants for applied research (PGfAR) and research for patient benefit (RfPB).
TCC: personal training awards and fellowships.
The TCC makes awards to individual health professionals or researchers. For this project, applications for individuals for awards at postdoctoral level and above were included.
The targeted groups of stakeholders were: board/panel members with responsibility for making funding decisions; applicants for NIHR funding (both successful and unsuccessful); external peer reviewers (including public contributors); and NIHR staff responsible for managing the NIHR peer review process. The role of NIHR staff members is different to the other three groups, as they do not review or make decisions, but act in a facilitative and supportive capacity to the peer review process.
Participants were selected by purposive sampling according to their primary role. For board members, reviewers and applicants, participants were selected across a range of area of expertise (including patient and public involvement (PPI) representatives) and across the range of funding programmes included in this study, with a balance in the gender of those selected. Criteria for selection for board members included current board membership with experience of at least three board meetings. For applicants, applications had been made within the previous 2 years; both those successful and unsuccessful in obtaining funding were selected. For reviewers, those who had submitted a review, or had agreed to review but had not delivered the review within the previous 2 years, were selected. Members of staff across three levels of seniority from the three coordinating centres were selected.
Interview schedules for each stakeholder group were developed by the research team and consisted of a list of prompts which were used to guide the interviews. Participants were assured that their responses would remain anonymous apart from questions about the background of the interviewee, their role in the reviewing process and the programme or programmes they related to. Three following main topics were included in the interview schedule.
What were the positive aspects and strengths of the peer review process?
What was not working so well—that is, challenges, barriers or difficulties?
What could be done to improve the process, or what might an alternative process look like?
Participants were also given the opportunity to offer their perspectives on any other matters they deemed relevant to the review of applications for funding. Participants were specifically asked ‘If you were able to build a proportionate peer review system from scratch, what would be its key features?’ A conversational style was adopted in the interviews to elicit the perceptions of participants, encourage them to share their opinions candidly and add to the richness of the interview content.
All interviews were conducted by telephone (using a speaker phone) apart from three that were face to face (with NIHR staff members). The same pair of researchers interviewed all participants for each stakeholder group; one interviewer asked the questions and the other took notes. This enabled discussion to take place afterwards between the interviewer and note taker, facilitating accurate interpretation of the data and reducing the risk of individual bias. Interviews were audio recorded and voice files were saved in a password protected folder on a university secure server.
The thematic template analytical method17 was used for initial data analysis. Templates were populated with data, based on a priori themes (generated from the interview questions), supplemented by additional points raised by participants. The final stage of the process was to bring together the data within and then across participant groups, by tabulating responses from all groups on a given theme.
Advice about the need for ethics approval for this study was sought from the University of Southampton’s Ethics and Research Governance Online (ERGO) service, and it was agreed that approval was not required for this study as it was a process evaluation, no patients were contacted as part of this study and no NHS premises were used to identify our study participants.
Patient and public involvement statement
No patients were involved in this study. The views of public contributors (both external peer reviewers and board members) were sought as part of this study.
Results
General views on the peer review process
A range of participants across the groups were included. The breakdown of the sample is shown in table 1. (Some committees within the NIHR with responsibility for making funding decisions are termed ‘boards’, others are ‘panels’, and the terms were used interchangeably by interviewees, but for clarity, this article uses the term ‘board’ to include both boards and panels.) There was some overlap in roles in relation to the first three of these groups in that board members could also have been applicants at some time, and they all had previous experience as peer reviewers for the NIHR and/or other funders. Similarly, applicants were also likely to have had experience as peer reviewers, and peer reviewers may have also been applicants at some time. Example quotations from the interviews are used to illustrate key themes. After each quotation, bracketed information describes which type of participant has been quoted (ie, board/panel members with responsibility for making funding decisions (BM); applicants for NIHR funding (both successful and unsuccessful) (A); external peer reviewers (including public contributors) (R); and NIHR staff responsible for managing the NIHR peer review process (S)).
Participant groups: roles, specialisms, gender and coordinating centre
Participants generally agreed that the process of peer review of funding applications was necessary, effective and, in the main, of good quality and fair. Applicants felt that peer review was a vital part of the process of seeking funding, suggesting that:
The peer review process should be retained as it is a really important part of ensuring that high quality research is funded.(A5)
Board members felt that:
Peer reviews (are) absolutely essential. (BM5)
Nice to know what the feeling is … at the coalface…of people actually involved in that particular type of disease or intervention. (BM8)
Staff, board members and reviewers were keen to focus on how well the process did or did not function, whether it was fit for purpose and whether it achieved what it set out to do. Applicants mentioned these points but in addition often reflected on their experience of being assessed.
Themes about positive aspects and strengths of the process
Expertise beyond that of the board membership
Several of the board members said that the peer reviews were helpful in covering areas of expertise not covered by the board membership. This was particularly useful for clinical issues. They added that the quality of peer reviews was good, they addressed the right issues and there was a range of expertise.
It is hugely valuable to have experts in the fields. (BM4)
The panel is very diverse with all sorts of professions… but it is really quite difficult for some members of the panel to understand the context. . so (when you get) the right sort of reviewers (that) is really helpful. (BM7)
NIHR staff participants also pointed out that peer review enhanced the assessment made by board members.
It enables you to get the detailed expert review that you would need for a piece of high quality research which you can’t always get from what makes up the panel of 12–15 individuals. That’s the key one really. (S1)
Help with making funding decisions
Reviewers, board members and NIHR staff saw the peer review process as a vital way of judging whether an application should be funded.
It is hugely valuable to have experts in the fields… Comments from peer reviewers that have sometimes changed the decision of the committee. (BM4)
It’s really helped them [BMs] to make decisions, especially if a peer reviewer has brought up a question or comment that none of the panel thought of, and then this has led to a bigger discussion in this area and it has sometimes swayed the panel[s] previous thoughts on the application. So I think that these peer reviews really help the panel make a decision on the application, or at least help to inform both sides.(S6)
It’s definitely useful; obviously you can judge the application on its merits but you need to know … about how relevant and applicable it will be to the wider health service. (BM8)
Generally helpful in the sense they give a strong steer to the panel. (BM1)
Scheduling and advance warning for reviewers
The current NIHR timeframe for reviewers received mixed comments, but actions such as giving prior warning to reviewers or ‘priming’ them, or providing some flexibility over the time allowed was viewed positively.
The NIHR timeframe in which we have to do a review is not (usually) a problem; timing is (mainly) good. (R4)
We find it’s really helpful to give the peer reviewers that advanced warning of when we require their review by and when it will be available to them—they can plan ahead. They are also able to tell us ‘no’ in advance if they have that time booked up for other work. They find it really helpful. (S6)
Training, development and motivation
Several reviewers commented on why they were prepared to take part in the process, citing ‘training and development reasons’, or through a ‘sense of stewardship and wanting to give back’. Many of the reviewers had been or were still applicants themselves, and they appreciated the relevant experience of being, at different times, both an applicant and a reviewer. In addition, reviewers commented that by reviewing, they learnt what makes a good application. There was a general feeling among the staff that people were motivated to review for the NIHR because they were interested in the topic area and in what new research was being proposed.
Challenges, barriers and difficulties
Volume of paperwork and time commitment
Board members referred to the large volume of paperwork they were required to deal with. Some also felt it might be off-putting for people considering becoming a board member.
It is difficult trying to balance all of this information … and board members can be swamped with overwhelming amounts of information. (BM5)
The idea of expecting panel members to have read every single application indepth is not possible. You couldn’t retain it for a start. When the paperwork arrives my heart sinks. (BM7)
Time commitment was raised by many of the interviewees. This was linked by staff members to the difficulty on occasions of gaining reviewers.
The biggest challenge… is the time and effort required to get the number of reviewers needed. Usually people you are approaching have a lot on their plates. They’ll be busy reviewers themselves and quite often they are being asked to do this by other funders or parts of NIHR as well. (S1)
The issue of time was related to motivation: while staff members did not express a lack of preparedness on the part of reviewers to take on the task, they were aware of the pressures on busy clinicians and researchers, with the request to review sometimes very substantial applications being a ‘step too far’ or not possible at certain times.
They just have not got the capacity to spend time doing peer reviews for academic research—it doesn’t factor into their day to day timetable. (S2)
Selection and numbers of reviewers
The need to select reviewers with expertise in particular topics, disciplines and methodologies was raised by many interviewees.
Need to be careful about the reviewers that are chosen. (BM7)
Ask right reviewers to review the right topic. (BM8)
Applicants considered that having a range of disciplines involved was important, but cautioned that this only worked if there was appropriate matching between the reviewers and the submission.
You get the impression that they have got good people doing the reviews but it’s vital to have the right ones. (A7)
Board members also commented on the numbers of reviewers per application.
2–3 not enough, 5–6 good number. (BM3)
Not enough if only 1–2, often 4–5 feels this about right with one being PPI (a patient or member of the public). (BM4)
Does become less helpful when there are lots (of reviews)—becomes too much information to process. (BM5)
Improvements, changes and innovations
None of the interviewees suggested that the current peer review process should be dispensed with or replaced by an entirely new process. When participants were asked where improvements to the current process could be made, however, several key points emerged.
Volume of paperwork and time commitment
Participants suggested this could be helped or exacerbated by the structure of the review and/or application forms, and by reliance on lead assessors (a designated member of the board with the most relevant expertise related to that application). Reviewers expressed a strong desire to receive the whole application form, but wished to only answer questions specific to their area(s) of expertise.
It’s important to see the whole application but (just to be expected) to review sections. (R1)
It would be less daunting (to strip back the form) but … the downside is that if you haven’t got the whole grant… there may be some context missing. (R5)
I do wonder about … asking reviewers to focus on particular areas, I think that would be quite helpful. If I knew what NIHR were looking to me for, I can … focus on the bit you’ve asked me to look at. That would be helpful as it would be less daunting in (terms of) the potential workload. (R5)
Keep review brief and to the point, brevity better. (BM1)
Ideally we need to somehow shorten what we are asking people to submit so that it is more manageable for people who are reviewing it. (BM5)
Make it easy, only relevant information in a way that is easy to access. (BM8)
Selection and number of peer reviewers
The issue of proportionality of the resource used for the peer review process as related particular applications was discussed positively and negatively. In funding programmes where the number of reviewers called on was related to the monetary value of the application, this was considered to be an appropriate feature. The actual number of reviewers was also commented on, in order to identify the balance of any positive and negative views (please see quotes above).
Ensure that peer review is proportionate—that their (the reviewers) expertise are proportionate and that the peer reviewers clearly do have expertise technically.(A8)
The lack of proportionality was raised by some who were working in funding programmes where the number of reviewers was not related to the cost and size of the application.
Reducing the number of questions that reviewers are asked to comment on
Participants suggested focusing peer reviewers on fewer questions and ‘fixable flaws’, developing a model where initially three key questions are asked: (1) Is the study important and in need of doing; (2) are there major methodological flaws; and (3) what are the strengths and weaknesses of the study? If the answer to question (1) is yes, and the flaws and weaknesses can be remedied, it should be allowed to proceed to the next stage with feedback.
We have to decide whether an issue is a ‘deal maker’ or ‘deal breaker’. ‘Fatal flaws’ that can’t be rectified are deal breakers. (BM1)
Peer reviews are rarely deal breakers’, they certainly help us to crystallise what the key outstanding issues would be and whether these are fixable or fundamental faults. (BM4)
Providing more feedback, guidance and training to reviewers
Interviewees suggested providing more feedback, guidance and training to peer reviewers.
I never had any training on how to peer review grants. Maybe if you offered a day where people came … and learnt how to be good peer reviewers then they might be more confident to say yes when they’re asked … I’d like to hear from a chair on an NIHR panel: what makes a really useful review. (R5)
I think it would be good if we were asked to rate the quality of the feedback from the reviewer. (A6)
Targeting of reviewers
Interviewees suggested targeting reviewers to gain well informed opinion and a balanced view of the application.
Fewer more highly selected reviewers (per application)—that is, people with selected experience so they have a good grasp of what is important and what is not. (A5)
Motivation of reviewers and acknowledgement of their work
The interviewees made suggestions about providing incentives to encourage reviewers and rewards for undertaking the review. These suggestions included: acknowledgement of the contribution of reviewers beyond an email from the panel/board; incentives for people to review such as continuing professional development points or payment (although payment was not universally supported or endorsed); emphasising the professional kudos; and professional and moral responsibility as part of a research community.
Attach some system of merit for being a reviewer; this would encourage reviewers with the best level of scrutiny. (A1)
with some sort of incentive so that people are delighted to get a request. (BM4)
Names on website might be helpful (BM4) but A certificate is the kind of thing that is needed, rather than a website. (BM8)
Discussion
A statement of the principal findings
Our findings show that peer review of applications is valued by applicants and board members. Minor adjustments to the process were suggested, but no one suggested that peer review should be dispensed with or completely changed. A strongly emerging theme from this work was that of the extent of what was required, the amount of work peer review currently entails and the difficulties of running such processes while relying on the goodwill and commitment of already busy professionals, academics and clinicians. Changes were suggested to make the experience less onerous for those involved while still maintaining the quality of the review to support board decision making and providing feedback to applicants. A reduction in the amount of paperwork and time committed would be welcomed.
Another strong emerging theme from this work was the need to concentrate on the importance of the proposed study, strengths and weaknesses of the application, ‘fixable flaws’ and tailoring the questions reviewers are asked to comment on depending on their expertise. Other strong themes included providing training, feedback and greater guidance to reviewers. The NIHR currently offers some training and guidance to peer reviewers, and a more comprehensive training scheme is in development. When asked to suggest more major changes, the participants suggested developing a more proportionate system across all funding programmes by varying the numbers of reviewers according to the value of the application (along the lines of practice in one NIHR centre). However, this was not a straightforward issue. Proportionality could be interpreted as relating the number of reviews to the cost of the application, or to the scale of the review task requested or the amount of effort for staff in managing the reviews. There are reasons why a minimum number of reviewers may need to be associated with any one bid because of the need for the right mix of expertise.
Strengths and weaknesses of the study
Care was taken to target a range of relevant internal and external stakeholders via the three NIHR centres and those familiar with all of the main programmes and funding streams managed by these centres. The interviews, their recording and analysis were judged (by the independent research consultant) to be undertaken systematically and meticulously, and there were always at least two researchers involved with each interview, which helped validate our interpretation of participant comments.
A weakness of the study was the fact that we only recruited participants within one—although large and well established—funding body, so that our findings may not be generalisable. Care should also be taken in interpreting verbal comments as recall bias, social response bias and post hoc rationalisation may be present. However, these issues are common to all studies relying on participant recall, and our participants’ level of experience was such that they were likely to be expressing their own views rather than matching the perceived expectations of the interviewers. Additionally, participants were reassured that we wanted their true opinions, and using the telephone provided an environment more distant than face to face, which may also have made it easier for participants to express any negative views.
Strengths and weaknesses in relation to other studies, discussing important differences in results
This study was part of an indepth examination by the NIHR, a large national funder, of the research pathway with the overarching aim of shortening the time from ‘bench to bedside’.2 The process of peer review is part of this pathway; modifications to the workload for stakeholders, or the numbers of reviews sought,16 and other innovations have the potential to shorten that pathway for the benefit of patients. There are few other similar accounts in the literature.18 19
There has, however, been considerable debate in the literature about the value and reliability of peer review in selecting applications for research funding.9 20–22 The interviewees in this study, including applicants, were in general supportive of the need for the peer review process in evaluating applications to NIHR for research funding. The process was valued and it was felt that current processes were working well. Increasing the motivation to carry out peer review could be a key factor. If the task was viewed as more prestigious and of value to career development, and appropriate incentives and acknowledgement were offered, more people would want to review, considerably less effort would be required from funding organisations to obtain suitable reviewers and reviewers would benefit from undertaking the task.
The meaning of the study: possible explanations and implications for clinicians and policy makers
The amount of work involved in peer review has been well documented previously.4 13 21 Some potential innovations or modifications to the peer review process could make the process quicker, less arduous for those involved and address some of the current challenges, barriers and difficulties. There are studies in the literature investigating how the process of peer review of applications for funding may be made less arduous or quicker4 21 23 and facilitated by the use of shorter application forms.7 8 14 Some changes to the current NIHR process, including modifications to the application form, were suggested by interviewees in our study. The structure of the standard NIHR application form has recently been reviewed, and a revised NIHR standard application form is being implemented in late 2017/early 2018. Understanding the impact it has on stakeholder groups may shed light on this area in the future.
Unanswered questions and future research
Pilot studies to evaluate alternative approaches to peer review identified by this study, and their impact, would be informative areas for future research. Areas of interest might include training and feedback for reviewers, and alternative proportionate approaches to the peer review process.
Conclusions
This study was helpful in revealing which parts of the NIHR peer review process are perceived to be working well and are valued; participants were supportive of the need for peer review in evaluating applications for research funding. We also identified barriers, difficulties, potential areas for improvement and development, and possible changes to current process which could be tested and evaluated in the future.
Acknowledgments
We would like to acknowledge Judith Lathlean for her invaluable help and guidance with the methodology and implementation of the study. We would also like to acknowledge the kind contribution of all participants who took part in the study and Wendy Gaisford who helped with the organisation of the interviews.
Footnotes
Contributors The study was conceived and designed by ST, JCW, FC, RM, AB, EWG, LW and JH; undertaken by ST, FC, RM, LW, NCM, AB, EWG, HP and JH; ST led the writing guided by JCW. All authors read and approved the final manuscript.
Funding This research was supported by the NIHR Evaluation, Trials and Studies Coordinating Centre (NETSCC) through its Research on Research Programme. The views and opinions expressed are those of the authors and do not necessarily reflect those of the Department of Health, or of NETSCC.
Competing interests All of the authors were employed by the University of Southampton to work at least part time for the NIHR Evaluation, Trials and Studies Coordinating Centre (NETSCC) at the time when the study was conducted.
Patient consent Not required.
Ethics approval Advice about the need for ethics approval for this study was sought from the University of Southampton’s Ethics and Research Governance Online (ERGO) service, and it was agreed that approval was not required for this study as it was a process evaluation.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement Due to the qualitative nature of the data, and as consent was not sought from participants for sharing data, we would not be able to share.