You are here

Article Text

Article menu

Download PDFPDF

XML
Evidence based practice
Research
The clinical relevance and newsworthiness of NIHR HTA-funded research: a cohort study
  1. D Wright1,
  2. A Young1,
  3. E Iserman2,
  4. R Maeso1,
  5. S Turner1,
  6. R B Haynes2,
  7. R Milne3
  1. 1National Institute for Health Research, Evaluation, Trials and Studies Coordinating Centre (NETSCC), University of Southampton, Southampton, Hampshire, UK
  2. 2Department of Clinical Epidemiology & Biostatistics, McMaster University, Hamilton, Ontario, Canada
  3. 3Wessex Institute, University of Southampton, Southampton, Hampshire, UK
  1. Correspondence to Dr D Wright; D.Wright{at}soton.ac.uk

Abstract

Objective To assess the clinical relevance and newsworthiness of the UK National Institute for Health Research (NIHR) Health Technology Assessment (HTA) Programme funded reports.

Study design Retrospective cohort study.

Setting The cohort included 311 NIHR HTA Programme funded reports publishing in HTA in the period 1 January 2007–31 December 2012. The McMaster Online Rating of Evidence (MORE) system independently identified the clinical relevance and newsworthiness of NIHR HTA publications and non-NIHR HTA publications. The MORE system involves over 4000 physicians rating publications on a scale of relevance (the extent to which articles are relevant to practice) and a scale of newsworthiness (the extent to which articles contain news or something clinicians are unlikely to know).

Main outcome measures The proportion of reports published in HTA meeting MORE inclusion criteria and mean average relevance and newsworthiness ratings were calculated and compared with publications from the same studies publishing outside HTA and non-NIHR HTA funded publications.

Results 286/311 (92.0%) of NIHR HTA reports were assessed by MORE, of which 192 (67.1%) passed MORE criteria. The average clinical relevance rating for NIHR HTA reports was 5.48, statistically higher than the 5.32 rating for non-NIHR HTA publications (mean difference=0.16, 95% CI 0.04 to 0.29, p=0.01). Average newsworthiness ratings were similar between NIHR HTA reports and non-NIHR HTA publications (4.75 and 4.70, respectively; mean difference=0.05, 95% CI −0.18 to 0.07, p=0.402). NIHR HTA-funded original research reports were statistically higher for newsworthiness than reviews (5.05 compared with 4.64) (mean difference=0.41, 95% CI 0.18 to 0.64, p=0.001).

Conclusions Funding research of clinical relevance is important in maximising the value of research investment. The NIHR HTA Programme is successful in funding projects that generate outputs of clinical relevance.

  • Statistics & Research Methods

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 3.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/3.0/

https://doi.org/10.1136/bmjopen-2013-004556

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Strengths and limitations of this study

    • The strength of this study is that a rigorous, extensive and independent approach to assessing the clinical relevance and newsworthiness of funded research provided by MORE was applied to the National Institute for Health Research (NIHR) Health Technology Assessment (HTA) reports.

    • A limitation of this study was that the data were reliant on funded research teams self-reporting their publications. It is most likely that the number of publications is under-reported from NIHR HTA-funded studies.

    • Raters contributing to McMaster Online Rating of Evidence are voluntary and are therefore likely to be interested in evidence-based medicine. Therefore, they may not be representative of all physicians.

    Introduction

    Commissioning new research in areas of high clinical relevance is an important part of minimising research waste. Chalmers and Glasziou1 in their paper on avoidable waste in research argue that selecting unimportant research questions is a major contributor to wasted research funds. Studies have shown, for example, that only moderate correlations exist between publicly funded research and burden of disease.2 Similarly, research conducted on patient or clinician research priorities has revealed potential mismatches between clinical priorities and funded research. Tallon et al,3 for example, examined the research priorities of patients with osteoarthritis of the knee and their clinicians and reported that their priorities differed from randomised controlled trials, which were typically focused on drug evaluations.

    One proxy measure of the clinical relevance of funded research is the assessment of the relevance of the final publication. McMaster University has established a rigorous approach to assessing the clinical relevance and newsworthiness of outputs from health research.4 The McMaster approach involves selecting relevant papers from 120 high-impact journals selected on the basis of suggestions from librarians, clinicians, editors and editorial staff, Science Citations Index impact factors, a systematic examination of journal contents and an ongoing assessment of the number of articles that meet inclusion criteria of quality. Articles from selected journals are assessed by trained staff using inclusion criteria on methodological quality and whether the article has at least one clinically important outcome.4 Articles passing these criteria are entered onto the Critical Appraisal Process (CAP) system before being transferred to the McMaster Online Rating of Evidence (MORE) system, which has in excess of 4000 physician raters in over 60 physician disciplines. Articles are sent to at least four raters for each matched discipline/specialty, who rate papers on two seven-point scales (seven—highest score; table 1). The first scale of relevance assesses the extent to which the article is relevant to practice. If relevance is rated three or above, a second scale is used assessing the extent to which the article contains news or something that clinicians are unlikely to know. Articles that average three or above for a discipline are then transferred to a Premium LiteratUre Service (PLUS) database, and are then used to alert subscribers via Evidence Updates, ACP JournalWise and other current awareness services. Ratings, which are collected at the time of journal article publication, have been shown to be predictive of subsequent citation counts.5

    View this table:
    Table 1

    The McMaster Online Rating of Evidence (MORE) criteria for clinical relevance and newsworthiness

    The McMaster approach to assessing clinical relevance and newsworthiness has enabled interesting comparisons to be made between research and publication types. One study explored the relevance and newsworthiness of high-quality original articles and systematic reviews from over 110 clinical journals and the Cochrane Database of Systematic Reviews.6 The study revealed that systematic reviews had a significantly higher rating for relevance than original articles, but a significantly lower rating for newsworthiness.6 In addition, clinicians accessed reviews more frequently than original articles.6

    The National Institute for Health Research (NIHR) Health Technology Assessment (HTA) Programme was established in 1993 to commission research that is useful to clinical practice and NHS decision-makers.7 Since its inception, significant effort has been made to identify and prioritise important topics from a range of stakeholders and to assess applications on the basis of scientific merit, feasibility and value for money. In the 10 years since 2002, 98% of NIHR HTA-funded studies have completed with a final report in the HTA Journal8 and the journal has always been included in the McMaster's top 120 journals. The reports (also referred to as monographs) are peer reviewed, are available freely in the public domain and contain a full record of the study. Unlike typical peer reviewed journals, the HTA Journal has a large word guide of approximately 50 000 words and unlimited appendices, thus enabling more detail to be included than standard peer review publications. Given the emphasis on identifying and commissioning important topics and having easily accessible and usable final reports, this study aimed to assess the clinical relevance and newsworthiness of NIHR HTA reports compared with non-NIHR HTA publications assessed by the MORE process.

    Methods

    Selection of reports and publications

    Of 314 studies published as full reports in the NIHR HTA Journal in the period 1 January 2007−31 December 2012, 311 were selected for the study. Three NIHR HTA reports were excluded from the analysis as they were updates of earlier Cochrane reviews, and therefore ratings of relevance and newsworthiness of these reports were not based on the first time these reviews were made available to the public. PubMed Identification numbers were identified for each report, which were obtained by hand searching PubMed.

    Publications arising from NIHR HTA-funded studies publishing outside the HTA Journal in the period 1 January 2007−31 December 2012 were identified using research management databases maintained by the NIHR HTA Programme. Again, PubMed Identification numbers were obtained for each publication and these data were used to remove NIHR HTA Programme-funded research outputs from the database of articles from the 120 selected journals that had passed CAP and MORE criteria. A subgroup was identified for publications from studies reporting in the HTA Journal in the period 2007–2012. This enabled a direct comparison between reports in the HTA Journal and related publications from the same studies publishing outside the Journal.

    All data were assessed by a second researcher to ensure that there were no duplications and that the publication and PubMed details were accurate. Data were held in an Excel spreadsheet, which was given to McMaster. The PubMed identification numbers were used to determine which articles were reviewed for and passed the CAP process, and then to automatically retrieve relevance and newsworthiness ratings from the MORE database.

    Units of analysis, hypotheses and statistical analyses

    Three datasets were generated that formed the basis of the analysis

    1. NIHR HTA-funded studies publishing as full reports in HTA in the period 1 January 2007–31 December 2012.

    2. Publications in the period 1 January 2007–31 December 2012 from NIHR HTA-funded studies publishing outside HTA for those studies included in category A above.

    3. Publications from non-NIHR HTA-funded publications publishing in the period 1 January 2007–31 December 2012 from Journals included by McMaster and passing CAP and MORE criteria.

    Two principal hypotheses were tested

    Hypothesis 1: Reports published in HTA have a higher MORE clinical relevance and newsworthiness rating than non-NIHR HTA publications. The rationale for this hypothesis is that the NIHR HTA Programme aims to ensure that funded research responds to clinical need and has a rigorous process of identification and prioritisation to reflect clinical priorities. We therefore wanted to see if outputs from NIHR HTA funded research rated higher in terms of clinical relevance and newsworthiness.

    Hypothesis 2: Reports published in HTA have a lower mean average MORE clinical relevance and newsworthiness rating than publications from the same research published outside the HTA Journal. The rationale for this is that funded research teams are encouraged to publish outside the HTA Journal, and therefore prior publication may affect the relevance ratings of the report if study findings are already in the public domain. Previous research had reported that some journal publications did precede publication in the HTA Journal, although it was recognised that this was not always the case and that for some studies, non-HTA Journal publications occurred well after the end of the project.9

    Subgroup analysis was undertaken for hypotheses exploring differences in clinical relevance and newsworthiness ratings between original studies and review articles. Original studies are defined as ‘Any full text article in which the authors report first-hand observations’. Review articles are defined as ‘any full text article that was bannered “review, overview, or meta-analysis” in the title or in a section heading, or it was indicated in the text of the article that the intention was to review, summarise, or highlight the literature on a particular topic’.10 Previous research had identified that original research studies were more newsworthy but less clinically relevant than review articles.6 We wished to explore whether this was also the case for NIHR HTA reports.

    As discussed elsewhere,6 article ratings were treated as continuous variables and were analysed by calculating mean averages and proportions and comparing CIs of computed results. SPSS V.20 was used for all analyses; differences were compared using p values.

    Results

    Proportions of reports and publications meeting CAP and MORE criteria

    Of 311 NIHR HTA reports published in the period 2007–2012, 286 (92%) were included in the initial CAP assessment. Twenty-five reports were not included in the analysis due to a technical error, 18 of which were published in the period 20 February 2012–8 May 2012. Of the 274 HTA publications publishing outside the HTA Journal in the period 2007—2012, 118 (43.1%) were assessed for CAP. Non-inclusion of NIHR HTA-funded publications was due to the publications appearing in journals not included in the 120 journals assessed by McMaster.

    Table 2 reveals that 67.8% (n=194) of NIHR HTA reports passed initial screening criteria of methodological rigour and relevance, compared with 57.6% (n=68) of other NIHR HTA-funded publications and 10.6% (n=20 194) of non-NIHR HTA publications included on the McMaster database. A total of 67.1% (n=192) of NIHR HTA reports passed MORE inclusion (ie, scoring ≥ 3/7 for both relevance and newsworthiness for at least one discipline), compared with 57.6% (n=68) of other NIHR HTA-funded publications and 10.5% (n=19 921) of non-NIHR HTA-funded publications.

    View this table:
    Table 2

    Proportion of publications passing CAP and MORE ratings of relevance and newsworthiness

    Clinical relevance ratings in MORE

    NIHR HTA reports had an average mean relevance rating of 5.48 (95% CI 5.36 to 5.59), which was higher than the mean relevance rating for non-NIHR HTA publications passing CAP and MORE criteria, which was 5.32 (95% CI 5.31 to 5.33) (mean difference=0.16, 95% CI 0.04 to 0.29, p=0.01; table 3). NIHR HTA-funded publications publishing outside HTA had an average mean relevance rating of 5.59 (95% CI 5.42 to 5.75), which was slightly higher than the rating for NIHR HTA reports (mean difference=0.11, 95% CI −0.10 to 0.32, p=0.419).

    View this table:
    Table 3

    Mean average MORE clinical relevance and newsworthiness ratings across publication types

    The mean relevance rating of NIHR HTA original research reports was 5.54 (95% CI 5.34 to 5.74), which was higher than non—NIHR HTA original research publications, which had a mean relevance rating of 5.27 (95% CI 5.26 to 5.29) (mean difference=0.27, 95% CI 0.03 to 0.51, p=0.03; table 4). The mean relevance rating of NIHR HTA original research reports was similar to publications from the same studies publishing outside the HTA Journal, which had a mean average relevance rating of 5.57 (95% CI 5.36 to 5.78) (mean difference=0.03, 95% CI −0.32 to 0.26, p=0.875; table 4).

    View this table:
    Table 4

    Mean average clinical relevance and newsworthiness ratings for original research by publication type

    The mean relevance rating of NIHR HTA-funded review reports was 5.46 (95% CI 5.32 to 5.60), which was similar to that for non-NIHR HTA-funded publications, which had a mean relevance rating of 5.41 (95% CI 5.39 to 5.43) (mean difference=0.05, 95% CI −0.09 to 0.19, p=0.46; table 5). NIHR HTA-funded review reports had a slightly lower mean relevance rating than publications from the same studies publishing outside the HTA Journal, which had a mean relevance rating of 5.62 (95% CI 5.38 to 5.87) (mean difference =0.16, 95% CI −0.052 to 0.20, p=0.41; table 5).

    View this table:
    Table 5

    Mean average clinical relevance and newsworthiness ratings for review research by publication type

    NIHR HTA-funded original research reports had a similar mean average relevance rating to NIHR HTA-funded review reports with a mean rating of 5.54 (95% CI 5.34 to 5.74) compared with 5.46 (95% CI 5.32 to 5.60) (mean difference=0.08, 95% CI −0.33 to 0.17, p=0.54).

    Newsworthiness ratings in MORE

    NIHR HTA-funded reports had a mean newsworthiness rating of 4.75 (95% CI 4.60 to 4.86), which was similar to that for non-NIHR HTA publications, which had a mean newsworthiness rating of 4.70 (95% CI 4.69 to 4.71; mean difference=0.05, 95% CI −0.18 to 0.07, p=0.402; table 3). NIHR HTA-funded publications publishing outside the HTA Journal had a higher average mean newsworthiness rating than NIHR HTA reports, with a rating of 4.93 (95% CI 4.73 to 5.13; mean difference=0.18, 95% CI −0.03 to 0.39, p=0.110; table 3).

    The mean newsworthiness rating of NIHR HTA-funded original research was 5.05 (95% CI 4.88 to 5.22), which was higher than that for non-NIHR HTA original research publications, which had a mean newsworthiness rating of 4.77 (95% CI 4.75 to 4.79; mean difference=0.28, 95% CI 0.03 to 0.53, p=0.03; table 4). The mean newsworthiness rating for NIHR HTA-funded original research reports was similar to publications from the same studies publishing outside the HTA Journal, which had a mean average newsworthiness rating of 4.96 (95% CI 4.69 to 5.23; mean difference=0.09, 95% CI −0.04 to 0.22, p=0.63; table 4).

    The mean average newsworthiness rating of NIHR HTA-funded review reports was 4.64 (95% CI 4.51 to 4.77), which was similar to that for non-NIHR HTA-funded reviews, which had a mean newsworthiness rating of 4.56 (95% CI 4.54 to 4.58; mean difference=0.08, 95% CI −0.05 to 0.21, p=0.24; table 5). The mean average newsworthiness rating of NIHR HTA-funded review reports was lower than that for the same NIHR HTA-funded reviews publishing outside the HTA Journal, which had an average rating of 4.87 (95% CI 4.56 to 5.17; mean difference=0.23, 95% CI −0.11 to 0.57, p=0.21; table 5).

    NIHR HTA original research reports had a higher mean newsworthiness rating than NIHR HTA review reports with a mean rating of 5.05 (95% CI 4.88 to 5.22) compared with 4.64 (95% CI 4.51 to 4.77; mean difference=0.41, 95% CI 0.18 to 0.64, p=0.001).

    Discussion

    This study found that hypothesis one was correct in part: NIHR HTA studies reporting in HTA are rated as statistically more clinically relevant than non-NIHR HTA articles publishing in the 120 high-impact journals included by McMaster that pass CAP and MORE criteria. However, ratings of newsworthiness were similar between NIHR HTA studies reporting in HTA and non-NIHR HTA publications. The second hypothesis was not correct: ratings of clinical relevance and newsworthiness were slightly lower for NIHR HTA reports in HTA than publications from the same studies publishing outside the journal, although these differences were not statistically significant. NIHR HTA-funded original research reports publishing in the HTA Journal were seen to be more newsworthy than NIHR HTA-funded review reports.

    The study also found that the proportion of NIHR HTA reports and other publications from NIHR HTA-funded research passing initial screening criteria was higher than that of non-NIHR HTA-funded publications (67.8% and 57.6%, compared with 10.6%, respectively). This provides a further indication of the comparatively high relevance and methodological rigour of NIHR HTA-funded studies.

    It is interesting to note that subsequent to the completion of this study, another project has been completed that suggests that the assumptions behind the second hypothesis were not correct. Chinnery et al assessed the time to publication of projects funded by the NIHR HTA Programme publishing in the HTA Journal and as a journal article in the wider biomedical literature.11 They examined NIHR HTA projects with a planned submission date for the draft final report on or before 9 December 2011 and reported that for primary research the median time to publication in the HTA Journal was faster than that in other non-HTA Journal publications (26.5 and 35.5 months, respectively). This indicates that further exploration of the clinical relevance and newsworthiness of NIHR HTA reports compared with publications from the same studies publishing outside the HTA Journal is warranted.

    The strength of this study is that a rigorous, extensive and independent approach to assessing the clinical relevance and newsworthiness of funded research provided by MORE was applied to NIHR HTA reports. However, there were limitations to the study. First, the study used internal research management data that were reliant on funded research teams self-reporting their publications. It is likely that the number of publications is under-reported from NIHR HTA-funded studies and therefore some NIHR HTA publications may be included in the non-NIHR HTA publication dataset. Second, the study compared NIHR HTA reports with non-NIHR HTA publications passing CAP and MORE. NIHR HTA-funded studies are typically randomised controlled trials, systematic reviews or Technology Assessment Reports (which identify, assess and synthesise research evidence from a number of healthcare interventions, providing estimates of the relative effectiveness and cost effectiveness of a range of interventions). While subgroup analyses matched original and review research, it could be more effective to match publications by specific research designs (eg, randomised controlled trials) or by similar research funders. Third, the MORE scales were treated as interval scales and therefore mean average ratings were obtained. However, the data may be viewed as ordinal, in which case median or mode averages would be more appropriate. Fourth, raters contributing to MORE do so on a voluntary basis and are likely to be interested in evidence-based medicine, and therefore may not be representative of all physicians. Fifth, no subgroup analyses were conducted comparing articles from NIHR HTA-funded studies published elsewhere prior to publication in the HTA Journal and those published after the HTA report. Conducting such subgroup analyses will provide interesting data on the impact of prior publication on clinical relevance and newsworthiness ratings.

    Findings from this study contribute to the relatively under-researched area of clinical relevance of health research. Commissioning research topics in areas of low relevance or in areas where the research evidence is already known is a significant contributing factor to research waste.1 The MORE process has previously been used to provide exploratory assessments of clinical research by research type. McKinlay et al,6 for example, reported that systematic reviews were rated significantly higher than original articles for relevance, but significantly lower for newsworthiness. This study reflected these findings in part with NIHR HTA-funded original research reports having a similar mean average relevance rating to review reports, but a statistically higher mean newsworthiness rating than review reports. The lower newsworthiness rating of reviews is to be expected, given that all the original articles which inform the reviews are already published and thus are in the public domain.6

    The high clinical relevance rating of NIHR HTA reports in relation to non-NIHR HTA-funded research could be explained by the NIHR HTA Programme's identification and prioritisation processes, which are specifically designed to ensure that important knowledge gaps are identified. Topics are generated via a wide range of stakeholder groups and sources, such as web submissions, workshops with clinicians, academics and patient and public representatives and outputs from the James Lind Alliance Priority Setting Partnerships (http://www.lindalliance.org/). These are screened by specialists and an ‘overlap check’ is undertaken to ensure that there is no duplication of current or completed research internally and externally, after which topics are developed and prioritised by panels comprising independent experts and patient and public contributors. The higher clinical relevance rating of NIHR HTA reports may therefore be attributed to the effectiveness of these processes.

    An additional factor may be due to the fact that the NIHR HTA Programme has its own journal HTA, which enables fuller descriptions of research than most other journals allow. The accessibility and structure of the reports in HTA may be a contributing factor to enhancing assessments of relevance. However, all NIHR HTA-funded studies are contractually obliged to report in HTA regardless of the study outcome, resulting in a high publication rate of funded studies.8 The HTA Journal is therefore not as likely to be affected by publication bias as other health research literature, which could therefore limit ratings of relevance and newsworthiness when compared to other journals.

    This study presents an exploratory assessment of clinical relevance and newsworthiness by assessing the outputs of one research funder. While this provides an initial assessment of relevance and newsworthiness in relation to highly rated publications in high-impact journals, further research is recommended to confirm the observations from the study. In particular, it is recommended that further analyses are undertaken on outputs that are matched by research type (eg, RCTs, systematic reviews), research funders or specific journals. Further research should also be conducted on assessing the degree to which prior publication of NIHR HTA-funded studies influences ratings of clinical relevance and newsworthiness of HTA reports.

    Commissioning or publishing research of low-clinical relevance is an important factor in the waste of research funds. It is recommended, therefore, that careful consideration is given to clinical relevance when funding and disseminating research.

    Conclusion

    Research commissioned by the NIHR HTA Programme and published in the HTA Journal is more clinically relevant than highly rated articles publishing in the 120 journals included in MORE. Reports publishing in the HTA Journal had similar ratings of newsworthiness. NIHR HTA-funded original research reports had a similar mean average relevance rating to review reports, but had a significantly higher newsworthiness rating.

    The clear and full accounts provided by the HTA Journal and the rigorous topic identification and prioritisation processes adopted by the NIHR HTA Programme may be a factor in generating the high clinical relevance rating. This study illustrates the importance of considering clinical relevance when commissioning and disseminating research in an effort to minimise waste in research funding.

    Acknowledgments

    The authors wish to thank the support of Carole Luke and Beata Ferris who assisted in extracting publication records and PubMed Identification numbers for National Institute for Health Research (NIHR) Health Technology Assessment (HTA)-funded studies. They also thank Dr Martin Ashton-Key for his assistance in commenting on an early draft of this paper.

    References

    1. ↵
      1. Chalmers I,
      2. Glasziou P
      . Avoidable waste in the production and reporting of research evidence. Lancet 2009;374:86–9.
      OpenUrlCrossRefPubMedWeb of Science
    2. ↵
      1. Gillum LA,
      2. Gouveia C,
      3. Dorsey ER,
      4. et al
      . NIH disease funding levels and burden of disease. PLoS ONE 2011;6:e16837.
      OpenUrlCrossRefPubMed
    3. ↵
      1. Tallon D,
      2. Chard J,
      3. Dieppe P
      . Relation between agendas of the research community and the research consumer. Lancet 2000;355:2037–40.
      OpenUrlCrossRefPubMedWeb of Science
    4. ↵
      1. McKibbon K,
      2. Wilczynski N,
      3. Haynes R
      . What do evidence-based secondary journals tell us about the publication of clinically important articles in primary healthcare journals? BMC Med 2004;2:33.
      OpenUrlCrossRefPubMed
    5. ↵
      1. Lokker C,
      2. McKibbon KA,
      3. McKinlay RJ,
      4. et al
      . Prediction of citation counts for clinical articles at two years using data available within three weeks of publication: retrospective cohort study. BMJ 2008;336:655–7.
      OpenUrlAbstract/FREE Full Text
    6. ↵
      1. McKinlay RJ,
      2. Cotoi C,
      3. Wilczynski NL,
      4. et al
      . Systematic reviews and original articles differ in relevance, novelty, and use in an evidence-based service for physicians: PLUS project. J Clin Epidemiol 2008;61:449–54.
      OpenUrlCrossRefPubMed
    7. ↵
      1. Raftery J,
      2. Powell J
      . Health technology assessment in the UK. Lancet 2013;382:1278–85.
      OpenUrlCrossRefPubMedWeb of Science
    8. ↵
      1. Turner S,
      2. Wright D,
      3. Maeso R,
      4. et al
      . Publication rate for funded studies from a major UK health research funder: a cohort study. BMJ Open 2013;3:e002521.
      OpenUrlAbstract/FREE Full Text
    9. ↵
      1. Hanney S,
      2. Buxton M,
      3. Green C,
      4. et al
      . An assessment of the impact of the NHS Health Technology Assessment Programme. Health Technol Assess 2007;11:iii–iv, ix–xi, 1–180.
    10. ↵
      1. Wilczynski N,
      2. Morgan D,
      3. Haynes RB,
      4. et al
      . The Hedges Team. An overview of the design and methods for retrieving high-quality studies for clinical care. BMC Med Inform Decis Mak 2005;5:20.
      OpenUrlCrossRefPubMed
    11. ↵
      1. Chinnery F,
      2. Young A,
      3. Goodman J,
      4. et al
      . Time to publication for NIHR HTA programme-funded research: a cohort study. BMJ Open 2013;3:e004121.
      OpenUrlAbstract/FREE Full Text

    Footnotes

    • Contributors The study was designed by DW, RMi, RBH, ST and RMa. DW, AY and EI extracted the data. EI conducted the data analyses. DW drafted the manuscript. All authors have read and approved the final manuscript.

    • Funding This project was funded as part of the NETSCC Research on Research Programme. The views and opinions expressed are those of the authors and do not necessarily reflect those of the Department of Health and of NETSCC.

    • Competing interests DW, AY, RMa and ST are employees at the NIHR Evaluation, Trials and Studies Coordinating Centre (NETSCC). RMi is employed as the Head of NETSCC and has worked for NETSCC (and its predecessor organisation) in senior roles intermittently since 1996. He was an editor of the Health Technology Assessment journal (1997–2007) and a founder editor for other journals in the new NIHR Journals Library (2011–12). RBH and EI received payment to McMaster University from NETSCC for time taken to analyse the MORE database.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Data sharing statement No additional data are available.