You are here

  • Home
  • Archive
  • Volume 9, Issue 3
  • Comprehensive Researcher Achievement Model (CRAM): a framework for measuring researcher achievement, impact and influence derived from a systematic literature review of metrics and models

Article Text

Article menu

Download PDFPDF

XML
Research methods
Research
Comprehensive Researcher Achievement Model (CRAM): a framework for measuring researcher achievement, impact and influence derived from a systematic literature review of metrics and models
  1. Jeffrey Braithwaite1,
  2. Jessica Herkes1,
  3. Kate Churruca1,
  4. Janet C Long1,
  5. Chiara Pomare1,
  6. Claire Boyling1,
  7. Mia Bierbaum1,
  8. Robyn Clay-Williams1,
  9. Frances Rapport1,
  10. Patti Shih1,
  11. Anne Hogden1,
  12. Louise A Ellis1,
  13. Kristiana Ludlow1,
  14. Elizabeth Austin1,
  15. Rebecca Seah1,
  16. Elise McPherson1,
  17. Peter D Hibbert1,2,
  18. Johanna Westbrook1
  1. 1 Australian Institute of Health Innovation, Macquarie University, North Ryde, New South Wales, Australia
  2. 2 Division of Health Sciences, University of South Australia, Adelaide, South Australia, Australia
  1. Correspondence to Professor Jeffrey Braithwaite; jeffrey.braithwaite{at}mq.edu.au

Abstract

Objectives Effective researcher assessment is key to decisions about funding allocations, promotion and tenure. We aimed to identify what is known about methods for assessing researcher achievements, leading to a new composite assessment model.

Design We systematically reviewed the literature via the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols framework.

Data sources All Web of Science databases (including Core Collection, MEDLINE and BIOSIS Citation Index) to the end of 2017.

Eligibility criteria (1) English language, (2) published in the last 10 years (2007–2017), (3) full text was available and (4) the article discussed an approach to the assessment of an individual researcher’s achievements.

Data extraction and synthesis Articles were allocated among four pairs of reviewers for screening, with each pair randomly assigned 5% of their allocation to review concurrently against inclusion criteria. Inter-rater reliability was assessed using Cohen’s Kappa (ĸ). The ĸ statistic showed agreement ranging from moderate to almost perfect (0.4848–0.9039). Following screening, selected articles underwent full-text review and bias was assessed.

Results Four hundred and seventy-eight articles were included in the final review. Established approaches developed prior to our inclusion period (eg, citations and outputs, h-index and journal impact factor) remained dominant in the literature and in practice. New bibliometric methods and models emerged in the last 10 years including: measures based on PageRank algorithms or ‘altmetric’ data, methods to apply peer judgement and techniques to assign values to publication quantity and quality. Each assessment method tended to prioritise certain aspects of achievement over others.

Conclusions All metrics and models focus on an element or elements at the expense of others. A new composite design, the Comprehensive Researcher Achievement Model (CRAM), is presented, which supersedes past anachronistic models. The CRAM is modifiable to a range of applications.

  • researcher assessment
  • research metrics
  • h-index
  • journal impact factor
  • citations
  • outputs
  • Comprehensive Researcher Achievement Model (CRAM)

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

http://dx.doi.org/10.1136/bmjopen-2018-025320

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Strengths and limitations of this study

    • A large, diverse dataset of over 478 articles, containing many ideas for assessing researcher performance, was analysed.

    • Strengths of the review include executing a wide-ranging search strategy, and the consequent high number of included articles for review; the results are limited by the literature itself, for example, new metrics were not mentioned in the articles, and therefore not captured in the results.

    • A new model combining multiple factors to assess researcher performance is now available.

    • Its strengths include combining quantitative and qualitative components in the one model.

    • The Comprehensive Researcher Achievement Model, despite being evidence oriented, is a generic one and now needs to be applied in the field.

    Introduction  

    Judging researchers’ achievements and academic impact continues to be an important means of allocating scarce research funds and assessing candidates for promotion or tenure. It has historically been carried out through some form of expert peer judgement to assess the number and quality of outputs and, in more recent decades, citations to them. This approach requires judgements regarding the weight that should be assigned to the number of publications, their quality, where they were published and their downstream influence or impact. There are significant questions about the extent to which human judgement based on these criteria is an effective mechanism for making these complex assessments in a consistent and unbiased way.1–3 Criticisms of peer assessment, even when underpinned by relatively impartial productivity data, include the propensity for bias, inconsistency among reviewers, nepotism, group-think and subjectivity.4–7

    To compensate for these limitations, approaches have been proposed that rely less on subjective judgement and more on objective indicators.3 8–10 Indicators of achievement focus on one or a combination of four aspects: quantity of researcher outputs (productivity); value of outputs (quality); outcomes of research outputs (impact); and relations between publications or authors and the wider world (influence).11–15 Online publishing of journal articles has provided the opportunity to easily track citations and user interactions (eg, number of article downloads) and thus has provided a new set of indices against which individual researchers, journals and articles can be compared and the relative worth of contributions assessed and valued.14 These relatively new metrics have been collectively termed bibliometrics 16 when based on citations and numbers of publications, or altmetrics 17 when calculated by alternative online measures of impact such as number of downloads or social media mentions.16

    The most established metrics for inferring researcher achievement are the h-index and the journal impact factor (JIF). The JIF measures the average number of citations of an article in the journal over the previous year, and hence is a good indication of journal quality but is increasingly regarded as a primitive measure of quality for individual researchers.18 The h-index, proposed by Hirsch in 2005,19 attempts to portray a researcher’s productivity and impact in one data point. The h-index is defined as the number (h) of articles published by a researcher that have received a citation count of at least h. Use of the h-index has become widespread, reflected in its inclusion in author profiles on online databases such as Google Scholar and Scopus.

    Also influenced by the advent of online databases, there has been a proliferation of other assessment models and metrics,16 many of which purport to improve on existing approaches.20 21 These include methods that assess the impact of articles measured by: downloads or online views received, practice change related to specific research, take-up by the scientific community or mentions in social media.

    Against the backdrop of growth in metrics and models for assessing researchers’ achievements, there is a lack of guidance on the relative strengths and limitations of these different approaches. Understanding them is of fundamental importance to funding bodies that drive the future of research, tenure and promotion committees and more broadly for providing insights into how we recognise and value the work of science and scientists, particularly those researching in medicine and healthcare. This review aimed to identify approaches to assessing researchers’ achievements published in the academic literature over the last 10 years, considering their relative strengths and limitations and drawing on this to propose a new composite assessment model.

    Method

    Search strategy

    All Web of Science databases (eight in total, including the Web of Science Core Collection, MEDLINE and BIOSIS Citation Index) were searched using terms related to researcher achievement (researcher excellence, track record, researcher funding, researcher perform*, relative to opportunity, researcher potential, research* career pathway, academic career pathway, funding system, funding body, researcher impact, scientific* productivity, academic productivity, top researcher, researcher ranking, grant application, researcher output, h*index, i*index, impact factor, individual researcher) and approaches to its assessment (model, framework, assess*, evaluat*, *metric*, measur*, criteri*, citation*, unconscious bias, rank*) with ‘*’ used as an unlimited truncation to capture variation in search terms, as seen in online supplementary appendix 1. These two searches were combined (using ‘and’), and results were downloaded into EndNote,22 the reference management software.

    Supplemental material

    Study selection

    After removing duplicate references in EndNote, articles were allocated among pairs of reviewers (MB–JL, CP–CB, KL–JH and KC-LAE) for screening against inclusion criteria. Following established procedures,23 24 each pair was randomly assigned 5% of their allocation to review concurrently against inclusion criteria, with inter-rater reliability assessed using Cohen’s kappa (ĸ). The ĸ statistic was calculated for pairs of researchers, with agreement ranging from moderate to almost perfect (0.4848–0.9039).25 Following the abstract and title screen, selected articles underwent full text review. Reasons for exclusion were recorded.

    Inclusion criteria

    The following inclusion criteria were operationalised: (1) English language, (2) published in the last 10 years (2007–2017), (3) full text for the article was available, and (4) the article discussed an approach to the assessment of an individual researcher’s achievements (at the researcher or singular output-level). The research followed the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols framework.26 Empirical and non-empirical articles were included because many articles proposing new approaches to assessment, or discussing the limitations of existing ones, are not level one evidence or research based. Both quantitative and qualitative studies were included.

    Data extraction

    Data from the included articles were extracted, including: the country of article origin, the characteristics of the models or metrics discussed, the perspective the article presented on the metric or model (positive, negative and indeterminable) including any potential benefits or limitations of the assessment model (and if these were perceived or based on some form of evidence). A customised data extraction sheet was developed in Microsoft Excel, trialled among members of the research team and subsequently refined. This information was synthesised for each model and metric identified through narrative techniques. The publication details and classification of each paper are contained in online supplementary appendix 2.

    Supplemental material

    Appraisal of the literature

    Due to the prevalence of non-empirical articles in this field (eg, editorial contributions and commentaries), it was determined that a risk of bias tool such as the Quality Assessment Tool could not be applied.27 Rather, assessors were trained in multiple meetings (24 October, 30 October and 13 November 2017) to critically assess the quality of articles. Given the topic of the review (focusing on the publication process), the type of models and metrics identified (ie, more metrics that use publication data metrics) may influence the cumulative evidence and subsequently create a risk of bias. In addition, three researchers (JH, EM and CB) reviewed every included article to extract documented conflicts of interests of authors.

    Patient and public involvement

    Patients and the public were not involved in this systematic review.

    Results

    The final dataset consisted of 478 academic articles. The data screening process is presented in figure 1.

    Figure 1
    Figure 1

    Data screening and extraction process for academic articles.

    Of the 478 included papers (see online supplementary appendix 2 for a summary), 295 (61.7%) had an empirical component, which ranged from interventional studies that assessed researcher achievement as an outcome measure (eg, a study measuring the outcomes of a training programme),28 as a predictor29–31 (eg, a study that demonstrated the association between number of citations early in one’s career and later career productivity) or reported a descriptive analysis of a new metric.32 33 One hundred and sixty-six (34.7%) papers were not empirical, including editorial or opinion contributions that discussed the assessment of research achievement, or proposed models for assessing researcher achievement. Seventeen papers (3.6%) were reviews that considered one or more elements of assessing researcher achievements. The quality of these contributions ranged in terms of the risk of bias in the viewpoint expressed. Only for 19 papers (4.0%) did the authors declare a potential conflict of interest.

    Across the study period, 78 articles (16.3%) involved authors purporting to propose new models or metrics. Most articles described or cited pre-existing metrics and largely discussed their perceived strengths and limitations. Figure 2 shows the proportion of positive or negative discussions of five of the most common approaches to assessing an individual’s research achievement (altmetrics, peer-review, h-index, simple counts and JIF). The approach with most support was altmetrics (51.0% of articles mentioning altmetrics). The JIF was discussed with mostly negative sentiments in relevant articles (69.4%).

    Figure 2
    Figure 2

    Percentages of positive and negative discussion regarding selected commonly used metrics for assessing individual researchers (n=478 articles). Positive discussion refers to articles that discuss the metric in favourable light or focus on the strengths of the metric; negative discussion refers to articles that focus on the limitations or shortcomings of the metric. JIF, journal impact factor.

    Citation-based metrics

    Publication and citation counts

    One hundred and fifty-three papers (32.0%) discussed the use of publication and citation counts for purposes of assessing researcher achievement, with papers describing them as a simple ‘traditional but somewhat crude measure’,34 as well as the building blocks for other metrics.35 A researcher’s number of publications, commonly termed an n-index,36 was suggested by some to indicate researcher productivity,14 rather than quality, impact or influence of these papers.37 However, the literature suggested that numbers of citations indicated the academic impact of an individual publication or researcher’s body of work, calculated as an author’s cumulative or mean citations per article.38 Some studies found support for the validity of citation counts and publications in that they were correlated with other indications of a researcher’s achievement, such as awards and grant funding,39 40 and were predictive of long-term success in a field.41 For example, one paper argued that having larger numbers of publications and being highly cited early in one’s career predicted later high-quality research.42

    A number of limitations of using citation or publication counts was observed. For example, Minasny et al 43 highlighted discrepancies between publications and citations counts in different databases because of their differential structures and inputs.43 Other authors38 44 45 noted that citation patterns vary by discipline, which they suggested can make them inappropriate for comparing researchers from different fields. Average citations per publication were reported as highly sensitive to change or could be skewed if, for example, a researcher has one heavily cited article.46 47 A further disadvantage is the lag-effect of citations48 49 and that, in most models, citations and publications count equally for all coauthors despite potentially differential contributions.50 Some also questioned the extent to which citations actually indicated quality or impact, noting that a paper may influence clinical practice more than academic thinking.51 Indeed, a paper may be highly cited because it is useful (eg, a review), controversial or even by chance, making citations a limited indication of quality or impact.40 50 52 In addition to limitations, numerous authors made the point that focusing on citation and publication counts can have unintended, negative consequences for the assessment of researcher achievement, potentially leading to gaming and manipulation, including self-citations and gratuitous authorship.53 54

    Singular output-level approaches

    Forty-one papers (8.6%) discussed models and metrics at the singular output or article-level that could be used to infer researcher achievement. The components of achievement they reported assessing were typically quality or impact.55 56 For example, some papers reported attempts to examine the quality of a single article by assessing its content.57 58 Among the metrics identified in the literature, the immediacy index focused on impact by measuring the average number of cites an article received in the year it was published.59 Similarly, Finch21 suggested adapting the Source Normalized Impact per Publication (a metric used for journal-level calculations across different fields of research) to the article-level.

    Many of the article-level metrics identified could also be upscaled to produce researcher-level indications of academic impact. For example, the sCientific currENcy Tokens (CENTs), proposed by Szymanski et al 60 involved giving a ‘cent’ for each new non-self-citation a publication received; CENTs are then used as the basis for the researcher-level i-index, which follows a similar approach as the h-index but removes self-citations.60 The temporally averaged paper-specific impact factor calculates an article’s average number of citations per year combined with bonus cites for the publishing journal’s prestige and can be aggregated to measure the overall relevance of a researcher (temporally averaged author-specific impact factor).61

    Journal impact factor

    The JIF, commonly recognised as a journal-level measure of quality,59 62 was discussed in 211 (44.1%) of the papers reviewed in relation to assessing singular outputs or individual researchers. A number of papers described the JIF being used informally to assess an individual’s research achievement at the singular output-level and formally in countries such as France and China.63 It implies article quality because it is typically a more competitive process to publish in journals with high impact factors.64 Indeed, the JIF was found to be the best predictor of a paper’s propensity to receive citations.65

    The JIF has a range of limitations when used to indicate journal quality,66 including that it is disproportionally affected by highly cited, outlier articles41 67 and is susceptible to ‘gaming’ by editors.17 68 Other criticisms focused on using the JIF to assess individual articles or the researchers who author them.69 Some critics claimed that using the JIF to measure an individual’s achievement encourages researchers to publish in higher impact but less appropriate journals for their field, which ultimately means their article may not be read by relevant researchers.70 71 Furthermore, the popularity of a journal was argued to be a poor indication of the quality of any one article, with the citation distributions for calculating JIF found to be heavily skewed (ie, a small subset of papers receive the bulk of the citations, while some may receive none).18 Ultimately, many commentators argued that the JIF is an inappropriate metric to assess individual researchers because it is an aggregate metric of a journal’s publication and expresses nothing about any individual paper.21 49 50 72 However, Bornmann and Pudovkin73 suggested one case in which it would be appropriate to use JIF for assessing individual researchers: in relation to their recently published papers that had not had the opportunity to accumulate citations.73

    Researcher-level approaches

    h-Index

    The h-index was among the most commonly discussed metrics in the literature (254 [53.1%] of the papers reviewed); in many of these papers, it was described by authors as more sophisticated than citation and publication counts but still straightforward, logical and intuitive.74–76 Authors noted its combination of productivity (h publications) and impact indicators (h citations) as being more reliable77 78 and stable than average citations per publications,41 because it is not skewed by the influence of one popular article.79 One study found that the h-index correlated with other metrics more difficult to obtain.76 It also showed convergent validity with peer-reviewed assessments80 and was found to be a good predictor of future achievement.41

    However, because of the lag-effect with citations and publications, the h-index increases with a researcher’s years of activity in the field, and cannot decrease, even if productivity later declines.81 Hence, numerous authors suggested it was inappropriate for comparing researchers at different career stages,82 or those early in their career.68 The h-index was also noted as being susceptible to many of the critiques levelled against citation counts, including potential for gaming and inability to reflect differential contributions by coauthors.83 Because disciplines differ in citation patterns,84 some studies noted variations in author h-indices between different methodologies85 and within medical subspecialties.86 Some therefore argued that the h-index should not be used as the sole measure of a researcher’s achievement.86

    h-Index variants

    A number of modified versions of the h-index were identified; these purported to draw on its basic strengths of balancing productivity with impact while redressing perceived limitations. For example, the g-index measures global citation performance87 and was defined similarly to the h-index but with more weight given to highly cited articles by assuming the top g articles have received at least g2citations.88 Azer and Azer89 argued it was a more useful measure of researcher productivity.89 Another variant of the h-index identified, the m-quotient, was suggested to minimise the potential to favour senior academics by accounting for the time passed since a researcher has begun publishing papers.90 91 Other h-index variations reported in the articles reviewed attempted to account for author contributions, such as the h-maj index, which includes only articles in which the researcher played a core role (based on author order), and the weighted h-index, which assigns credit points according to author order.87 92

    Recurring issues with citation-based metrics

    The literature review results suggested that no one citation-based metric was ideal for all purposes. All of the common metrics examined focused on one aspect of an individual’s achievement and thus failed to account for other aspects of achievement. The limitations with some of the frequently used citation-based metrics are listed in box 1.

    Box 1

    Common limitations in the use of citation-based metrics

    1. Challenges with reconciling differences in citation patterns across varying fields of study.

    2. Time-dependency issues stemming from differences in career length of researchers.

    3. Prioritising impact over merit, or quality over quantity, or vice versa.

    4. The lag-effect of citations.

    5. Gaming and the ability of self-citation to distort metrics.

    6. Failure to account for author order.

    7. Contributions from authors to a publication are viewed as equal when they may not be.

    8. Perpetuate ‘publish or perish’ culture.

    9. Potential to stifle innovation in favour of what is popular.

    Non-citation-based approaches

    Altmetrics

    In contradistinction with the metrics discussed above, 54 papers (11.3%) discussed altmetrics (or ‘alternative metrics’), which included a wide range of techniques to measure non-traditional, non-citation based usage of articles, that is, influence.17 Altmetric measures included the number of online article views,93 bookmarks,94 downloads,41 PageRank algorithms95 and attention by mainstream news,63 in books96 and social media, for example, in blogs, commentaries, online topic reviews or Tweets.97 98 These metrics typically measure the ‘web visibility’ of an output.99 A notable example is the social networking site for researchers and scientists, ResearchGate, which uses an algorithm to score researchers based on the use of their outputs, including citations, reads and recommendations.100

    A strength of altmetrics lies in providing a measure of influence promptly after publication.68 101 102 Moreover, altmetrics allows tracking of the downloads of multiple sources (eg, students, the general public, clinicians, as well as academics) and multiple types of format (eg, reports and policy documents),103 which are useful in gauging a broader indication of impact or influence, compared with more traditional metrics that solely or largely measure acknowledgement by experts in the field through citations.17

    Disadvantages noted in the articles reviewed included that altmetrics calculations have been established by commercial enterprises such as Altmetrics LLC (London, UK) and other competitors,104 and there may be fees levied for their use. The application of these metrics has also not been standardised.96 Furthermore, it has been argued that, because altmetrics are cumulative and typically at the article-level, they provide more an indication of influence or even popularity,105 instead of quality or productivity.106 Hence, one study suggested no correlation between attention on Twitter and expert analysis of an article’s originality, significance or rigour.107 Another showed that Tweets predict citations.108 Overall, further work needs to assess the value of altmetric scores in terms of their association with other traditional indicators of achievement.109 Notwithstanding this, there were increasing calls to consider altmetrics alongside more conventional metrics in assessing researchers and their work.110

    Past funding

    A past record of being funded by national agencies was identified as a common measurement of individual academic achievement (particularly productivity, quality and impact) in a number of papers and has been argued to be a reliable method that is consistent across medical research.111–113 For example, the National Institute of Health’s (NIH) Research Portfolio Online Reporting Tools system encourages public accountability for funding by providing online access to reports, data and NIH-funded research projects.111 114

    New metrics and models identified

    The review also identified and assessed new metrics and models that were proposed during the review period, many of which had not gained widespread acceptance or use. While there was considerable heterogeneity and varying degrees of complexity among the 78 new approaches identified, there were also many areas of overlap in their methods and purposes. For example, some papers reported on metrics that used a PageRank algorithm,115 116 a form of network analysis based on structural characteristics of publications (eg, coauthorship or citation patterns).14 Metrics based on PageRank purported to measure both the direct and indirect impacts of a publication or researcher. Other approaches considered the relative contributions of authors to a paper in calculating productivity.117 Numerous metrics and models that built on existing approaches were also reported.118 For example, some developed composite metrics that included a publication’s JIF alongside an author contribution measure119 or other existing metrics.120 However, each of these approaches reported limitations, in addition to their strengths or improvements on other methods. For example, in focusing on productivity, a metric necessarily often neglected impact.121 Online supplementary appendix 3 provides a summary of these new or refashioned metrics and models, with details of their basis and purpose.

    Supplemental material

    Discussion

    This systematic review identified a large number of diverse metrics and models for assessing an individual’s research achievement that have been developed in the last 10 years (2007–2017), as evidenced in online supplementary appendix 3. At the same time, other approaches that pre-dated our study time period of 2007–2017 were also discussed frequently in the literature reviewed, including the h-index and JIF. All metrics and models proposed had their relative strengths, based on the components of achievement they focused on, and their sophistication or transparency.

    The review also identified and assessed new metrics emerging over the past few decades. Peer-review has been increasingly criticised for reliance on subjectivity and propensity for bias,7 and there have been arguments that the use of specific metrics may be a more objective and fair approach for assessing individual research achievement. However, this review has highlighted that even seemingly objective measures have a range of shortcomings. For example, there are inadequacies in comparing researchers at different career stages and across disciplines with different citation patterns.84 Furthermore, the use of citation-based metrics can lead to gaming and potential ethical misconduct by contributing to a ‘publish or perish’ culture in which researchers are under pressure to maintain or improve their publication records.122 123 New methods and adjustments to existing metrics have been proposed to explicitly address some of these limitations; for example, normalising metrics with ‘exchange rates’ to remove discipline-specific variation in citation patterns, thereby making metric scores more comparable for researchers working in disparate fields.124 125 Normalisation techniques have also been used to assess researchers’ metrics with greater recognition of their relative opportunity and career longevity.126

    Other criticisms of traditional approaches centre less on how they calculated achievement and more on what they understood or assumed about its constituent elements. In this review, the measurement of impact or knowledge gain was often exclusively tied to citations.127 Some articles proposed novel approaches to using citations as a measure of impact, such as giving greater weight to citations from papers that were themselves highly cited128 or that come from outside the field in which the paper was published.129 However, even other potential means of considering scientific contributions and achievement, such as mentoring, were still ultimately tied to citations because mentoring was measured by the publication output of mentees.130

    A focus only on citations was widely thought to disadvantage certain types of researchers. For example, researchers who aim to publish with a focus on influencing practice may target more specialised or regional journals that do not have high JIFs, where their papers will be read by the appropriate audience and findings implemented, but they may not be well cited.51 In this regard, categorising the type of journal in which an article has been published in terms of its focus (eg, industry, clinical and regional/national) may go some way towards recognising those publications that have a clear knowledge translation intention and therefore prioritise real-world impact over academic impact.122 There were only a few other approaches identified that captured broader conceptualisations of knowledge gain, such as practical impact or wealth generation for the economy, and these too were often simplistic, such as including patents and their citations131 or altmetric data.96 While altmetrics hold potential in this regard, their use has not been standardised,96 and they come with their own limitations, with suggestions that they reflect popularity more so than real-world impact.105 Other methodologies have been proposed for assessing knowledge translation and real-world impact, but these can often be labour intensive.132 For example, Sutherland et al 133 suggested that assessing individual research outputs in light of specific policy objectives through peer-review based scoring, may be a strategy, but this is typically not feasible in situations such as grant funding allocation, where there are time constraints and large applicant pools to assess.

    In terms of how one can make sense of the validity of many of these emerging approaches for assessing an individual’s research achievements, metrics should demonstrate their legitimacy empirically, as well as having a theoretical basis for their use and clearly differentiating what aspects of quality, achievement or impact they purport to examine.55 65 If the recent, well-publicised134–136 San Francisco Declaration on Research Assessment137 is anything to go by, internationally, there is a move away from the assessment of individual researchers using the JIF and the journal in which the research has been published.

    There is momentum, instead, for assessment of researcher achievements on the basis of a wider mix of measures, hence our proposed Comprehensive Researcher Achievement Model (CRAM) (figure 3). On the left-hand side of this model is the researcher to be assessed and key characteristics that influence the assessment. Among these factors, some (ie, field or discipline, coauthorship and career longevity) can be controlled for depending on the metric, while other components, such as gaming or the research topic (ie, whether it is ‘trendy’ or innovative), are less amenable to control or even prediction. Online databases, which track citations and downloads and measure other forms of impact, hold much potential and will likely be increasingly used in the future to assess both individual researchers and their outputs. Hence, assessment components (past funding, articles, citations, patents, downloads and social media traction) included in our model are those primarily accessible online.

    Figure 3
    Figure 3

    The Comprehensive Researcher Achievement Model.

    Strengths and limitations

    The findings of this review suggest assessment components should be used with care, with recognition of how they can be influenced by other factors, and what aspects of achievement they reflect (ie, productivity, quality, impact and influence). No metric or model singularly captures all aspects of achievement, and hence use of a range, such as the examples in our model, is advisable. CRAM recognises that the configuration and weighting of assessment methods will depend on the assessors and their purpose, the resources available for the assessment process and access to assessment components. Our results must be interpreted in light of our focus on academic literature. The limits of our focus on peer-reviewed literature were evident in the fact that some new metrics were not mentioned in articles and therefore not captured in our results. While we defined impact broadly at the outset, overwhelmingly, the literature we reviewed focused on academic, citation-based impact. Furthermore, although we assessed bias in the ways documented, the study design limited our ability to apply a standardised quality assessment tool. A strength of our focus was that we set no inclusion criteria with regard to scientific discipline, because novel and useful approaches to assessing research achievement can come from diverse fields. Many of the articles we reviewed were broadly in the area of health and medical research, and our discussion is concerned with the implications for health and medical research, as this is where our interests lie.

    Conclusion

    There is no ideal model or metric by which to assess individual researcher achievement. We have proposed a generic model, designed to minimise risk of the use of any one or a smaller number of metrics, but it is not proposed as an ultimate solution. The mix of assessment components and metrics will depend on the purpose. Greater transparency in approaches used to assess achievement including their evidence base is required.37 Any model used to assess achievement for purposes such as promotion or funding allocation should include some quantitative components, based on robust data, and be able to be rapidly updated, presented with confidence intervals and normalised.37 The assessment process should be difficult to manipulate and explicit about the components of achievement being measured. As such, no current metric suitably fulfils all these criteria. The best strategy to assess an individual’s research achievement is likely to involve the use of multiple approaches138 in order to dilute the influence and potential disadvantages of any one metric while providing more rounded picture of a researcher’s achievement83 139; this is what the CRAM aims to contribute.

    All in all, achievement in terms of impact and knowledge gain is broader than the number of articles published or their citation rates and yet most metrics have no means of factoring in these broader issues. Altmetrics hold promise in complementing citation-based metrics and assessing more diverse notions of impact, but usage of this type of tool requires further standardisation.96 Finally, despite the limitations of peer-review, the role of expert judgement should not be discounted.41 Metrics are perhaps best applied as a complement or check on the peer-review process, rather than the sole means of assessment of an individual’s research achievements.140

    References

    1. 1.↵
      2. Ibrahim N ,
      3. Habacha Chaibi A ,
      4. Ben Ahmed M
      . New scientometric indicator for the qualitative evaluation of scientific production. New Libr World 2015;116:661–76.doi:10.1108/NLW-01-2015-0002
      OpenUrl
    2. 2.↵
      2. Franco Aixelá J ,
      3. Rovira-Esteva S
      . Publishing and impact criteria, and their bearing on translation studies: In search of comparability. Perspectives 2015;23:265–83.doi:10.1080/0907676X.2014.972419
      OpenUrl
    3. 3.↵
      2. Belter CW
      . Bibliometric indicators: opportunities and limits. J Med Libr Assoc 2015;103:219–21.doi:10.3163/1536-5050.103.4.014
      OpenUrl
    4. 4.↵
      2. Frixione E ,
      3. Ruiz-Zamarripa L ,
      4. Hernández G
      . Assessing individual intellectual output in scientific research: Mexico’s national system for evaluating scholars performance in the humanities and the behavioral sciences. PLoS One 2016;11:e0155732.doi:10.1371/journal.pone.0155732
    5. 5.↵
      2. Marzolla M
      . Assessing evaluation procedures for individual researchers: The case of the Italian National Scientific Qualification. J Informetr 2016;10:408–38.doi:10.1016/j.joi.2016.01.009
      OpenUrl
    6. 6.↵
      2. Marsh HW ,
      3. Jayasinghe UW ,
      4. Bond NW
      . Improving the peer-review process for grant applications: reliability, validity, bias, and generalizability. Am Psychol 2008;63:160–8.doi:10.1037/0003-066X.63.3.160
      OpenUrlCrossRefPubMedWeb of Science
    7. 7.↵
      2. Kaatz A ,
      3. Magua W ,
      4. Zimmerman DR , et al
      . A quantitative linguistic analysis of National Institutes of Health R01 application critiques from investigators at one institution. Acad Med 2015;90:69–75.doi:10.1097/ACM.0000000000000442
      OpenUrlCrossRefPubMed
    8. 8.↵
      2. Aoun SG ,
      3. Bendok BR ,
      4. Rahme RJ , et al
      . Standardizing the evaluation of scientific and academic performance in neurosurgery--critical review of the "h" index and its variants. World Neurosurg 2013;80:e85–90.doi:10.1016/j.wneu.2012.01.052
      OpenUrl
    9. 9.↵
      2. Hicks D ,
      3. Wouters P ,
      4. Waltman L , et al
      . Bibliometrics: the Leiden Manifesto for research metrics. Nature 2015;520:429–31.doi:10.1038/520429a
      OpenUrlCrossRefPubMed
    10. 10.↵
      2. King J
      . A review of bibliometric and other science indicators and their role in research evaluation. J Inf Sci 1987;13:261–76.doi:10.1177/016555158701300501
      OpenUrlCrossRef
    11. 11.↵
      2. Abramo G ,
      3. Cicero T ,
      4. D’Angelo CA
      . A sensitivity analysis of researchers’ productivity rankings to the time of citation observation. J Informetr 2012;6:192–201.doi:10.1016/j.joi.2011.12.003
      OpenUrl
    12. 12.↵
      2. Arimoto A
      . Declining symptom of academic productivity in the Japanese research university sector. High Educ 2015;70:155–72.doi:10.1007/s10734-014-9848-4
      OpenUrl
    13. 13.↵
      2. Carey RM
      . Quantifying scientific merit is it time to transform the impact factor? Circ Res 2016;119:1273–5.doi:10.1161/CIRCRESAHA.116.309883
      OpenUrlFREE Full Text
    14. 14.↵
      2. Durieux V ,
      3. Gevenois PA
      . Bibliometric indicators: quality measurements of scientific publication. Radiology 2010;255:342–51.doi:10.1148/radiol.09090626
      OpenUrlCrossRefPubMed
    15. 15.↵
      2. Selvarajoo K
      . Measuring merit: take the risk. Science 2015;347:139–40.doi:10.1126/science.347.6218.139-c
      OpenUrlFREE Full Text
    16. 16.↵
      2. Wildgaard L ,
      3. Schneider JW ,
      4. Larsen B
      . A review of the characteristics of 108 author-level bibliometric indicators. Scientometrics 2014;101:125–58.doi:10.1007/s11192-014-1423-3
      OpenUrlCrossRef
    17. 17.↵
      2. Maximin S ,
      3. Green D
      . Practice corner: the science and art of measuring the impact of an article. Radiographics 2014;34:116–8.doi:10.1148/rg.341134008
      OpenUrl
    18. 18.↵
      2. Callaway E
      . Beat it, impact factor! Publishing elite turns against controversial metric. Nature 2016;535:210–1.doi:10.1038/nature.2016.20224
      OpenUrl
    19. 19.↵
      2. Hirsch JE
      . An index to quantify an individual’s scientific research output. Proc Natl Acad Sci USA 2005;102:16569–72.doi:10.1073/pnas.0507655102
      OpenUrlAbstract/FREE Full Text
    20. 20.↵
      2. Bollen J ,
      3. Crandall D ,
      4. Junk D , et al
      . An efficient system to fund science: from proposal review to peer-to-peer distributions. Scientometrics 2017;110:521–8.doi:10.1007/s11192-016-2110-3
      OpenUrl
    21. 21.↵
      2. Finch A
      . Can we do better than existing author citation metrics? Bioessays 2010;32:744–7.doi:10.1002/bies.201000053
      OpenUrlPubMed
    22. 22.↵
      EndNote. Clarivate analytics, 2017.
    23. 23.↵
      2. Schlosser RW
      . Appraising the quality of systematic reviews. Focus: Technical Briefs 2007;17:1–8.
      OpenUrl
    24. 24.↵
      2. Braithwaite J ,
      3. Herkes J ,
      4. Ludlow K , et al
      . Association between organisational and workplace cultures, and patient outcomes: systematic review. BMJ Open 2017;7:e017708.doi:10.1136/bmjopen-2017-017708
    25. 25.↵
      2. Landis JR ,
      3. Koch GG
      . The measurement of observer agreement for categorical data. Biometrics 1977;33:159–74.doi:10.2307/2529310
      OpenUrlCrossRefPubMedWeb of Science
    26. 26.↵
      2. Shamseer L ,
      3. Moher D ,
      4. Clarke M , et al
      . Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ 2015;349:g7647.doi:10.1136/bmj.g7647
      OpenUrlAbstract/FREE Full Text
    27. 27.↵
      2. Hawker S ,
      3. Payne S ,
      4. Kerr C , et al
      . Appraising the evidence: reviewing disparate data systematically. Qual Health Res 2002;12:1284–99.doi:10.1177/1049732302238251
      OpenUrlCrossRefPubMedWeb of Science
    28. 28.↵
      2. Thorngate W ,
      3. Chowdhury W
      . By the numbers: track record, flawed reviews, journal space, and the fate of talented authors. In: Kaminski B , Koloch G , eds. Advances in Social Simulation: Proceedings of the 9th Conference of the European Social Simulation Association. Advances in Intelligent Systems and Computing. 229. Heidelberg, Germany: Springer Berlin, 2014:177–88.
    29. 29.↵
      2. Sood A ,
      3. Therattil PJ ,
      4. Chung S , et al
      . Impact of subspecialty fellowship training on research productivity among academic plastic surgery faculty in the United States. Eplasty 2015;15:e50.
    30. 30.↵
      2. Mutz R ,
      3. Bornmann L ,
      4. Daniel H-D
      . Testing for the fairness and predictive validity of research funding decisions: a multilevel multiple imputation for missing data approach using ex-ante and ex-post peer evaluation data from the Austrian science fund. J Assoc Inf Sci Technol 2015;66:2321–39.doi:10.1002/asi.23315
      OpenUrlCrossRef
    31. 31.↵
      2. Rezek I ,
      3. McDonald RJ ,
      4. Kallmes DF
      . Pre-residency publication rate strongly predicts future academic radiology potential. Acad Radiol 2012;19:632–4.doi:10.1016/j.acra.2011.11.017
      OpenUrlPubMed
    32. 32.↵
      2. Knudson D
      . Kinesiology faculty citations across academic rank. Quest 2015;67:346–51.doi:10.1080/00336297.2015.1082144
      OpenUrl
    33. 33.↵
      2. Wang D ,
      3. Song C ,
      4. Barabási AL
      . Quantifying long-term scientific impact. Science 2013;342:127–32.doi:10.1126/science.1237825
      OpenUrlAbstract/FREE Full Text
    34. 34.↵
      2. Efron N ,
      3. Brennan NA
      . Citation analysis of Australia-trained optometrists. Clin Exp Optom 2011;94:600–5.doi:10.1111/j.1444-0938.2011.00652.x
      OpenUrlPubMed
    35. 35.↵
      2. Perlin MS ,
      3. Santos AAP ,
      4. Imasato T , et al
      . The Brazilian scientific output published in journals: a study based on a large CV database. J Informetr 2017;11:18–31.doi:10.1016/j.joi.2016.10.008
      OpenUrl
    36. 36.↵
      2. Stallings J ,
      3. Vance E ,
      4. Yang J , et al
      . Determining scientific impact using a collaboration index. Proc Natl Acad Sci USA 2013;110:9680–5.doi:10.1073/pnas.1220184110
      OpenUrlAbstract/FREE Full Text
    37. 37.↵
      2. Kreiman G ,
      3. Maunsell JH
      . Nine criteria for a measure of scientific output. Front Comput Neurosci 2011;5.doi:10.3389/fncom.2011.00048
    38. 38.↵
      2. Mingers J
      . Measuring the research contribution of management academics using the Hirsch-index. J Oper Res Soc 2009;60:1143–53.doi:10.1057/jors.2008.94
      OpenUrlCrossRefWeb of Science
    39. 39.↵
      2. Halvorson MA ,
      3. Finlay AK ,
      4. Cronkite RC , et al
      . Ten-year publication trajectories of health services research career development award recipients: collaboration, awardee characteristics, and productivity correlates. Eval Health Prof 2016;39:49–64.doi:10.1177/0163278714542848
      OpenUrlCrossRefPubMed
    40. 40.↵
      2. Stroebe W
      . The graying of academia: will it reduce scientific productivity? Am Psychol 2010;65:660–73.doi:10.1037/a0021086
      OpenUrlCrossRefPubMed
    41. 41.↵
      2. Agarwal A ,
      3. Durairajanayagam D ,
      4. Tatagari S , et al
      . Bibliometrics: tracking research impact by selecting the appropriate metrics. Asian J Androl 2016;18:296–309.doi:10.4103/1008-682X.171582
      OpenUrl
    42. 42.↵
      2. Jacob JH ,
      3. Lehrl S ,
      4. Henkel AW
      . Early recognition of high quality researchers of the German psychiatry by worldwide accessible bibliometric indicators. Scientometrics 2007;73:117–30.doi:10.1007/s11192-006-1729-x
      OpenUrl
    43. 43.↵
      2. Minasny B ,
      3. Hartemink AE ,
      4. McBratney A , et al
      . Citations and the h index of soil researchers and journals in the Web of Science, Scopus, and Google Scholar. Peer J 2013;1:1.doi:10.7717/peerj.183
    44. 44.↵
      2. Gorraiz J ,
      3. Gumpenberger C
      . Going beyond citations: SERUM — a new tool provided by a network of libraries. LIBER Quarterly 2010;20:80–93.doi:10.18352/lq.7978
      OpenUrlCrossRef
    45. 45.↵
      2. van Eck NJ ,
      3. Waltman L ,
      4. van Raan AF , et al
      . Citation analysis may severely underestimate the impact of clinical research as compared to basic research. PLoS One 2013;8:e62395.doi:10.1371/journal.pone.0062395
      OpenUrlCrossRefPubMed
    46. 46.↵
      2. Meho LI ,
      3. Rogers Y
      . Citation counting, citation ranking, and h -index of human-computer interaction researchers: a comparison of Scopus and Web of Science. J Assoc Inf Sci Technol 2008;59:1711–26.doi:10.1002/asi.20874
      OpenUrl
    47. 47.↵
      2. Selek S ,
      3. Saleh A
      . Use of h index and g index for American academic psychiatry. Scientometrics 2014;99:541–8.doi:10.1007/s11192-013-1204-4
      OpenUrl
    48. 48.↵
      2. Kali A
      . Scientific impact and altmetrics. Indian J Pharmacol 2015;47:570–1.doi:10.4103/0253-7613.165184
      OpenUrl
    49. 49.↵
      2. Neylon C ,
      3. Wu S
      . Article-level metrics and the evolution of scientific impact. PLoS Biol 2009;7:e1000242.doi:10.1371/journal.pbio.1000242
      OpenUrlCrossRefPubMed
    50. 50.↵
      2. Sahel JA
      . Quality versus quantity: assessing individual research performance. Sci Transl Med 2011;3:84cm13.doi:10.1126/scitranslmed.3002249
      OpenUrlFREE Full Text
    51. 51.↵
      2. Pinnock D ,
      3. Whittingham K ,
      4. Hodgson LJ
      . Reflecting on sharing scholarship, considering clinical impact and impact factor. Nurse Educ Today 2012;32:744–6.doi:10.1016/j.nedt.2012.05.031
      OpenUrlPubMed
    52. 52.↵
      2. Eyre-Walker A ,
      3. Stoletzki N
      . The assessment of science: the relative merits of post-publication review, the impact factor, and the number of citations. PLoS Biol 2013;11:e1001675.doi:10.1371/journal.pbio.1001675
      OpenUrlCrossRefPubMed
    53. 53.↵
      2. Ferrer-Sapena A ,
      3. Sánchez-Pérez EA ,
      4. Peset F , et al
      . The Impact Factor as a measuring tool of the prestige of the journals in research assessment in mathematics. Res Eval 2016;25:306–14.doi:10.1093/reseval/rvv041
      OpenUrl
    54. 54.↵
      2. Moustafa K
      . Aberration of the citation. Account Res 2016;23:230–44.doi:10.1080/08989621.2015.1127763
      OpenUrl
    55. 55.↵
      2. Abramo G ,
      3. D’Angelo CA
      . Refrain from adopting the combination of citation and journal metrics to grade publications, as used in the Italian national research assessment exercise (VQR 2011–2014). Scientometrics 2016;109:2053–65.doi:10.1007/s11192-016-2153-5
      OpenUrl
    56. 56.↵
      2. Páll-Gergely B
      . On the confusion of quality with impact: a note on Pyke’s M-Index. BioScience 2015;65:117.doi:10.1093/biosci/biu207
    57. 57.↵
      2. Niederkrotenthaler T ,
      3. Dorner TE ,
      4. Maier M
      . Development of a practical tool to measure the impact of publications on the society based on focus group discussions with scientists. BMC Public Health 2011;11:588.doi:10.1186/1471-2458-11-588
      OpenUrlCrossRefPubMed
    58. 58.↵
      2. Kreines EM ,
      3. Kreines MG
      . Control model for the alignment of the quality assessment of scientific documents based on the analysis of content-related context. J Comput Syst Sci 2016;55:938–47.doi:10.1134/S1064230716050099
      OpenUrl
    59. 59.↵
      2. DiBartola SP ,
      3. Hinchcliff KW
      . Metrics and the scientific literature: deciding what to read. J Vet Intern Med 2017;31:629–32.doi:10.1111/jvim.14732
      OpenUrl
    60. 60.↵
      2. Szymanski BK ,
      3. de la Rosa JL ,
      4. Krishnamoorthy M
      . An internet measure of the value of citations. Inf Sci 2012;185:18–31.doi:10.1016/j.ins.2011.08.005
      OpenUrl
    61. 61.↵
      2. Bloching PA ,
      3. Heinzl H
      . Assessing the scientific relevance of a single publication over time. S Afr J Sci 2013;109:1–2.doi:10.1590/sajs.2013/20130063
      OpenUrlCrossRef
    62. 62.↵
      2. Benchimol-Barbosa PR ,
      3. Ribeiro RL ,
      4. Barbosa EC
      . Additional comments on the paper by Thomas et al: how to evaluate "quality of publication". Arq Bras Cardiol 2011;97:88–9.
      OpenUrlPubMed
    63. 63.↵
      2. Slim K ,
      3. Dupré A ,
      4. Le Roy B
      . Impact factor: an assessment tool for journals or for scientists? Anaesth Crit Care Pain Med 2017;36:347–8.doi:10.1016/j.accpm.2017.06.004
      OpenUrl
    64. 64.↵
      2. Diem A ,
      3. Wolter SC
      . The use of bliometrics to measure research performance in education sciences. Res High Educ 2013;54:86–114.doi:10.1007/s11162-012-9264-5
      OpenUrl
    65. 65.↵
      2. Bornmann L ,
      3. Leydesdorff L
      . Does quality and content matter for citedness? A comparison with para-textual factors and over time. J Informetr 2015;9:419–29.doi:10.1016/j.joi.2015.03.001
      OpenUrl
    66. 66.↵
      2. Santangelo GM
      . Article-level assessment of influence and translation in biomedical research. Mol Biol Cell 2017;28:1401–8.doi:10.1091/mbc.e16-01-0037
      OpenUrlAbstract/FREE Full Text
    67. 67.↵
      2. Ravenscroft J ,
      3. Liakata M ,
      4. Clare A , et al
      . Measuring scientific impact beyond academia: an assessment of existing impact metrics and proposed improvements. PLoS One 2017;12:e0173152.doi:10.1371/journal.pone.0173152
      OpenUrl
    68. 68.↵
      2. Trueger NS ,
      3. Thoma B ,
      4. Hsu CH , et al
      . The altmetric score: a new measure for article-level dissemination and impact. Ann Emerg Med 2015;66:549–53.doi:10.1016/j.annemergmed.2015.04.022
      OpenUrl
    69. 69.↵
      2. Welk G ,
      3. Fischman MG ,
      4. Greenleaf C , et al
      . Editorial board position statement regarding the Declaration on research assessment (DORA) recommendations rith respect to journal impact factors. Res Q Exerc Sport 2014;85:429–30.doi:10.1080/02701367.2014.964104
      OpenUrl
    70. 70.↵
      2. Taylor DR ,
      3. Michael LM ,
      4. Klimo P
      . Not everything that matters can be measured and not everything that can be measured matters response. J Neurosurg 2015;123:544–5.
      OpenUrl
    71. 71.↵
      2. Christopher MM
      . Weighing the impact (factor) of publishing in veterinary journals. J Vet Cardiol 2015;17:77–82.doi:10.1016/j.jvc.2015.01.002
      OpenUrl
    72. 72.↵
      2. Jokic M
      . H-index as a new scientometric indicator. Biochem Med 2009;19:5–9.doi:10.11613/BM.2009.001
      OpenUrl
    73. 73.↵
      2. Bornmann L ,
      3. Pudovkin AI
      . The journal impact factor should not be discarded. J Korean Med Sci 2017;32:180–2.doi:10.3346/jkms.2017.32.2.180
      OpenUrl
    74. 74.↵
      2. Franceschini F ,
      3. Galetto M ,
      4. Maisano D , et al
      . The success-index: an alternative approach to the h-index for evaluating an individual’s research output. Scientometrics 2012;92:621–41.doi:10.1007/s11192-011-0570-z
      OpenUrl
    75. 75.↵
      2. Prathap G
      . Citation indices and dimensional homogeneity. Curr Sci 2017;113:853–5.
      OpenUrl
    76. 76.↵
      2. Saad G
      . Applying the h-index in exploring bibliometric properties of elite marketing scholars. Scientometrics 2010;83:423–33.doi:10.1007/s11192-009-0069-z
      OpenUrl
    77. 77.↵
      2. Duffy RD ,
      3. Jadidian A ,
      4. Webster GD , et al
      . The research productivity of academic psychologists: assessment, trends, and best practice recommendations. Scientometrics 2011;89:207–27.doi:10.1007/s11192-011-0452-4
      OpenUrlCrossRefWeb of Science
    78. 78.↵
      2. Prathap G
      . Evaluating journal performance metrics. Scientometrics 2012;92:403–8.doi:10.1007/s11192-012-0746-1
      OpenUrl
    79. 79.↵
      2. Lando T ,
      3. Bertoli-Barsotti L
      . A new bibliometric index based on the shape of the citation distribution. PLoS One 2014;9:e115962.doi:10.1371/journal.pone.0115962
      OpenUrl
    80. 80.↵
      2. Bornmann L ,
      3. Wallon G ,
      4. Ledin A
      . Is the h index related to (standard) bibliometric measures and to the assessments by peers? An investigation of the h index by using molecular life sciences data. Res Eval 2008;17:149–56.doi:10.3152/095820208X319166
      OpenUrl
    81. 81.↵
      2. Pepe A ,
      3. Kurtz MJ
      . A measure of total research impact independent of time and discipline. PLoS One 2012;7:e46428.doi:10.1371/journal.pone.0046428
    82. 82.↵
      2. Haslam N ,
      3. Laham S
      . Early-career scientific achievement and patterns of authorship: the mixed blessings of publication leadership and collaboration. Res Eval 2009;18:405–10.doi:10.3152/095820209X481075
      OpenUrl
    83. 83.↵
      2. Ioannidis JP ,
      3. Klavans R ,
      4. Boyack KW
      . Multiple citation indicators and their composite across scientific disciplines. PLoS Biol 2016;14:e1002501.doi:10.1371/journal.pbio.1002501
      OpenUrl
    84. 84.↵
      2. van Leeuwen T
      . Testing the validity of the Hirsch-index for research assessment purposes. Res Eval 2008;17:157–60.doi:10.3152/095820208X319175
      OpenUrl
    85. 85.↵
      2. Ouimet M ,
      3. Bédard PO ,
      4. Gélineau F
      . Are the h-index and some of its alternatives discriminatory of epistemological beliefs and methodological preferences of faculty members? The case of social scientists in Quebec. Scientometrics 2011;88:91–106.doi:10.1007/s11192-011-0364-3
      OpenUrlPubMed
    86. 86.↵
      2. Kshettry VR ,
      3. Benzel EC
      . Research productivity and fellowship training in neurosurgery. World Neurosurg 2013;80:787–8.doi:10.1016/j.wneu.2013.10.005
      OpenUrl
    87. 87.↵
      2. Biswal AK
      . An absolute index (Ab-index) to measure a researcher’s useful contributions and productivity. PLoS One 2013;8:e84334.doi:10.1371/journal.pone.0084334
      OpenUrl
    88. 88.↵
      2. Tschudy MM ,
      3. Rowe TL ,
      4. Dover GJ , et al
      . Pediatric academic productivity: Pediatric benchmarks for the h- and g-indices. J Pediatr 2016;169:272–6.doi:10.1016/j.jpeds.2015.10.030
      OpenUrl
    89. 89.↵
      2. Azer SA ,
      3. Azer S
      . Bibliometric analysis of the top-cited gastroenterology and hepatology articles. BMJ Open 2016;6:e009889.doi:10.1136/bmjopen-2015-009889
      OpenUrlAbstract/FREE Full Text
    90. 90.↵
      2. Joshi MA
      . Bibliometric indicators for evaluating the quality of scientifc publications. J Contemp Dent Pract 2014;15:258–62.doi:10.5005/jp-journals-10024-1525
      OpenUrl
    91. 91.↵
      2. Danielson J ,
      3. McElroy S
      . Quantifying published scholarly works of experiential education directors. Am J Pharm Educ 2013;77:167.doi:10.5688/ajpe778167
      OpenUrl
    92. 92.↵
      2. Ion D ,
      3. Andronic O ,
      4. Bolocan A , et al
      . Tendencies on traditional metrics. Chirurgia 2017;112:117–23.doi:10.21614/chirurgia.112.2.117
      OpenUrl
    93. 93.↵
      2. Suiter AM ,
      3. Moulaison HL
      . Supporting scholars: An analysis of academic library websites' documentation on metrics and impact. J Acad Librariansh 2015;41:814–20.doi:10.1016/j.acalib.2015.09.004
      OpenUrl
    94. 94.↵
      2. Butler JS ,
      3. Kaye ID ,
      4. Sebastian AS , et al
      . The evolution of current research impact metrics: From bibliometrics to altmetrics? Clin Spine Surg 2017;30:226–8.doi:10.1097/BSD.0000000000000531
      OpenUrl
    95. 95.↵
      2. Krapivin M ,
      3. Marchese M ,
      4. Casati F
      . Exploring and understanding scientific metrics in citation networks. Zhou J , ed. Complex Sciences, Pt 2. Lecture Notes of the Institute for Computer Sciences Social Informatics and Telecommunications Engineering. 2009 5:1550–63.
      OpenUrl
    96. 96.↵
      2. Carpenter TA
      . Comparing digital apples to digital apples: background on NISO’s effort to build an infrastructure for new forms of scholarly assessment. Inf Serv Use 2014;34:103–6.doi:10.3233/ISU-140739
      OpenUrl
    97. 97.↵
      2. Gasparyan AY ,
      3. Nurmashev B ,
      4. Yessirkepov M , et al
      . The journal impact factor: moving toward an alternative and combined scientometric approach. J Korean Med Sci 2017;32:173–9.doi:10.3346/jkms.2017.32.2.173
      OpenUrl
    98. 98.↵
      2. Moed HF ,
      3. Halevi G
      . Multidimensional assessment of scholarly research impact. J Assoc Inf Sci Technol 2015;66:1988–2002.doi:10.1002/asi.23314
      OpenUrl
    99. 99.↵
      2. Chuang K-Y ,
      3. Olaiya MT
      . Bibliometric analysis of the Polish Journal of Environmental Studies (2000-11). Pol J Environ Stud 2012;21:1175–83.
      OpenUrl
    100. 100.↵
      2. Vinyard M
      . Altmetrics: an overhyped fad or an important tool for evaluating scholarly output? Computers in Libraries 2016;36:26–9.
      OpenUrl
    101. 101.↵
      2. van Noorden R
      . A profusion of measures. Nature 2010;465:864–6.
      OpenUrlCrossRefPubMedWeb of Science
    102. 102.↵
      2. Van Noorden R
      . Love thy lab neighbour. Nature 2010;468:1011.doi:10.1038/4681011a
    103. 103.↵
      2. Dinsmore A ,
      3. Allen L ,
      4. Dolby K
      . Alternative perspectives on impact: the potential of ALMs and altmetrics to inform funders about research impact. PLoS Biol 2014;12:e1002003.doi:10.1371/journal.pbio.1002003
      OpenUrlCrossRefPubMed
    104. 104.↵
      2. Cress PE
      . Using altmetrics and social media to supplement impact factor: maximizing your article’s academic and societal impact. Aesthet Surg J 2014;34:1123–6.doi:10.1177/1090820X14542973
      OpenUrlCrossRefPubMed
    105. 105.↵
      2. Moreira JA ,
      3. Zeng XH ,
      4. Amaral LA
      . The distribution of the asymptotic number of citations to sets of publications by a researcher or from an academic department are consistent with a discrete lognormal model. PLoS One 2015;10:e0143108.doi:10.1371/journal.pone.0143108
    106. 106.↵
      2. Waljee JF
      . Discussion: are quantitative measures of academic productivity correlated with academic rank in plastic surgery? A National Study. Plast Reconstr Surg 2015;136:622–3.doi:10.1097/PRS.0000000000001566
      OpenUrlCrossRefPubMed
    107. 107.↵
      2. Fazel S ,
      3. Wolf A
      . What is the impact of a research publication? Evid Based Ment Health 2017;20:33–4.doi:10.1136/eb-2017-102668
      OpenUrlAbstract/FREE Full Text
    108. 108.↵
      2. Eysenbach G
      . Can tweets predict citations? Metrics of social impact based on Twitter and correlation with traditional metrics of scientific impact. J Med Internet Res 2011;13:e123.doi:10.2196/jmir.2012
      OpenUrlCrossRefPubMed
    109. 109.↵
      2. Hoffmann CP ,
      3. Lutz C ,
      4. Meckel M
      . Impact factor 2.0: applying social network analysis to scientific impact assessment. 2014 47th Hawaii International Conference on System Sciences. Proceedings of the Annual Hawaii International Conference on System Sciences, 2014:1576–85.
    110. 110.↵
      2. Maggio LA ,
      3. Meyer HS ,
      4. Artino AR
      . Beyond citation rates: a real-time impact analysis of health professions education research using altmetrics. Acad Med 2017;92:1449–55.doi:10.1097/ACM.0000000000001897
      OpenUrl
    111. 111.↵
      2. Raj A ,
      3. Carr PL ,
      4. Kaplan SE , et al
      . Longitudinal analysis of gender differences in academic productivity among medical faculty across 24 medical schools in the United States. Acad Med 2016;91:1074–9.doi:10.1097/ACM.0000000000001251
      OpenUrl
    112. 112.↵
      2. Markel TA ,
      3. Valsangkar NP ,
      4. Bell TM , et al
      . Endangered academia: preserving the pediatric surgeon scientist. J Pediatr Surg 2017;52:1079–83.doi:10.1016/j.jpedsurg.2016.12.006
      OpenUrl
    113. 113.↵
      2. Mirnezami SR ,
      3. Beaudry C ,
      4. Larivière V
      . What determines researchers’ scientific impact? A case study of Quebec researchers. Sci Public Policy 2016;43:262–74.doi:10.1093/scipol/scv038
      OpenUrlCrossRef
    114. 114.↵
      2. Napolitano LM
      . Scholarly activity requirements for critical care fellowship program directors: what should it be? How should we measure it? Crit Care Med 2016;44:2293–6.doi:10.1097/CCM.0000000000002120
      OpenUrl
    115. 115.↵
      2. Bai X ,
      3. Xia F ,
      4. Lee I , et al
      . Identifying anomalous citations for objective evaluation of scholarly article impact. PLoS One 2016;11:e0162364.doi:10.1371/journal.pone.0162364
    116. 116.↵
      2. Gao C ,
      3. Wang Z ,
      4. Li X , et al
      . PR-Index: Using the h-index and PageRank for determining true impact. PLoS One 2016;11:e0161755.doi:10.1371/journal.pone.0161755
    117. 117.↵
      2. Assimakis N ,
      3. Adam M
      . A new author’s productivity index: p-index. Scientometrics 2010;85:415–27.doi:10.1007/s11192-010-0255-z
      OpenUrl
    118. 118.↵
      2. Petersen AM ,
      3. Succi S
      . The Z-index: A geometric representation of productivity and impact which accounts for information in the entire rank-citation profile. J Informetr 2013;7:823–32.doi:10.1016/j.joi.2013.07.003
      OpenUrlCrossRef
    119. 119.↵
      2. Claro J ,
      3. Costa CAV
      . A made-to-measure indicator for cross-disciplinary bibliometric ranking of researchers performance. Scientometrics 2011;86:113–23.doi:10.1007/s11192-010-0241-5
      OpenUrl
    120. 120.↵
      2. Sahoo BK ,
      3. Singh R ,
      4. Mishra B , et al
      . Research productivity in management schools of India during 1968-2015: A directional benefit-of-doubt model analysis. Omega 2017;66:118–39.doi:10.1016/j.omega.2016.02.004
      OpenUrl
    121. 121.↵
      2. Aragón AM
      . A measure for the impact of research. Sci Rep 2013;3:1649.doi:10.1038/srep01649
      OpenUrl
    122. 122.↵
      2. Shibayama S ,
      3. Baba Y
      . Impact-oriented science policies and scientific publication practices: the case of life sciences in Japan. Res Policy 2015;44:936–50.doi:10.1016/j.respol.2015.01.012
      OpenUrl
    123. 123.↵
      2. Tijdink JK ,
      3. Schipper K ,
      4. Bouter LM , et al
      . How do scientists perceive the current publication culture? A qualitative focus group interview study among Dutch biomedical researchers. BMJ Open 2016;6:e008681.doi:10.1136/bmjopen-2015-008681
      OpenUrlAbstract/FREE Full Text
    124. 124.↵
      2. Crespo JA ,
      3. Li Y ,
      4. Li Y , et al
      . The measurement of the effect on citation inequality of differences in citation practices across scientific fields. PLoS One 2013;8.doi:10.1371/annotation/d7b4f0c9-8195-45de-bee5-a83a266857fc
    125. 125.↵
      2. Teixeira da Silva JA ,
      3. da Silva JAT
      . Does China need to rethink its metrics- and citation-based research rewards policies? Scientometrics 2017;112:1853–7.doi:10.1007/s11192-017-2430-y
      OpenUrl
    126. 126.↵
      2. Devos P
      . Research and bibliometrics: a long history…. Clin Res Hepatol Gastroenterol 2011;35:336–7.doi:10.1016/j.clinre.2011.04.008
      OpenUrlPubMed
    127. 127.↵
      2. Slyder JB ,
      3. Stein BR ,
      4. Sams BS , et al
      . Citation pattern and lifespan: a comparison of discipline, institution, and individual. Scientometrics 2011;89:955–66.doi:10.1007/s11192-011-0467-x
      OpenUrlCrossRefWeb of Science
    128. 128.↵
      2. Zhou Y-B ,
      3. Lu L ,
      4. Li M
      . Quantifying the influence of scientists and their publications: distinguishing between prestige and popularity. New J Phys 2012;14:033033.doi:10.1088/1367-2630/14/3/033033
      OpenUrl
    129. 129.↵
      2. Sorensen AA ,
      3. Weedon D
      . Productivity and impact of the top 100 cited Parkinson’s disease investigators since 1985. J Parkinsons Dis 2011;1:3–13.doi:10.3233/JPD-2011-10021
      OpenUrl
    130. 130.↵
      2. Jeang KT
      . H-index, mentoring-index, highly-cited and highly-accessed: how to evaluate scientists? Retrovirology 2008;5:106.doi:10.1186/1742-4690-5-106
      OpenUrlCrossRefPubMed
    131. 131.↵
      2. Franceschini F ,
      3. Maisano D
      . Publication and patent analysis of European researchers in the field of production technology and manufacturing systems. Scientometrics 2012;93:89–100.doi:10.1007/s11192-012-0648-2
      OpenUrl
    132. 132.↵
      2. Sibbald SL ,
      3. MacGregor JC ,
      4. Surmacz M , et al
      . Into the gray: a modified approach to citation analysis to better understand research impact. J Med Libr Assoc 2015;103:49–54.doi:10.3163/1536-5050.103.1.010
      OpenUrl
    133. 133.↵
      2. Sutherland WJ ,
      3. Goulson D ,
      4. Potts SG , et al
      . Quantifying the impact and relevance of scientific research. PLoS One 2011;6:e27537.doi:10.1371/journal.pone.0027537
    134. 134.↵
      2. Team NE
      . Announcement: Nature journals support the San Francisco Declaration on Research Assessment. Nature 2017;544:394.doi:10.1038/nature.2017.21882
    135. 135.↵
      2. Pugh EN ,
      3. Gordon SE
      . Embracing the principles of the San Francisco Declaration of Research Assessment: Robert Balaban’s editorial. J Gen Physiol 2013;142:175.doi:10.1085/jgp.201311077
      OpenUrlFREE Full Text
    136. 136.↵
      2. Zhang L ,
      3. Rousseau R ,
      4. Sivertsen G
      . Science deserves to be judged by its contents, not by its wrapping: Revisiting Seglen’s work on journal impact and research evaluation. PLoS One 2017;12:e0174205.doi:10.1371/journal.pone.0174205
    137. 137.↵
      DORA—ASCB San Francisco, US . San Francisco Declaration on Research Assessment (DORA). 2016 http://www.ascb.org/dora/.
    138. 138.↵
      2. Cabezas-Clavijo A ,
      3. Delgado-López-Cózar E
      . [Google Scholar and the h-index in biomedicine: the popularization of bibliometric assessment]. Med Intensiva 2013;37:343–54.doi:10.1016/j.medin.2013.01.008
      OpenUrl
    139. 139.↵
      2. Iyengar R ,
      3. Wang Y ,
      4. Chow J , et al
      . An integrated approach to evaluate faculty members' research performance. Acad Med 2009;84:1610–6.doi:10.1097/ACM.0b013e3181bb2364
      OpenUrlCrossRefPubMed
    140. 140.↵
      2. Jacsó P
      . Eigenfactor and article influence scores in the Journal Citation Reports . Online Information Review 2010;34:339–48.doi:10.1108/14684521011037034
      OpenUrl

    Footnotes

    • Contributors JB conceptualised and drafted the manuscript, revised it critically for important intellectual content and led the study. JH, KC and JCL made substantial contributions to the design, analysis and revision of the work and critically reviewed the manuscript for important intellectual content. CP, CB, MB, RC-W, FR, PS, AH, LAE, KL, EA, RS and EM carried out the initial investigation, sourced and analysed the data and revised the manuscript for important intellectual content. PDH and JW critically commented on the manuscript, contributed to the revision and editing of the final manuscript and reviewed the work for important intellectual content. All authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.

    • Funding The work on which this paper is based was funded by the Australian National Health and Medical Research Council (NHMRC) for work related to an assessment of its peer-review processes being conducted by the Council. Staff of the Australian Institute of Health Innovation undertook this systematic review for Council as part of that assessment.

    • Disclaimer Other than specifying what they would like to see from a literature review, NHMRC had no role in the conduct of the systematic review or the decision to publish.

    • Competing interests None declared.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Data sharing statement All data have been made available as appendices.

    • Patient consent for publication Not required.