Article Text

Download PDFPDF

Original research
Overall and COVID-19-specific citation impact of highly visible COVID-19 media experts: bibliometric analysis
  1. John P Ioannidis1,2,
  2. Alangoya Tezel3,
  3. Reshma Jagsi4
  1. 1Meta-Resarch Innovation Center at Stanford (METRICS), Stanford University, Stanford, California, USA
  2. 2Departments of Medicine, of Epidemiology and Population Health, and of Statistics, Stanford University, Stanford, California, USA
  3. 3University of Michigan, Ann Arbor, Michigan, USA
  4. 4Center for Bioethics and Social Sciences in Medicine, University of Michigan, Ann Arbor, Michigan, USA
  1. Correspondence to Dr John P Ioannidis; jioannid{at}stanford.edu

Abstract

Objective To evaluate whether the COVID-19 experts who appear most frequently in media have high citation impact for their research overall, and for their COVID-19 peer-reviewed publications in particular and to examine the representation of women among such experts.

Design Cross-linking of data sets of most highly visible COVID-19 media experts with citation data on the impact of their published work (career-long publication record and COVID-19-specific work).

Setting Cable news appearance in prime-time programming or overall media appearances.

Participants Most highly visible COVID-19 media experts in the USA, Switzerland, Greece and Denmark.

Interventions None.

Outcome measures Citation data from Scopus along with discipline-specific ranks of overall career-long and COVID-19-specific impact based on a previously validated composite citation indicator.

Results We assessed 76 COVID-19 experts who were highly visible in US prime-time cable news, and 50, 12 and 2 highly visible experts in media in Denmark, Greece and Switzerland, respectively. Of those, 23/76, 10/50, 2/12 and 0/2 were among the top 2% of overall citation impact among scientists in the same discipline worldwide. Moreover, 37/76, 15/50, 7/12 and 2/2 had published anything on COVID-19 that was indexed in Scopus as of 30 August 2021. Only 18/76, 6/50, 2/12 and 0/2 of the highly visible COVID-19 media experts were women. 55 scientists in the USA, 5 in Denmark, 64 in Greece and 56 in Switzerland had a higher citation impact for their COVID-19 work than any of the evaluated highly visible media COVID-19 experts in the respective country; 10/55, 2/5, 22/64 and 14/56 of them were women.

Conclusions Despite notable exceptions, there is a worrisome disconnect between COVID-19 claimed media expertise and scholarship. Highly cited women COVID-19 experts are rarely included among highly visible media experts.

  • COVID-19
  • public health
  • medical journalism

Data availability statement

Data are available upon reasonable request. All the data are in the manuscript and tables and additional details on citation data are available in publicly deposited data sets in Mendeley.

http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

Statistics from Altmetric.com

Strengths and limitations of this study

  • We examined the citation impact in the scientific literature of highly visible COVID-19 media experts in four different countries (USA, Denmark, Greece and Switzerland).

  • We also examined whether these highly visible media experts had published anything on COVID-19.

  • We identified women experts who have contributed with high impact in the COVID-19 literature but have not been among these highly visible media experts.

  • The findings need to be extrapolated cautiously in other countries and other media (eg, social media).

  • Most scientists may not wish to be visible in media, but the disconnect between COVID-19 claimed media expertise and scholarship is worrisome.

Introduction

The COVID-19 pandemic has been accompanied by an unprecedented infodemic in the news and social media.1 2 Media coverage has been intensive, continuous, massive and heated and has involved a very large number of alleged experts. The involvement of knowledgeable scholars in the public discussion and dissemination of information on such a monumental crisis is clearly welcome and indispensable. However, how knowledgeable are the experts recruited by media?

Knowledge and expertise are difficult to appraise with full objectivity. Weinstein3 argued that there are two kinds of experts, those who are recognised as experts based on what they know (epistemic expertise) and those who are worthy of being called experts based on what they do (performative expertise). According to this classification, an epistemic expert is a person who is capable of providing strong justification for a range of claims in a domain, while performative expertise characterises a person who is able to perform a skill well according to the rules and virtues of a practice.3 Performative experts may not necessarily be contributors to the scientific literature themselves, but may still know their job well and have extensive practical experience. It is very difficult, however, to appraise in a standardised manner and with consistency and quantitative metrics such performative expertise. Conversely, epistemic experts are likely to be contributors to the scientific literature and their level of contribution and impact in the science of their field is a key hallmark of their expertise. What can be readily appraised in a non-subjective fashion is the publication and citation track record of scientists who appear in news media as experts. One can use objective data to quantify the citation impact of the published work of these scientists across science throughout their career, as well as the specific impact that they are having with their scientific publications about COVID-19. While publications and citations do have limitations (as all bibliometric metrics), they are objective, readily quantifiable and offer useful information about scientific impact. Here we aimed to evaluate the overall and COVID-19-specific citation impact of the most highly visible COVID-19 experts in the USA, Denmark, Greece and Switzerland. We also paid particular attention to probing the representation of women among highly visible COVID-19 experts, as it has been previously suggested that women are under-represented among COVID-19 experts in the USA.4

Methods

Highly visible media COVID-19 experts

We examined bibliometric indicators of top media experts in the USA, Switzerland, Greece and Denmark. These are countries for which we could identify pre-existing lists of experts who had prominent visibility in media. These lists are typically not published in the peer-reviewed scientific literature (with the exception of the US list that was previously generated and published by members of our team),4 but in media news items in different countries, thus defying the possibility for efficient systematic searches. We therefore asked our colleagues at the Meta-Research Innovation Center at Stanford (and affiliated colleagues who come from different countries if they were aware of any such publicised lists. We accepted these lists regardless of how visibility had been defined in these surveys.

For the USA, we examined the scientific citation impact of all scientists who had appeared between 18 May and 19 June 2020 during prime-time programming on three popular American cable news networks: Fox News Network, CNN and MSNBC. Details on the data collection and selection for the US list and features of the sample have been previously described.4 Of the 220 people who appeared during these programmes, 76 were scientists (47 physicians and 29 PhDs).

For European countries, searches for visible experts were made by local organisations in each country and they pertain to national media visibility. For Denmark, we found a news article that listed the 50 experts who had the highest number of appearances in media during 2020 (television, radio, newspapers).5 For Greece, we found a news article that listed the 12 COVID-19 experts who had the highest television exposure based on measured time of television appearances.6 For Switzerland, the Swiss Media Database (SMD) captures appearances in media in Switzerland. We could find information from a news article7 on the two most commonly appearing names of COVID-19 experts in SMD between mid-January and June 2020.

Citation databases for overall impact of scientific work and COVID-19-specific work

For overall (career-long) impact of scientific work, we used a previously developed, publicly available data set8 that includes the top 2% of scientists across each of the 174 disciplines of science (classified according to the Science-Metrix classification).8 All ~8 million scientists who have published at least five Scopus-indexed full papers (counting articles, reviews and conference papers) are considered. The ranking uses a previously developed and validated composite citation indicator9 that merges six citation metrics (total citations, Hirsch h-index, coauthorship-adjusted Schreiber hm-index, citations to single-authored papers, citations to first-authored or single-authored papers and citations to single-authored, first-authored or last-authored papers).

We also examined which of the evaluated highly visible COVID-19 experts had published anything pertaining to COVID-19 in the scientific literature. We used a previously created database10 that includes all the authors with at least five Scopus-indexed full papers in their career (on any topic) who had also published at least one Scopus-indexed item on COVID-19 (peer reviewed or preprint) as of 1 March 2021. Details on the search strategy and retrieval of authors can be found in the paper describing the compilation of that database.10 In brief, the search string was: TITLE-ABS-KEY (sars-cov-2 OR ‘coronavirus 2’ OR ‘corona virus 2’ OR covid-19 OR {novel coronavirus} OR {novel corona virus} OR 2019-ncov OR covid OR covid19 OR ncovid-19 OR ‘coronavirus disease 2019’ OR ‘corona virus disease 2019’ OR corona-19 OR SARS-nCoV OR ncov-2019) with items limited to a publication date in 2020 or 2021. We have previously10 generated the same composite citation indicator (that includes the six citation metrics described above) limited to the citation impact of COVID-19 publications for each author as of 1 March 2021. We therefore noted how many of the highly visible COVID-19 experts were among the top 2000 or top 10 000 ranked scientists for the citation impact of their COVID-19 publications as of 1 March 2021. We also updated the searches on 30 August 2021 to see how many of the experts had published any COVID-19-related paper by then.

We tabulated the experts who are in the top 2% of citation impact for their career-long published work and concurrently are among the top 2% based on the citation impact of their COVID-19 published work in their primary scientific disciplines among other scientists with the same primary scientific discipline.

We also noted how many of the highly visible media COVID-19 experts were women in each country; how many scientists in each country had a higher citation impact for their COVID-19 published work than all the highly visible media COVID-19 experts in the respective country; and how many of these scientists with higher citation impact for their COVID-19 published work were women.

All citation metrics and rankings thereof exclude all self-citations.

Results

Experts in the USA

Of the 76 highly visible cable news COVID-19 expert scientists, only 18 were women. Only 23 of the 76 were in the top 2% of their main scientific discipline in terms of their citation impact during their careers until the end of 2019. The main disciplines of these 23 top-cited experts’ previous scientific work were general and internal medicine (n=7), economics (n=2), health policy and services (n=2), microbiology (n=2), public health (n=2) and 8 other disciplines (1 each). Thirteen appeared on MSNBC, nine on CNN and one on Fox News. Only 3 of the 23 were women.

Only 37 of the 76 COVID-19 scientists had published anything that was COVID-19 related by 30 August 2021, more than a year months after they appeared as COVID-19 experts in these major media. Using the same composite citation indicator focused specifically on the citation impact of their COVID-19 work, only seven were among the top 2000 scientists worldwide for COVID-19-related citation impact and 18 were among the top 10 000 scientists worldwide in this regard. Nineteen appeared on MSNBC, 13 on CNN and two on Fox News. Nine of the 34 were women.

Experts in Denmark

For Denmark, only 6 of the 50 top media experts were women. The most frequently appearing expert was Søren Brostrøm, director of the National Board of Health (9324 mentions, about 25 per day). Forty-eight of 50 were Danes and two were foreigners. Forty-three of 50 had commented on COVID-19, but it was not stated who are the seven who only commented on other non-COVID-19-related topics. Ten of the 50 (nine Danes, one foreigner) were among the 2% top cited for career-long scientific work. A perusal of their listed scientific subfield suggested that 33/50 worked in a biomedical or potentially related field; thus, apparently, at least 10 of the experts were from other fields, which included economics or law. Among the 50, only 15 (all of them among the 33 biomedical) had published any COVID-19-related work indexed in Scopus by 30 August 2021. Only three were among the top 10 000 scientists worldwide for COVID-19-related citation impact as of 1 March 2021 (Anthony Fauci, rank 224; Thomas Benfield, rank 4461; Lone Simonsen, rank 8307). Among the 48 visible experts who were Danes, the best ranked for published COVID-19 work (Thomas Benfield) was ranked sixth in COVID-19-related impact among scientists in Denmark.

Experts in Greece

For Greece, only 2 of the 12 most visible television COVID-19 experts were women. Ten of the 12 were local and two were in the diaspora (in UK and Switzerland) but appeared massively in Greek media. Two of the 12 (Elias Mossialos and Emmanouil Dermitzakis, both in the diaspora) were among the 2% top cited for career-long scientific work. Only 7 of the 12 had published any COVID-19-related work indexed in Scopus by 30 August 2021. Only one was among the top 10 000 scientists worldwide for COVID-19-related citation impact (Elias Mossialos, rank 4435). Among the 10 visible experts who were living in Greece, the best ranked for published COVID-19 work (Charalambos Gogos, worldwide rank 10 710) was ranked 65th in COVID-19-related impact among scientists in Greece.

Experts in Switzerland

The two most commonly appearing names of COVID-19 experts in SMD between mid-January and early June 2020 were Marcel Salathé (École Polytechnique Fédérale de Lausanne), with 1400 entries in SMD during this period (as opposed to nine entries in the entire 2019), and Christian Althaus (University of Bern) with approximately 700 entries. Neither of them have been in the top 2% of the most cited scientists for their career work across all scientific topics. They have both published scientific work related to COVID-19. For their COVID-19-related published work, they are ranked 58th and 57th among the scientists in Switzerland (ranks 3839 and 3819, respectively, worldwide among all scientists publishing on COVID-19).

Top cited on both COVID-19 and during overall career

Based on our prime-time US cable news sample, table 1 shows the experts who were top cited for their citation impact in the scientific literature during their overall career and also specifically for their work on COVID-19. Ezekiel Emanuel was the most highly ranked (ranked 227th among over 495 000 scientists)9 in global COVID-19 citation impact among our sample. There were 55 US authors with higher COVID-19 citation impact. Among those 55, some such as Bill Gates (global COVID-19 citation ranking 212th) and Anthony Fauci (global COVID-19 citation ranking 224th) have attracted substantial media attention but simply did not appear in our prime-time programming sample. For example, there were numerous video clippings and references to Dr Fauci that aired on the prime-time programmes that we screened, but he was not personally interviewed in the 5 weeks studied here. However, it is likely that the majority of the most influential scientists on COVID-19 research have not appeared prominently in the lay media and many of them are probably entirely absent.

Table 1

Some outstanding scientists with media presence (those of the 76 with prime-time appearance in CNN, MSNBC or Fox News on 18 May to 19 June 2020 who are at the top 2% of citation impact in their discipline both for all their work and for their COVID-19 work)

Our samples of massively visible experts from Switzerland and Greece were small, but only 1/12 visible experts (Elias Mossialos) was in the top 2% of both overall career-long and COVID-19-specific citation impact. For the Denmark sample, only Anthony Fauci (and no local Danish scientists) was in the top 2% of both COVID-19 and overall career-long citation impact.

Missing expert women with top citation impact

Among the 55 US authors with higher COVID-19 citation impact than all the 76 scientists in our US cable news sample, there were 10 women. They are shown along with their affiliations and their primary and secondary disciplines of expertise in table 2. We searched the CNN site (https://www.cnn.com/search?size=10&q=) and the Fox News site (https://www.foxnews.com/#) for videos in 2020 or 2021 up to 14 March 2021 (we found no way to search only for videos in MSNBC), that is, covering the entire period of the pandemic and all time slots, not just prime time. We found no stored videos with the name of any of these 10 female scientists, as compared with 272 and 32 400 videos, respectively, retrieved with a search for ‘Fauci’.

Table 2

Women scientists with higher citation impact of their COVID-19-related published work than all of the 76 experts who appeared in the analysed sample of US cable news prime time

In Denmark, Greece and Switzerland, 2/5, 22/64 and 14/56 scientists with higher COVID-19 citation impact ranking than the highest ranked massively visible news experts were women. These highly cited women scientists are shown along with their affiliations and their primary and secondary disciplines of expertise in table 3 for Denmark and Greece. For Switzerland where we only had the two most visible COVID-19 experts, we could not exclude that several of the 56 scientists with higher COVID-19 citation impact ranking may also had been highly visible in the media.

Table 3

Women scientists with higher citation impact of their COVID-19-related published work than all of the 50 most visible media experts in Denmark; or than all the 12 most visible television COVID-19 experts in Greece

Discussion

The present analysis suggests that only a minority of media-visible COVID-19 experts have had major scientific citation impact in their careers. Moreover, only a minority have any scientific record of any COVID-19-related publications. Highly visible COVID-19 media experts are very rarely influential in the scientific literature overall and/or in the COVID-19 scientific literature in particular. Women are markedly under-represented among those visible experts, although additional female experts exist. The under-representation of women in our examined samples was more prominent than what Fletcher et al found by analysing articles in 10 US newspapers in April 2020 where 34% of the authors were women.11 We could not assess the racial background of media experts, but we suspect that minorities would also be under-represented, as they are under-represented in many aspects of both academics and societal power structures.12 13

One other recent study has examined the scientific productivity of COVID-19 experts.14 Murayama et al assessed the 11 most frequently appearing medical experts in Japanese television during the first 6 months of 2020, 10 of which were men. They found that only one of the 11 experts had published a single scientific paper on COVID-19 indexed in PubMed as of 14 August 2020. The very low rate of publishing experts in this Japanese sample may be due to the fact that the search date for publications was too early. It is possible that some experts may publish COVID-19-related papers later and the same applies also to experts without COVID-19-related publications in our evaluation. Indeed, on 30 August 2021 we re-examined the publication record of the 11 Japanese experts and found that five (45%) had published at least one Scopus-indexed COVID-19-related paper until that time. By analysing data on payments from the pharmaceutical industry made in 2017, Murayama et al also found that 7 of the 11 experts had received payments from the pharmaceutical industry amounting to $317 324 for that single year. We did not assess potential financial conflicts in our study. However, certainly this is an important issue for all countries and it may often be difficult to ascertain in the absence of comprehensive payment databases that cover all potential financial conflicts for all scientists, not just clinicians.

The lack of sufficient representation of top scientists among the most visible experts in media may not be specific to COVID-19 and may affect all topics where science is invoked in public discourse. A preprinted analysis of experts in German media (no individual names were available) suggested that media coverage on the COVID-19 pandemic actually used experts with higher citation indicators compared with earlier pandemics.15 Past empirical evaluations of experts on various topics have shown that many of them, typically the majority, have not done any research themselves on the topics on which they pontificate.16–18 Concurrently, there is an increasing hunger for having more and more experts in popular media.19 20

The best or the most cited scientists should not necessarily be the ones who appear the most frequently in media. Some of the experts whom we analysed have accumulated a track record of massive media engagements that require an enormous commitment of time and psyche. Many highly competitive, excellent scientists would find it difficult or even impossible to pursue their scholarly work and have an intense media presence at the same time. Moreover, especially for COVID-19, polarisation, politics and an environment of conspiracy, mistrust, and public unrest and rage may have disincentivised many leading scientists from engaging with media. Women and minorities may feel even more disincentivised in this environment.

Nevertheless, communication with the wider public is an important mission of science, medicine and public health. Information on COVID-19 in media has been shown to be of questionable quality.1 2 Its quantity is clearly immense. The vast majority is produced and disseminated by people without any scientific training and with little or no self-reflection on their inadequacy to judge complex and rapidly evolving scientific concepts. It may be impossible to diminish the bulk of information, but at a minimum its quality should be improved. Engaging qualified experts may be critical in this regard.

Empirical studies show that non-experts are very poor at making predictions about COVID-19, and they are worse than experts—even though experts do not account sufficiently for uncertainty in their estimates and are therefore often also wrong.21 Both models and empirical data suggest that media can have an impact on the course of the pandemic22 and it can also affect mental health during its course.23 While there can be questions and concerns even about the media appearances of the best and most knowledgeable experts, media without involvement of scientific expertise is likely to be far worse.24 25

Of note, several of the highly visible media COVID-19 experts were probably invited in some capacity other than their research scholarship, for example, some scientists had political or administrative roles and others were front-line clinicians rather than academics. These aspects of non-research expertise are also very useful, and it could well be that all experts analysed here should be applauded for their willingness to engage and inform the wider public. Even if they lack focused research expertise on COVID-19, certainly many practising physicians and scientists may still elevate the discourse compared with people without any medical and scientific training. However, it is worrisome that ‘scientific experts’ in the news include so few of the scientists who have themselves made substantial scholarly contributions.

The current study has used diverse data from several countries for experts who are highly visible in media and has linked their profiles to objective data from bibliometric analyses. While methods for selecting experts are different in each country, the observed patterns seem to be consistent across countries. However, there are also several limitations that need to be discussed.

First, we should acknowledge that citation metrics are far from being perfect measures of epistemic expertise. Moreover, we focused on using already existing data on the 2% top-cited scientists across each scientific discipline, and we could not examine whether scientists who were not in the top 2% of these pre-existing lists might be in the top 3% or in the bottom 5% of citation impact. Obviously, many scientists may still have considerable epistemic expertise even if they are not strictly in the top 2% of citation indicators.

Second, our examined lists of media visible experts were precompiled independently of the current analysis. The precompilation had happened either by our team (in the case of US experts) or by news and media organisations in different countries and these compilations may use different criteria for identifying and ranking experts for visibility. It is possible that some different names might have entered these lists, if different visibility criteria had been used. Nevertheless, all of the experts analysed here had prominent media exposure and all analysed experts in European countries had massive media exposure in order to be able to reach such high ranks of visibility (even if variously defined).

We encourage media to look more carefully at the diversity and scholarly qualifications of the experts they invite, even more so for experts that have massive media appearances. Special attention should be given to inviting women, and our evaluation offers examples of many women scientists who might be considered in this regard. We suspect that many top experts may still wish to avoid media exposure and this should be respected. However, transparency and availability of opportunity are still important to ensure.

Data availability statement

Data are available upon reasonable request. All the data are in the manuscript and tables and additional details on citation data are available in publicly deposited data sets in Mendeley.

Ethics statements

Patient consent for publication

References

Footnotes

  • Contributors JPI had the original idea and wrote the first draft. JPI, AT and RJ brainstormed on the topic and revised the manuscript. JPI, AT and RJ approved the final paper. JPI is the guarantor of the article.

  • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

  • Competing interests JPI has given some COVID-19 media interviews (a hundredfold less compared with some of the listed experts) that have resulted in smearing, hate emails, threats, censoring, hostile behaviour, harassment and a life-threatening experience for a family member. He is among the top-cited scientists in both the overall and COVID-19-specific citation databases used in the presented analyses. In his Stanford web page, he admits that despite being ‘among the 10 scientists worldwide who are currently the most commonly cited; when contrasted against my vast ignorance, these values offer excellent proof that citation metrics can be horribly unreliable’. RJ is also listed among the top-cited scientists in the overall citation database.

  • Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

  • Provenance and peer review Not commissioned; externally peer reviewed.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.