1328 e-Letters

  • The Fallacy of Equating Citation Data with Scientific Qualkity or Impact

    The paper by Ioannidis is fatally flawed by the assumption that citation counts and Twitter activity correlate with scientific quality or health and policy impact. No hypothesis has been presented to support this. Opportunities to publish at scale will be influenced by length of career, and by other academic commitments. Reasons for high citation of papers may be to challenge or seek more clarity as much to commend or build on them. This point has been demonstrated in a peer reviewed paper which contrasted the examples of publication of contentious views gaining high citations, while a research finding which quickly got applied in practice had a single publication (1); the high-cited author eventually had his licence to practice withdrawn, while he who led to many lives being saved globally had no profile. Ioannidis seems to confuse noise and twittering with good grounding and integrity of evidence.

    (1) Rigby M. Citation Analysis in Health Care Sciences - Innovative Investigation or Seductive Pseudo-science?; Methods Inf Med 2014; 53(06): 459-463, DOI: 10.3414/ME14-05-0004

  • Reply to Gorski, Dahly, Pimenta and to second comments by Yamey and Bak-Coleman

    I thank Gorski, Dahly, and Pimenta for their criticism and Yamey and Bak-Coleman for their second round of comments. As already stated, I signed neither GBD nor JSM, my study did not aim to elevate or downgrade one or the other narrative, and I congratulate all GBD and JSM signatories. The 443 signatories from GBD include 4 scientists with whom I have co-authored, and 3 with Stanford affiliation. The respective first 443 signatories of JSM include 5 scientists with whom I have co-authored, and 15 with Stanford affiliation. I have co-authored COVID-19 scientific papers with both GBD and JSM signatories (more with the latter). I have more close ongoing collaborators and friends in JSM than GBD. According to Scopus I have 6590 co-authors and probably >200 have signed GBD or JSM. I have learned from both JSM and GBD colleagues and I thank them all for sharing their wisdom.

    As I did in my original paper, I applaud Pimenta again for his amazing work. Additional studies of engagement, impressions and reach would be very useful to perform. Pimenta defends fervently but needlessly some of the JSM main points, since my paper attacked neither JSM nor GBD. It only showed that both lists include many stellar scientists and that JSM had overwhelming Twitter presence. This is emphatically obvious also in the Twitter reception of my paper.

    Gorski apparently submitted his rapid response and his 7591 words long blog in his sciencebasedmedicine.org website before seeing my...

    Show More
  • Time for the science Kanyes to stop harassing the science Kardashians

    Time for the science Kanyes to stop harassing the science Kardashians

    This study doesn't adequately measure what it seeks to measure and may instead harmfully discourage people from engaging with the stakeholders of science.

    We share many of the concerns presented by other rapid reviews, but we contribute to the discussion by critiquing this study from a science communication perspective, which we would argue is the most appropriate domain for this kind of study.

    The study aims to measure whether it is scientific citations or social media metrics that influence the apparent perception that one group of scientists' policy responses to COVID-19 are more supported by credible scientists compared to another group's.

    We could point to concerns about the precision and measurement of the outcomes under investigation. The description of "perceptions" (presumably an attitudinal concept) or "dominant narrative" or “prevailing narrative” (presumably a relative measure of message prevalence) are never defined or measured in this paper. Instead, there is an assumption that the number of Twitter followers axiomatically leads to such attitudinal or message prevalence outcomes. As science communication researchers, we wish it were so easy.

    While one reviewer queried this assumption, it is disappointing that it was not fully addressed. As noted by that reviewer, "The author attempted to examine a group of scien...

    Show More
  • Yamey: Reply to Ioannidis

    I am grateful to John Ioannidis for replying to my rapid response, but I do not agree with him when he says that I have misrepresented his study. I also believe he has failed to address my two original concerns: (1) along with a group of GBD signatories, he lobbied the Trump administration, yet he failed to declare this competing interest, and (2) his paper demeans, belittles, and humiliates named scientists and yet he did not seek ethics review. I also am very concerned indeed that the author has not described the GBD's competing interests (he merely says, " GBD leaders have repeatedly denied conflicts of interest"). He merely takes the GBD, who he is allied with, at face value.

    I'd like to more fully explain my three concerns.


    It is a matter of public record that Prof Ioannidis worked with three GBD signatories, one of who was an author of the GBD, to lobby the Trump Administration, as described in a March 2020 investigative report, titled "An Elite Group Of Scientists Tried To Warn Trump Against Lockdowns In March." [1]

    The author of the investigation, Stephanie Lee, writes that: "John Ioannidis’s controversial studies claim that the coronavirus isn’t that big a threat. Before the Stanford scientist did any of them, he wanted to take that message to the White House."

    Lee notes:

    "Stanford University scientist John Ioan...

    Show More
  • And for the right reasons

    I am reminded of Doug Altman's seminal paper, The Scandal of Medical Research, published in the British Medical Journal almost 30 years ago. It begins:

    "We need less research, better research, and research done for the right reasons."


    I believe these are the most important words ever written about our field. They are also the most ignored. If asked for evidence of the latter, I would start by highlighting this paper, published under that same BMJ banner, and from which nothing of scientific substance can be learned. The bar remains entirely too low.

  • Elevating pettiness, downgrading science

    The BMJOpen has published a 'paper' by a close friend and supporter of Great Barrington authors, that attempts to measure 'social media influence' by Twitter followers (ignoring an entire field of data science that studies engagement, impressions and reach) in correlation to scientific impact (using an equally flawed citation count metric) combined into an index that was published and later confirmed to be, an actual joke. It's called the Kardashian Index, in order to say what exactly?
    That letting millions of people become infected with COVID-19 when a vaccine was only a month or two away was actually a good idea? That shutting away approximately 30% of the population for an indeterminate amount of time was both feasible and ethical? That natural infection would confer lasting immunity (it didn't)? Or that variants wouldn't arise as a consequence of widespread infection (they did?). And all of these good ideas would've been gladly received by the global scientific community and it's decision makers, nearly 100% of which completely ignored these ideas as the nonsense they were, if not for a group of 30 scientists on Twitter?

    Is that really what the BMJ, the British Medical Journal, thought was a scientific and academically rigorous concept? Did they even read any of the references? Such as when the Kardashian Index 'author referred to his own work as "just a bit of fun"?

    These discussions wer...

    Show More
  • Is this incredibly subtle satire?

    It is very puzzling how someone of Prof. Ioannidis' stature could so lower himself as to write such a methodologically flawed a manuscript. However, to me its methodological flaws, which have been well covered by other Rapid Responses, are completely overshadowed by a far more glaring problem, a conceptual one at the heart of the very premise of the manuscript. The Kardashian index was conceived as satire. If you do not believe me, look no further than to Neil Hall himself, who, having been tagged in Tweets about Prof. Ioannidis' article, took to Twitter to say that the Kardashian index was "a dig at metrics not Kardashians. It’s like taking a quiz to see what character from Game of Thrones you are and finding out you’re Joffrey Baratheon. It doesn’t matter - it’s not a real test. Thankfully," adding that "the tells that the entire premise is satire could not be made more obvious." (https://twitter.com/neilhall_uk/status/1492259823114723329)

    Indeed, various points in his paper (https://genomebiology.biomedcentral.com/articles/10.1186/s13059-014-0424-0), Hall's "tells" include passages like:

    1. "I had intended to collect more data but it took a long time and I therefore decided 40 would be enough to make a point. Please don’t take this as representativ...

    Show More
  • Bak-Coleman: Reply to Ioannidis

    I welcome Professor Ioannidis’ engagement on these issues. The response however falls short on several points and raises additional concerns.

    Ethical: Professor Ioannidis is correct that many papers make use of Twitter and other publicly available sources of data. However, he is mistaken in the assertion that as a result, such work does not need to be reviewed by an IRB (my own research is a case in point). IRBs exist in order to safeguard the rights of human subjects, and the decision about whether an IRB is required critically must rest with an IRB rather than with individual scientists conducting research.

    Professor Ioannidis argues that IRB approval should not be required to publish publicly searchable information in a deanonymized context. This is a dangerous position to take. A researcher using this standard could publish, for instance, the sleuthed addresses, family members, places of work, and identities of political dissidents or victims of hate crime. In this particular case, Dr. Ioannidis indirectly ascertained non-use of Twitter, a decision that was made in a reasonably private context. No reasonable definition of consent to participate in the Twitter portion of the study could be applied for these signatories. Given the inflamed nature of the discussions around these issues, the potential for signatories of either to experience further negative attention on social media as a result of their Twitter accounts being publicly identified and linked...

    Show More
  • Typo

    In composing my response, I have a typo regarding the Kardashian Index. When someone does not use twitter, it goes to zero not infinity.

  • In reply to Sheldrick, Bak-Coleman and Yamey

    I thank Sheldrick, Bak-Coleman and Yamey for their constructive criticism. As stated clearly in the paper, the main analysis focuses on the key signatories of both documents: All the key signatories are included without random sampling. As the paper already explains in detail, since thousands of additional people signed each document, a few randomly selected signatories from the longer lists were also explored. Random numbers were generated in Excel. There was no power calculation for this secondary analysis. The paper already explains that this secondary analysis deserves caution, since only 443 GBD signatories were listed by name when the two documents were accessed online in April 2021.
    The Twitter data represents information readily retrieved by a Google search by anyone. The notion of requiring IRB approval to report the results of searching Google or free publicly available databases (e.g. citation databases) contradicts the practice of hundreds of thousands of published papers reporting on such searches without IRB approval. Moreover, contrary to what Bak-Coleman asserts (“failure to disclose the author's well-documented history of interaction (co-authorship, affiliations, debate, etc..) with the named signatories—positive and negative”), my disclosures clarified explicitly that “The author has signed neither of the two documents and has many friends, collaborators and other people who he knows and he admires among those who have signed each of them...

    Show More