CommentariesLimitations are not properly acknowledged in the scientific literature
Introduction
All research work unavoidably has some limitations. Even the most important breakthroughs are unlikely to be devoid of them. Knowledge and discussion of limitations are essential for genuine scientific progress: they are useful for understanding a research finding, translating the importance of the potential errors involved, placing the current work in context, and ascribing a credibility level to it [1]. Limitations are also likely to reveal how the current research work may be improved in future experiments and what caveats should be considered in trying to incorporate this new information in the evolving body of scientific evidence. However, are limitations properly acknowledged in the scientific literature?
The question is difficult to answer. Researchers may have different ways to discuss and address caveats in their work. For example, a perceived drawback may lead to further experiments that clarify the remaining issues. In particular, empirical research papers with statistical hypothesis testing, may indeed present some measure of the errors and the uncertainty surrounding the measurements and inferences thereof. However, this does not necessarily reflect a specific acknowledgment of limitations by any means.
Claiming limitations is unavoidably a subjective process to a considerable extent. It is not simply an issue of listing the magnitude and direction of various errors, random or systematic, that have been introduced in the measurements; or the problems in the theoretical concepts and methods used in the research; or the generalizability of the findings. Limitations go beyond a proper list of methods, errors, and validity. They require an overall view, appraisal, and interpretation: do these problems with errors, methods, and validity eventually matter, and, if so, to what extent? At the end of the day, this is a crucial link to interpret the credibility of the published research and make sense of it. For measurement errors, larger errors may be more likely to be perceived as limitations, but what is a large enough error that would affect the conclusions? For external validity, such subjective appraisal is even more necessary: if a research project is limited to a very narrow focus, is this a limitation or strength? Someone needs to interpret this dilemma, ideally both sides of it. Finally, for research work that has no measurements, for example, mathematical or theoretical constructs, it is still important to know what the perceived weaknesses are.
One may argue that scientific work should be objective; subjective statements have no place in scientific writing. I could strongly sympathize with this view, which would mean that research articles should have no Discussion section, just transparent Methods and full Results. But then, not only subjective interpretations of limitations would be eliminated from scientific writing, but also subjective claims of importance, relevance and (nonstatistical) significance would be dispensed. All these “positive” aspects of research are eventually subjective interpretations based on—hopefully—objective data and constructs. The same applies to limitations. If we accept subjective claims of importance in scientific writing, we should accept and encourage also statements of limitations.
Section snippets
Limitations in the high-profile research literature
I searched electronically the full-text (except references) of the first 50 original research articles published in 2005 in each of the six scientific journals that receive the highest total number of citations [2]. These journals include multidisciplinary publications (Science, Nature, Proceedings of the National Academy of Sciences [PNAS]) and leading journals in biochemistry (Journal of Biological Chemistry), physics (Physical Review Letters), and chemistry (Journal of the American Chemical
Limitations in abstracts, section headings, first sentence in paragraph
Only four of the 400 articles used a word on limitations in their abstract. One of them was a technical term (“growth limitation”) in a Journal of Biological Chemistry article. Another was an opening sentence in a Nature article that set the setting about the superiority of the presented nano-device methodology: “devices have been investigated… that could overcome the physical and economic limitations of current semiconductor devices”. A PNAS article claimed that there is “an important existing
Errors and importance vs. limitations
The sparse allusion to limitations does not mean that these articles did not discuss errors in their scientific work. In fact, 243 of 400 articles (61%) used at least once one or more words detected by the electronic roots error, valid, bias, reproducib, or false. However, the vast majority of these appearances had a neutral context, for example, measures of errors or error bars were given without further comment, or estimates were given of validity or validation procedures. However, it was not
Weaknesses of the empirical survey
This survey has limitations. Capturing all the language that touches on limitations is impractical. Thus, I relied on automated searches and simple context analysis. Additional research using more sophisticated qualitative linguistic methods might be useful to expand our understanding of this problem. Secondly, I focused on top-cited, highly competitive journals. The pressure to highlight importance and hide weaknesses may be stronger in these journals. However, if the most prestigious journals
Instructions to the authors and editorial policies
What do journals seek in the articles they publish and what do they instruct authors to do for limitations? I downloaded from the web the instructions to the authors and editorial policies for the 25 scientific journals with highest total number of citations (Journal Citation Reports, 2004 edition) (Table 1). Importance, novelty, originality, significance, and relevance figure prominently in the language of what drives acceptance for publication. Cell asks for “unusual significance”, PNAS for
Final considerations: limitations, expectations and discussions
I argue that a section of limitations should be routinely considered in publications of original research. This does not have to be extensive, but should alert the reader to key weaknesses in the presented work. Identifying weaknesses is a first important step toward formulating new research questions. It may also help toward improving research methodology and the validity of extrapolation of results. Given the dearth of attention given to discussion of limitations, I suspect that the current
References (13)
- et al.
Publication bias in clinical research
Lancet
(1991) - et al.
Translation of highly promising basic science research into clinical applications
Am J Med
(2003) Why most published research findings are false
PLoS Med
(2005)- Thomson Institute for Scientific Information, Journal Citation Reports,...
PLoS biology in action
PLoS Biol
(2004)- et al.
The PLoS community journals
PLoS Biol
(2005)
Cited by (115)
Automatic categorization of self-acknowledged limitations in randomized controlled trial publications
2024, Journal of Biomedical InformaticsHow We Write a Manuscript Discussion
2023, Research and Practice in Thrombosis and HaemostasisBiobanking and research quality: think locally, act globally
2023, Trends in GeneticsStrange face illusions: A systematic review and quality analysis
2023, Consciousness and CognitionBusiness-community relations under COVID-19: A study of micro and small firms
2023, Journal of Business Research