Article Text
PDF
Abstract
Objectives In comparison to quantitative research, the impact of qualitative articles in the medical literature has been questioned by the BMJ; to explore this, we compared the impact of quantitative and qualitative articles published in BMJ.
Design Cross-sectional survey.
Setting Articles published in the BMJ between 2007 and 2017.
Main outcome measures Bibliometric and altmetric measures of research impact were collected using Web of Science, Google Scholar, Scopus, Plum Analytics and ProQuest Altmetric. Bibliometric measures consisted of citation numbers, field weighted citation impact and citation percentile. Altmetric measures consisted of article usage, captures, mentions, readers, altmetric attention score and score percentile. Scores were compared using the Wilcoxon Rank-sum test.
Results We screened a total of 7777 articles and identified 42 qualitative articles. Each qualitative article was matched to 3 quantitative articles published during the same year (126 quantitative articles). Citation numbers were not statistically different between the two research types; the median number of citations (google scholar) per quantitative article was 62 (IQR 38–111) versus 58 (IQR 36–85) per qualitative article (p=0.47). Using Plum Analytics, qualitative articles were found to have a significantly higher usage, with a median of 984 (IQR 581–1351) versus 379 (IQR 177–763) for quantitative (p<0.001). The Altmetric Attention Score was higher for quantitative articles at 16 (IQR 7–37) versus qualitative articles at 9 (IQR 5–23, p=0.05), as was the Altmetric Score percentile 93 (IQR 87–96) versus 88 (IQR 76–95; p=0.02).
Conclusion Qualitative and quantitative articles published in the BMJ between 2007 and 2017 both have a high impact. No article type was consistently superior in terms of bibliometric or altmetric measures, suggesting that type of article is not the major driver of impact.
- qualitative research
- epidemiology
- biotechnology & bioinformatics
- health services administration & management
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
- qualitative research
- epidemiology
- biotechnology & bioinformatics
- health services administration & management
Strengths and limitations of this study
Impact of qualitative and quantitative research evaluated using two valid impact measurement tools.
Evaluation of impact included both the academic and social impact of research.
Use of five different data management sources to collect a variety of impact measures.
Impact was assessed via bibliometrics and altmetrics; however, there are many other ways to evaluate impact.
Comparison of impact was limited to articles published in the BMJ.
Introduction
The number of qualitative studies published in medical journals has increased over the last 20 years1–3; however, the overall number of qualitative articles published to date in these journals remains low.4–6 Qualitative research allows researchers to explore the meaning given by participants to their experiences and understand the context that informs those experiences.7 Strauss and Corbin define qualitative research as ‘any type of research that produces findings not arrived at by statistical procedures or other means of quantification’.8 Because qualitative researchers reject quantification, and therefore generalisability, and rely on subjective accounts, some view qualitative research findings as limited in their usefulness and scientific credibility.9 The Strength of Recommendation Taxonomy classifies qualitative research as the lowest level of evidence, similar to case reports, expert opinion, and anecdotal findings.10 These negative views may explain the low publication rate of qualitative research in medical journals.
In 2016, the BMJ developed a policy of rejecting qualitative articles on the grounds that such studies were ‘low priority’, ‘unlikely to be highly cited’, ‘lacking practical value’, or ‘not of interest to readers’.11 12 In response, several authors wrote letters to the editors reinforcing the importance of qualitative research publications in the BMJ.13–19 Following discussions, the BMJ editors stated that qualitative articles would be considered for publication, although the editors stated that this research methodology was less accessed and cited than other quantitative designs, and hence less helpful for doctors in making decisions.20
The impact of a research article can be measured through a variety of means. Traditionally, bibliometrics measured the academic impact of articles by reporting journal impact factors and citations numbers.6 21 22 Bibliometrics specifically capture research productivity and impact through citation analysis: papers per researcher, citations per paper, journal of publication and place of production. Bibliometrics focus on the academic impact of articles and do not assess the influence of articles beyond the research community.23–25 In 2001, the editor of the BMJ underlined weaknesses with current measures of impact such as bibliometrics, highlighting a lack of assessment of social impact of research.26 Altmetrics were created to complement bibliometrics and provide immediate social impact of scholarly publications. Altmetrics collect measures from a variety of sources including reference management tools, Wikipedia, and popular social media applications such as Facebook, Twitter, news outlets and blogs. Using various algorithms, altmetric platforms transform the gathered data into altmetric scores which denote the social impact of research articles. Altmetrics broaden impact evaluation beyond researchers by including a diverse audiences such as policy-makers, clinicians, researchers, educators and the general public.27–30
To investigate the 2016 BMJ editors’ statements claiming lower impact of qualitative articles, we performed a bibliometric and altmetric analysis to compare the academic and social impact of qualitative and quantitative research articles published in the BMJ over the recent period of eleven years. The BMJ was specifically selected as it publishes the most qualitative research among the high impact medical journals.1 3
Methods
Overview
Bibliometric and altmetric measures included citation numbers, article usage, abstract views, article captures, article mentions online and social media usage. Google Scholar, Web of Science, Scopus, Plum Analytics (https://plumanalytics.com/) and ProQuest Altmetric (www.altmetrics.com) were reviewed to obtain data (tables 1–3).
Bibliometric data reported
Plum Analytics measures definitions36
ProQuest Altmetric measures definition37
It was not appropriate to involve patients or the public in the study.
Selection and categorisation of articles
All articles published in the BMJ between 1 January 2007 and 31 December 2017 were screened. One author specifically screened the title and methods section of all articles to determine the article type. The research team was consulted if uncertainty arose during the screening process.
Articles between 2007 and 2017 were included as this marks the beginning of online platforms such as Twitter and provides 2 years for bibliometric measures to be generated. Previous authors have suggested that 2 years post publication are needed to allow for reliable bibliometric indicators.31
Only research articles presenting primary sources of data were included in our analysis. Research articles included quantitative studies (observational studies, economic evaluations and clinical trials) and qualitative articles. Article types such as systematic reviews, meta-analysis, research methods, editorials, rapid responses, personal views and opinions were excluded. Systematic reviews and meta-analysis were excluded as they present secondary data, that is, synthesis of published primary data from qualitative and/quantitative sources.
The research articles identified were classified as qualitative or quantitative studies through title and abstract review (and full text as necessary). Qualitative publications included various methodologies such as ethnography, grounded theory and case study and used methods such as interviews, focus groups, observation and discourse analysis. In addition, qualitative publications included articles using (1) purely qualitative (see above list), and (2) mixed methods methodology with at least one component being qualitative. Qualitative research was defined as ‘any type of research that produces findings not arrived at by statistical procedures’ which involved ‘an interpretive, naturalistic approach to its subject matter’.32 Quantitative publications included observational studies (cross-sectional studies, case-control studies, cohort studies), economic evaluations and clinical trials.
Qualitative research articles were matched 1:3 without replacement to quantitative articles published during the same year using an excel random number generator (Microsoft Excel V.14.6.2; Microsoft). Specifically, each qualitative article published in a given year was given a unique identifier, which was then matched to three unique quantitative articles from that same year. We matched each qualitative article to three quantitative articles in the aim of balancing covariates, decreasing variance, while controlling for year of publication.33–35
Source of data
Bibliometrics
Bibliometric measures collected consisted of citation numbers, field-weighted citation impact and citation percentile. We determined these measures from Web of Science, Google Scholar, Scopus and ProQuest Altmetric for all articles in April 2019. We selected multiple sources for bibliometric measures as each platform collects citation numbers in a slightly different manner (table 1).
Altmetrics
Altmetric measures were obtained from Plum Analytics (table 2, https://plumanalytics.com/) and ProQuest Altmetric (table 3, https://www.altmetric.com) for all articles in April 2019 (tables 2 and 3).
We selected Plum Analytics as it provides a detailed breakdown of five categories (usage, captures, mentions, social media and citations) allowing for comprehensive assessment of impact. We selected ProQuest Altmetric as this platform calculates an overall altmetric score, a useful tool to compare individual articles.
Plum Analytics generates PlumX Metrics which collects article level metrics: (1) usage, (2) captures, (3) mentions and (4) social media.36 ProQuest Altmetric measures included the (1) Altmetric attention score, (2) Altmetric score percentile, (3) mentions and (4) Mendeley (free reference manager) readers.37
Statistical analysis
Bibliometric and altmetric measures were assessed using medians with IQR as the data were not normally distributed (Kolmogorov-Smirnov test, p<0.05). The Wilcoxon Rank-sum test was performed to compare individual altmetric and bibliometric measures for qualitative versus quantitative articles. P value<0.05 was considered statistically significant. If percentile scores were available (ie, Scopus citation percentile and Altmetric score percentile), measures above 75th were considered high. A sensitivity analysis was performed to evaluate the effect of excluding mixed methods articles and considering only purely qualitative research article types. All statistical analyses were performed with the SAS Studio University Edition, V.3.8 (SAS Institute).
Results
A total of 7777 articles were screened and we identified 42 qualitative articles published in the BMJ between 2007 and 2017 (figure 1). These were matched to 126 quantitative articles. Notably, of the qualitative articles 41 of 42 were published from 2007 to 2011, while only 1 article was published after this time period, in 2013. Most qualitative articles were purely qualitative (83%), rather than mixed method. Quantitative articles included randomised trials (47%), observational (48%) and decision analysis/economic evaluation (5%) studies.
Flow diagram.
Based on bibliometric measures, citation numbers were not statistically different between the two research types (table 4). Using Web of Science data, we found a median of 62 citations (IQR 38–111) per quantitative article as compared with 58 (IQR 36–85) for qualitative articles (p=0.47). This was similar for Scopus, Google Scholar and ProQuest Altmetrics. For the field weighted citation impact, no statistical differences were noted for the two research types; quantitative articles had a median field weighted impact at 7.06 (IQR 4.58–12) as compared with 5.96 (IQR 4.33–10.26) for qualitative (p=0.16). Both qualitative and quantitative articles were frequently cited with a citation percentile above 91, potentially indicating high impact.
Summary statistics for bibliometrics
For altmetrics measures, using Plum Analytics, qualitative research was found to have a significantly higher usage, with a median of 984 (IQR 581–1351) clicks, downloads or views as compared with 379 (IQR 177–763) for quantitative research (p<0.001, table 5). Likewise, qualitative articles had significantly higher captures with a median of 191 bookmarks, favourites, or readers (IQR 98–292) as compared with 88.5 (IQR 35.5–191) for quantitative research (p<0.001, table 5). The two research types had similar mentions and social media impact on Scopus (p=0.13 and p=0.43, respectively).
Summary statistics for Altmetrics
Using ProQuest Altmetric, the Altmetric Attention Score was higher for quantitative research at 16 (IQR 7–37) as compared with qualitative research at 95–23 (p=0.05, table 5). This translated to a significantly higher Altmetric Score percentile with quantitative research scoring a median Altmetric Score percentile of 93 (IQR 87–96) compared with 88 (IQR 76–95) for qualitative (p=0.02). Of note, both percentiles were considered high (above 75th percentile). Despite differences in the overall Altmetric Attention Score, the individual factors included in this weighted score were no different between the two types of research. In fact, new outlets, blogs, policy sources, tweets, Facebook posts, Wikipedia mentions were not statistically different between both types of research (table 5). Both qualitative and quantitative articles were highly used online, with frequent reports on social media outlets.
The exclusion of mixed methods articles with purely qualitative research articles did not alter our findings (data not shown).
Discussion
This bibliometric and altmetric analysis compared the academic and social impact of qualitative and quantitative article published in the BMJ from 2007 to 2017 and did not find a dominant article type using these metrics. Bibliometrics as evaluated by citation analysis were similar between both article types. Altmetric measures differed depending on the method of assessment, with usage and captures significantly higher in qualitative articles and Altmetric Score Percentile higher in quantitative articles. With regards to ProQuest Almetrics, both research types had high percentile scores (>88th Altmetric score percentile) indicating high impact for both article types. Interestingly, although our analysis spanned 11 years, no qualitative articles were published in the BMJ after 2013.
Our analysis highlights that research articles published in the BMJ had high impact regardless of whether the article was quantitative or qualitative, likely due to selective publication process of this medical journal. Schroter performed an audit of publications in the BMJ from 2003 to 2012 and reported that while over 3000 research articles were submitted each year to the BMJ, only 2.1%–7.7% were accepted for publication.38 Our findings indicate that articles published in the BMJ score in the highest percentiles with regards to citations and altmetric scores, reinforcing the notion that research articles published in this journal are widely used regardless of their methodology. We found no indication that qualitative articles published in BMJ had had less impact than quantitative articles.
In contrast to the 2016 editorial, our analysis did not find that qualitative articles published in the BMJ were ‘unlikely to be highly cited’, ‘lacking practical value’, and ‘not of interest to readers’.11 12 First, using a variety of citation trackers, we found high citations numbers for both qualitative and quantitative articles, with the median score for both times of articles above the 90th citation percentile. To evaluate reader interest and practical value, we used altmetrics as measures of the social impact of articles. Qualitative and quantitative articles had similar mentions online on news outlets, blogs, twitter, Facebook and Wikipedia. On twitter, both article types influenced a large audience with more than 19 000 individuals reached by the articles. These findings suggest high social impact of both qualitative and quantitative articles published in the BMJ.
Previous authors have compared the impact of qualitative and quantitative research using citation analysis. In 2013, Mori and Nakayama reviewed the academic impact of qualitative studies in healthcare.6 Using citation numbers, the authors concluded that qualitative studies were cited less frequently as compared with systematic reviews and randomised trials, but were similarly cited compared with the median of the impact factor of the journal. Importantly, the authors did not compare qualitative studies to quantitative observational studies. In 2011, Weiner et al compared citation numbers for qualitative articles published in nine major health services and management journals to a random sample of quantitative articles and found similar citation frequencies between the two, suggesting that qualitative research articles contribute comparably to the field’s scientific knowledge base.4 Our analysis reinforces the findings that qualitative and quantitative articles have similar academic impact, and brings forward additional information on the social impact of qualitative research in order to provide a broader analysis of article impact.
In a variety of fields, individuals have evaluated impact using bibliometrics and altmetrics to determine research interests of the members of the public and of the scientific community.39–43 In 2018, Powell et al compared the altmetric score and bibliometric data of the 100 most cited articles in surgery and found, similar to our study, that bibliometric and altmetric analyses provide important but different perspectives on article impact.43 Altmetrics offered the unique advantage of timely assessment of articles generating discussions online and positively correlated with citation numbers.43 In 2018, Banshal et al compared the top 100 Altmetric papers and the top 100 cited papers and found minimal overlap between these.42 The authors suggested that it was unlikely that one assessment tool predicts the other. The authors concluded that altmetric score provided additional rather than duplicate information to citation scores. In 2019, Azer and Azer found no correlation between the number of citations and the altmetric scores for top-cited articles in medical professionalism; though, for articles published after 2007, the authors found a significant correlation between number of citations and altmetric scores.44 In contrast, in 2020, Luc et al found that tweeting significantly increased citations numbers over time.45 The relationship between altmetric and bibliometric measures thus remains controversial.46 47 Because of the debatable correlation between altmetrics and article citations, some authors had cautioned the use of altmetrics as a measure of impact.27 48 Rather, we suggest that altmetrics should be viewed as complementary to existing tools to measure impact.
Our study strengths include the use of two valid impact measurement tools to evaluate both the academic and social impact of research articles published by the BMJ. In addition, we use five different data management sources (Web of Science, Google Scholar, Scopus, Plum Analytics and ProQuest Altmetric) to conclude that both qualitative and quantitative articles published in this journal have a high impact. Our study has some limitations. We used bibliometrics and altmetrics to assess impact; however, many of the measures generated by the websites do not have reference values to help adjudicate what is a high or low score (other than percentile scores). Furthermore, the comparison of impact between qualitative and quantitative research focused only on articles published in the BMJ, and therefore these findings may not be generalisable.
In conclusion, using bibliometric and altmetric measures, our study has found both qualitative and quantitative research published in the BMJ have a similar impact on the public as well as the academic community. Based on our findings, there is no evidence to support claims that qualitative studies published in the BMJ are of lower impact than quantitative articles.
References
Footnotes
Twitter @HeleneRetrouvey
Contributors HR made substantial contributions to study conception and design, drafted the manuscript, approved the version to be published, as well as agreed to be accountable for all aspects of the work. NNB made substantial contributions to study conception and design, provided critical review of study content, manuscript drafting, approved the version to be published, was the study guarantor, gave final approval of the manuscript, as well as agreed to be accountable for all aspects of the work. ARG, FW and TZ made substantial contributions to study conception approved the version to be published, provided critical review of study content, gave final approval of the manuscript, as well as agreed to be accountable for all aspects of the work.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Patient consent for publication Not required.
Ethics approval Ethics was not required for this study.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement Data are available upon reasonable request. Relevant data are available upon request.