Article Text
PDF
Abstract
Objectives In 2009, not all journal editors considered systematic reviews (SRs) to be original research studies, and not all PubMed Core Clinical Journals published SRs. The aim of this study was to conduct a new analysis about editors’ opinion regarding SRs as original research.
Design We conducted a survey and qualitative interview study of journal editors.
Participants All editors listed as editor-in chief of 118 PubMed Core Clinical Journals.
Methods We contacted editors via email and asked them whether they considered SRs original research, whether they published SRs in the journal and, if yes, in which section. We searched PubMed for any SRs (or meta-analyses) published in the included journals in 2017; if we did not find any, we hand-searched these journals. Editors were invited to participate in a follow-up qualitative interview study.
Results We received responses from 73 editors representing 72 (62%) journals. Fifty-two (80%) editors considered SRs original research, either for any type of SR (65%) or only for SRs with a meta-analysis (15%) and almost all (91%) of editors published SRs. Compared with the results of the 2009 study of Core Clinical Journals, a similar proportion of editors considered SRs to be original studies (71%), accepted SRs as original on certain condition such as presence of meta-analysis (14%) or published SRs (94%). Interviews with editors showed that they used various criteria to decide whether a SR is original research, including methodology, reproducibility, originality of idea and level of novelty.
Conclusion The majority of editors of core clinical journals consider that SRs are original research. Among editors, there was no uniform approach to defining what makes a SR, or any study, original. This indicates that the concepts of originality of SRs and research are evolving and that this would be a relevant topic for further discussion.
- systematic reviews
- editors
- original research
- opinions
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
This mixed-methods study combines an online survey with qualitative research methods.
This study had a high response rate of editors (62%).
The study provides more detailed data about editors’ reasoning about the originality of systematic reviews (SRs).
A limitation of this study is our use of a sample of journals indexed as PubMed Core Clinical Journals and the possibility that some journal editors were surveyed both in the earlier study and in this follow-up study, although none of the editors indicated that they participated in the earlier study.
The study relied on PubMed’s indexing to identify SRs and meta-analyses published by the included journals; hand-searching of all journals may have yielded some additional SRs.
Introduction
Global scientific output is growing exponentially,1 leaving users of research evidence to grapple with numerous individual studies, which may have different conclusions even if they cover the same topic. The development of research methodology for qualitative and quantitative synthesis of evidence led to the emergence of systematic reviews (SRs) as research synthesis method.2 This was also suggested in previous studies examining the status of SRs in research community.3 4
In 2012, Meerpohl et al published a survey of editors of core clinical journals regarding their attitudes about the value of SRs for journals.3 The study included 65 editors of the 118 journals, who were surveyed in April 2009. The editors were asked if they considered SRs to be original research, whether their journal published SRs and in which section SRs were published. The results indicated that 71% of the respondents considered SRs to be original studies and the majority of their journals published SRs.3
A study published in 2017 examined the acceptance of SRs as a doctoral thesis in European biomedical doctoral (PhD) programme.4 Almost half of the surveyed participants, who identified themselves as being in charge of doctoral programmes, reported that in their institutions a SR is an acceptable piece of research for an entire or at least part of a PhD thesis, but the majority of surveyed individuals did not have sufficient knowledge about basic concepts of SR methodology.4 However, more than a half of the participants indicated agreement with the statement that ‘systematic reviews do not produce enough new knowledge for a dissertation’. A third of the respondents indicated that there was a lack of appreciation for SR methodology among faculty members.4
In the years since the study of Meerpohl et al was conducted, the number of published SRs has increased, as well as their influence.5 On the other hand, it has been suggested that there is an overproduction of redundant, misleading and conflicted SRs and MAs in addition to the marked publication growth.6 The aim of this study was to follow up on the attitudes of journal editors towards SRs. We hypothesised that the proportion of editors who consider SR to be original research had increased since 2009 and that more of the surveyed journals now publish SRs.
Methods
Survey
Participants
We invited editors of 118 journals labelled as Core Clinical Journals by the National Library of Medicine, USA in February 2018.7 The list of journals that were considered Core Clinical Journals in 2009 (when the previous study was conducted)3 and in 2018 is available in online supplementary file 1. We retrieved editors’ names and contact details from journal web sites. Ten journals indicated that they had more than one editor-in-chief; eight of those 10 journals had two editors-in chief, one had three and one had four editors-in-chief. In total, there were 131 editors-in-chief listed for the 118 journals.
Supplemental material
Survey
We surveyed editors via email, using the following four questions: (i) Do you consider a systematic review manuscript an original research project? (ii) Do you publish systematic reviews in the journal you edit? (iii) In which section of your journal would you publish a systematic review? (iv) Would you participate in a follow-up qualitative study via Skype?
We sent up to three email reminders approximately 1 week apart (MK, LP). We did not send reminders to the editors who responded that they do not wish to participate in the study.
Each study participant was assigned a code and coded responses were entered in to a spreadsheet with anonymised responses that were shared with other coauthors. One author initially evaluated and categorised responses (MK), in consultation with another author if necessary (LP).
Analysis of systematic reviews published in targeted journals
For all included journals, we assessed the types and characteristics of the published SRs. We performed a search on PubMed using journal name and limits for SRs and meta-analyses (MAs) and for articles published in 2017; search results were screened by two authors independently to verify that these publications were indeed SRs or MAs. Characteristics we analysed in those SRs are shown in box 1. We hand-searched the contents of journals published in 2017 if we did not find any SRs or MAs by searching PubMed. If by hand searching we did not find any SRs or MAs published in the journals included in 2017, we evaluated instructions for authors of journals that did not publish any SR to see if they had any guidance about submitting SRs and MAs.
Characteristics of systematic reviews published by target journals in 2017
Methods: We analysed the following characteristics of systematic reviews (SRs) published by target journals in 2017: (i) topic (therapeutic, epidemiology, diagnosis/prognosis or other), (ii) types of studies included, such as randomised controlled studies (RCTs), non-randomised studies (NRS), both RCTs and NRS, SRs or meta-analyses (MAs), SR/MA and primary research, SR and overview of SR (OSR) and those that did not define which studies were included, (iii) presence of meta-analysis (yes, no), (iv) update of a previous review (yes, no), (v) type of review (SR, OSRs, rapid review, scoping review).
Results: Detailed analysis of 1187 published SRs indicated that they most commonly addressed therapeutic interventions (n=585, 49%), epidemiology (n=281, 24%), diagnostic accuracy (n=105, 9%), prognostic issues (n=69, 6%) or other topics (n=147, 12%).
Regarding the type of included studies, the majority of SRs did not indicate in their methods what kind of studies were eligible for inclusion (n=301, 25%). There were 295 (25%) SRs that included both RCTs and NRS, 281 (24%) studies included only RCTs and 264 (22%) studies included only NRS. In 28 (2%) SRs, both primary research and SR/MA were included; 17 (1%) reviews included SR/MA and 1 review (0.1%) included both SR and OSR in the review.
Of all the analysed SRs, only 19 (2%) were an update of a previous SR. In 750 (63%) SRs, a meta-analysis was conducted. Most of the reviews were classic SRs that included only primary studies (n=1126, 95%); 28 (2%) SRs included both primary research and SR/MA so we defined them as SR/OSR; 19 (2%) OSRs, including one study that did not define which studies it included but called itself an OSR and one that included both SR/MA and OSR. Twelve (1%) SRs were scoping reviews and one was a rapid review (0.1%).
Statistics
We analysed data using frequencies, percentages and IQR to describe the results. Differences in proportions of key results between this study and the previous study3 were analysed with the χ² test. Proportions were reported with 95% CI. For analyses, we used MedCalc (MedCalc, Mariakerke, Belgium). Statistical significance was set at p<0.05.
Qualitative study
Participants
We contacted the editors who accepted to take part in the follow-up interview when responding to the survey. Participants were informed who will conduct the interview and that the authors intend to publish the study results.
Interviews
The qualitative study was conducted using semistructured interviews. Twenty-four editors volunteered to be interviewed. After 15 interviews, we stopped inviting further editors because we reached the level of saturation of identified themes which is usual methodology in qualitative research. The first interview was conducted on 6 July 2018 and the last on 7 September 2018. We used open-ended semistructured questions to enhance the discussion about originality of SRs (online supplementary file 2). Participants were not led to provide specifically any answer. We used the Consolidated Criteria for Reporting Qualitative Studies (COREQ) to guide the reporting of the study.8
Supplemental material
Research team and reflexivity
One author conducted all interviews (LP). The author that conducted interviews did not personally know any editors that were invited to participate.
Conduct of interviews
Interviews were conducted individually via teleconferencing software (Skype or Zoom) or via telephone, based on the choice of participants. All conversations were recorded. After each interview, a transcript was made and analysed, to monitor for the point of information saturation. Transcripts and recordings of interviews were not sent to study participants for checking, commenting and/or correction. Instead, the final draft of the manuscript was sent to the editors for their insight into the collected results.
Data analysis and reporting
To ensure uniformity, all transcripts of interview recordings were made by one researcher (LP), and another member of the team checked all transcripts and analysed them (MK). For the purpose of reporting, the participants’ names were coded. The transcripts were analysed via conventional qualitative content analysis.9 10 The transcripts were first read as a whole to obtain general attitude of one editor, and afterwards the text was analysed word-by-word highlighting parts of the text that produced meaningful units, including complete sentences or parts of sentences, and then a code was attributed to those selected parts of text. Codes were sorted into categories related to editors’ attitude towards SRs., Two researchers did the coding; the results were then compared, and any discrepancies were resolved by discussion until reaching consensus. The codes were entered into an Excel spreadsheet for further quantitative analysis. Complete sentences were also copied from the interview transcripts to provide original thoughts of editors in the manuscript.
Results
Survey among editors of core clinical journals
Of 118 contacted journals, we received responses from 73 editors representing 72 (62%) journals. Interestingly, two editors from a single journal gave us opposite answers to our survey questions. Response rate based on the number of editors was 73 out of 131 (56%). Online supplementary file 1 shows a list of included Core Clinical Journals in 2009 and 2018. Since 2009, 13 new journals were added to the list of Core Clinical Journals, while the same number was dropped from the list. So the total number of Core Clinical Journals in both years was the same.
Editors’ responses
Of the 73 respondents, 8 (11%) did not answer the first question despite our repeated attempts to obtain missing answers. Among the remaining 65 editors who responded the first question, 42 (65%) considered SRs to be original studies, 13 (20%) did not and 10 (15%) indicated that only a SR with a MA was an original study. Overall, 52 (80%) editors considered SRs original research, either for any type of SR or only for SRs with a meta-analysis. This proportion was 71% (46 of 65) in the previous study;3 the difference was not statistically significant (p=0.23; χ2=1.412, df=1; 9% increase, 95% CI −5.8 to 23.3). The proportion of editors with conditional acceptance of SRs was 15% (10 of 65). In the previous study,3 it was 14% (9 of 65). The difference was not significantly different (p=0.87; χ2=0.026, df=1; 1% increase, 95% CI −11.4 to 13.4).
Four editors did not answer the second question. Of the remaining 69 editors, 63 (91%) responded that they had published SRs in their journals, 6 (9%) that they had not. In the previous study, 94% (60 of 64) of editors indicated they published SRs; there was no significant difference between these proportions (p=0.51; χ2=0.425, df=1; 3% increase, 95% CI −6.9 to 12.8).
Out of 63 editors that declared they had published SRs, 16 (25%) editors published SRs in a journal section devoted to original studies, 19 (30%) in the review section or separate section of the journal, 10 (16%) of them publish SRs in the section for original studies if they have a MA, and in the review section if they do not.
More than a quarter of editors (n=18, 29%) indicated that they did not have a particular section for publishing SRs in their journal and that they simply published them in the section that corresponds to the topic of the manuscript. Five (8%) editors did not answer the third survey question.
Even though 10 journals had more than one editor-in-chief, we received response from more than one editor from only one journal, for which two editors responded. From those two editors we received opposite responses on the first two questions. One did not consider SRs original and said that their journal generally does not publish SRs, and the other one considered SRs original and said SRs are published in their journal. Only the answer to the third question about the section in which SRs are published was the same as both answered SRs were published in no particular section of the journal.
Among the surveyed editors, 24 (33%) accepted to participate in the follow-up interview, 17 (23%) declined and 32 (44%) editors did not answer, even after a reminder.
Of the 24 editors who accepted to participate in the interview, 14 (58%) answered the first question that they considered SRs to be original studies, 6 (25%) answered that they did not, 3 (13%) answered that they considered SRs to be original if they had a MA incorporated and 1 (4%) accepted to participate in the interview without providing the answer to the first survey question.
Additionally, four editors who responded to the survey, but did not participate in the qualitative part of the study, offered their opinions regarding originality of SRs via email together with responses to the three survey questions. Two of them indicated that SR is original if it has methodological rigour; as one of them put it: ‘Cochrane-type review, with good scientific rigour combined with a thorough review of the literature, is original research.’ One of them remarked that a review can never be original research, with one exception: That exception is when a systematic exploration of the evidence (usually meta-analysis) is used to test a new hypothesis, one that had not previously been considered or addressed in the data analysis. Even then, I would not consider it completely original but an original application of the work of other investigators. The fourth elaborated that MA is considered new data and therefore only SR with MA would be considered original study.
Analysis of systematic reviews published in core clinical journals
Among the 118 included Core Clinical Journals, 110 published a median of 14 articles (range 1–528) indexed by PubMed as SR or MA, while 8 did not publish a single such manuscript.
Among the six editors that previously declared that they did not publish SRs, there were only two editors in whose journals our search did not find a SR or a MA published in 2017. Four stated that they did not publish but our search found SRs or MAs published in their journal in 2017.
The analysis of instructions for authors of eight journals that did not publish SR in analysed period showed that only one journal had guidance for authors regarding submission of SR or MA. Because one journal published 528 articles indexed as SR or MA by PubMed, we only analysed in detail the first 30% of these articles in order of publication (n=158). Altogether we analysed 2240 journal articles. Using those criteria, we found that 1187 (53%) were indeed a SR, scoping review, overview of systematic reviews (OSR) or a rapid review that used systematic searching methods and the others were not. For individual journals, the median per cent of articles that were SRs was 50% (IQR 33%–67%). We also found that two of the six journals for which editors responded that they did not publish SRs actually published at least one in 2017. A detailed analysis of 1187 published SRs is shown in box 1.
Qualitative study
Fifteen (21%) editors participated in the follow-up interview. We stopped recruiting new editors after achieving thematic saturation. Interviews lasted between 6 and 25 min (median 12 min, IQR 9–16.5 min). The names of the interviewed editors were coded as ‘E1’-‘E15’.
Eight editors reported in the initial survey that SRs are original studies, three considered SR to be an original study only if it incorporated a MA and four did not consider SRs to be original studies.
Is any type of systematic review an original study?
Eight editors who considered SRs original studies were asked if they felt any different regarding the type of evidence synthesis, for example, SR with or without meta-analysis, scoping review, OSRs and any other type of SR or they thought any type of SR should be considered an original study.
Four editors indicated that all types of SR are original studies. One editor considered all SRs original if they passed peer-review scrutiny. One editor indicated that all SRs can be considered original studies but pointed out that he would still not publish a study which was not informative, such as an empty SR, that is, a review in which no eligible studies were found after literature search.11 One editor stated that a SR with a MA was still more novel than SR without MA. This indicates that certain editors equate originality of research with novelty and that certain types of studies, such as SRs, can have a whole spectrum of considerations:
E12: …from my point of view, the actual name or type of the systematic review is not that important. I think what I look is did they searched certain number of databases, did they do it in a systematic way, did they follow methods.
Three editors, who considered only SRs with MA as an original study, were asked if they thought meta-analysis was the only thing that makes a SR an original study. Two editors insisted that MA is essential for SRs to be considered original, while the third was more ambivalent:
E1: … I think that in the academic world of medicine we begin to recognise that not all science is discovery science, that there is also science of integrating results and so I think in that sense systematic reviews perhaps could be eventually consider original. But, in my research it is only a matter of in which section you put it, and we could as well put it in the section of original studies.
Four editors who did not consider SRs to be original studies had the same opinion about any type of SR and analytic methods used in such studies.
Two editors stated that there may be a continuum of considerations about originality of SRs. Editor E5 indicated that original SRs are those that bring new knowledge and, therefore, that there is no universal answer; that this should be judged individually for each SR:
E5: This should be judged on a case-by-case scenario. In my field, if you look at systematic reviews about influenza or vaccines, there have been multiple systematic reviews that did not bring any new knowledge. So, these studies should be judged individually.
Editor E15 was ambivalent about the response; this editor considered that SRs were original if they contain MA but acknowledged that SR authors may have planned to do a SR with MA, but whether MA is possible or not, this is often difficult to determine at the onset. MA may not be possible because of clinical or statistical heterogeneity:
E15: So it is not the fault of authors; they have done everything by the book. So I think there is some fluidity in this respect, and this division of original and non-original systematic review is artificial.
Elements of originality in systematic reviews
All editors were asked to define what makes a SR an original study or which elements are missing in SRs to be considered original. The answers are stated in table 1. Responses included specific elements of SR methodology, originality of idea and usefulness of SRs. Editor E9 opined that ‘original is bad descriptor for research’.
Opinions of interviewed editors about elements of originality present or missing in systematic reviews
Four editors who did not consider SRs original studies were asked if they considered any secondary study to be original. Two indicated that only primary studies could be considered original, one did not provide an answer to this question while the fourth editor was ambiguous:
E10: Systematic reviews are taking data and analyzing them in different ways, and asking different questions, and so I do not value them as highly as primary or plain secondary research. …secondary studies could be considered original as long as they are not systematic reviews.
Editors’ opinion about value of systematic reviews
The editors provided the following responses to describe the main value of SRs: (i) synthesis of data (nine editors); (ii) providing answers to the posed clinical question (nine editors); (iii) data analysis (two editors); (iv) impartial and free of bias (three editors); (v) underlining guidelines (one editor); (vi) pointing where evidence is insufficient (one editor); (vii) independent and transparent methods using standardised assessment (one editor).
E4: independent assessment based on transparent methods of the evidence on a topic, including not only thorough search and systematic search of the evidence and the synthesis but also a standardized assessment of its quality and overall value … it’s systematic methods and their assessment is independent from conflict of interest in a broad sense…
Definition of original research
A question that emerged during one of the first interviews was the issue of a definition of original research. An editor asked interviewer to provide our definition of originality and explained that the answer depends on the definition of originality:
E9: This whole issue of originality depends on how one defines original research. If you do not define what original research is, then the question about whether systematic reviews are original studies is not fair.
Since there is no universally agreed definition of what constitutes an original research, we discussed this remark within our team and decided to ask all subsequent editors how they would define an original study. Ten editors that were interviewed subsequently provided their definition of original studies, which we divided into six categories (table 2). The original research was defined as: (i) one that brings new knowledge, data, information and conclusions; (ii) study that provides primary data analysis; (iii) definition according to the Boyer’s model of scholarship; (iv) study that has not been published before; (v) one which is replicable and done in accordance with criteria for the study in terms of scholarly inquiry and (vi) study where authors generate hypothesis, proceed in formal manner, choose methodology, share results, discuss limitations, provide conclusions and where people deserve authorship (table 2).
Definitions of original research provided by editors
Using systematic reviews for making clinical decisions
All editors were asked if clinical decisions should be based on SRs or primary studies. Eight editors responded that clinical decision should be based both on SRs and primary studies, six editors gave some advantage to SRs for making such decisions and one editor stated that decisions should be made on high quality evidence, regardless of the study type.
Additional relevant themes
At the end of the interview, editors were asked to freely express anything else they would like to add regarding originality of SRs. Among the most common comments were questions whether it matters at all how SRs are categorised in terms of originality as long as they are useful. Editors gave various comments about production and publication of SRs, proliferation of SRs and their quality (table 3).
Additional comments of editors regarding systematic reviews
Some of the practical aspects of this categorisation included eligibility of SRs as publications qualifying candidates for academic advancement or acceptance of SRs as a PhD thesis:
E1: I guess that depends on what you call original, and does it matter if you call it original or not—I don’t know does it matter. I guess it could matter to a PhD thesis committee or a promotion committee, but maybe they need to stop and think about what they are doing, and why they are doing it, and who it is that they are trying to train and what it means. In our institution we definitely have scholarship of synthesis, so meta-analyses and systematic reviews would get strong weight on our promotion committee. We have already moved in that direction.
Six editors indicated that they would consider SRs to be an eligible study design for a PhD thesis, one said that SRs should not be eligible for a PhD thesis and one editor considered that a candidate should prepare at least two SRs or a SR and a primary study if this study design will be considered for a PhD thesis:
E4: Somebody can oppose systematic review within a PhD thesis because they don’t know that people learn a lot by doing systematic reviews. They become better researchers if they do them.
Editor E6 indicated that there should not be restrictions regarding study designs while conducting academic thesis, because it is important to focus on learning outcomes and that the only goal of a thesis should not be putting hands on patients because we are moving towards electronic medical records anyway:
E6: We are not going to be doing individual data collection in 5 years or so. So we should be paying less attention to how we were doing things when we were younger, because then the only way to collect data was putting hands on a patient, or getting data from their chart, getting information by asking patient, observing, video-taping, etc., etc. But if now the data has already been collected, the old-fashioned way is very expensive method to gain knowledge. But my main point is whether that is our key competency, to collect an information directly from a patient.
Discussion
We followed up on the study of Meerpohl et al3 and found a similar proportion of editors of Core Clinical Journals who considered SRs original studies. Most editors published SRs, and a quarter of them published them in the section of their journal devoted to original studies. Interviews with editors indicated that various elements are considered when deciding whether a SR is an original study.
Eighty per cent of the surveyed editors considered SRs per se or only SRs with MA original research. This number is slightly higher compared with 71% of editors who considered that SRs are original research projects in 2009,3 but the difference is not statistically significant. There was no difference between the proportion of editors who considered SRs to be original research compared with the earlier study.3 These results indicate minimal changes of editors’ attitudes between these two studies in terms of the premise that SRs are original research.
We went beyond the initial study by Meerpohl et al3 with a qualitative study because we wanted to gather richer data, to give the editors an opportunity to explain their attitudes, and to get more details about what they consider original or non-original regarding SRs.
In the qualitative part of our study, many editors indicated that the concept of originality of SRs is still evolving and that there is a continuum of considerations to be made. One editor, who considered that SR is an original study, stated that a SR with a MA is still more novel than SR without MA. This indicates that certain editors equate originality of research with novelty. This also indicates that for some editors, there are no firm categories about novelty and originality of SRs. Instead, these are judged as a whole spectrum, where different characteristics or certain items of methodology can influence perception of a study.
Another editor, with the same opinion that SR with MA is an original study acknowledged that MA may not be justifiable and that this can be only determined after SR authors have already done a lot of work. In this case, the definition of originality of the study would depend on data that were found and not on the initial idea. This would make decisions about originality of a SR highly unpredictable; someone could embark on doing an original study and end up with a non-original study, depending on the results.
Furthermore, responses of editors during interviews indicate that there is a lot of fluidity in defining original research. There was no consensus among editors about what constitutes original research and what makes SRs original. Some editors quoted methodology and reproducibility and some originality of idea and level of novelty. Some insisted on analyses, presence of meta-analysis or another type of analysis, which would bring quantitative aspect to a qualitative summary in a SR. For one editor, SR without MA is semiquantitative and thus not original. However, if we duly take quantitative data as a paramount defining element of original research, then the whole field of qualitative studies would come under question as non-original.
Furthermore, some editors did not consider SRs or SRs without MA original research projects, because authors of such studies did not produce the data but relied on data collected by others. However, as other editors have mentioned, there are now a plethora of study designs in clinical medicine that include data which were not collected first hand. Studies that rely on electronic health records, or data collected via instruments, or any type of retrospective studies relying on data that somebody else collected, would then not be considered original according to these criteria.
Two editors cited Ernest Boyer’s model of scholarship as an argument why they do not consider SR to be original research.12 According to this model, introduced in 1990, there are four categories of scholarship, where original research is one category and the integration that involves synthesis of information is another category.12 However, one of the interviewed editors remarked that indeed there are different types of scholarship and therefore different types of original research.
Our survey also indicated that the majority (91%) of editors published SRs. The difference to the previously reported proportion (94%)3 is not significant. A quarter of editors published SRs in the original study section of the journal, which is a decline in comparison to earlier results that a third of editors published them in an original study section in 2009.3 However, it also has to be emphasised that some editors remarked in interviews that they find it completely irrelevant in which section a SR is published, because they have thematic sections where they publish manuscript of any study design, including SRs. Therefore, the finding that fewer SRs are published in sections devoted to original studies may simply be an indication that more journals organise manuscript in thematic sections and not according to the perceived originality of a contribution.
Few manuscripts so far were devoted to the considerations of originality of SRs. Aveyard and Sharp13 postulated that SRs are ‘original empirical research’ because they (quote): ‘review, evaluate and synthesise all the available primary data, which can be either quantitative or qualitative’. One editor in our qualitative study remarked that it does not matter whether a SR is considered original research, as long as it is valuable. This was also pointed out before by Biondi-Zoccai et al, who considered that the main criteria to judge a SR should be its novelty and usefulness and not whether a SR is original/primary or secondary research.14
There may be practical aspects in considerations whether a SR is an original study or not, such as during evaluating candidates for academic promotion, or allowing students to use SR as a study design in their academic thesis. Although these topics were not the subject of our study, several editors indicated that they would be in favour of recognising SRs both as studies that can be counted for academic promotion and for conducting PhD thesis. There were also discrepant opinions regarding SRs within PhD theses as well as considerations that acceptability of a SR for PhD thesis may depend on a research field. It has been shown recently that the topic of acceptability of a SR for PhD theses is debated in the academic community as well; about half of surveyed individuals in charge of European PhD programmes indicated that SRs are accepted as a study design in their schools.4
In addition to analysing editors' opinions, we also assessed SRs that the target journals published in 2017. Overwhelming majority of the analysed journals (93%) published at least one SR in 2017, and among the remaining eight journals, one had instructions for authors regarding submission of SRs. Six editors indicated that their journal does not publish SRs, but our analysis indicated that two of those six journals actually did publish SRs in year 2017.
Strengths and limitations
One strength of this study is its high response rate (62%). Furthermore, the proportion of editors with different answers to the question about whether SRs are original studies was very similar among editors who participated in the survey and editors who participated in follow-up interviews, indicating the editors were equally keen to participate in the interview regarding their consideration of SRs as original studies or not. Therefore, we can argue that interviews did not suffer from selection bias in terms of including only editors with uniform opinions regarding SRs being or not being original studies. Another strength of this study is the addition of a qualitative data collection to the study design as compared with Meerpohl et al.3 Through the interviews, we obtained more nuanced responses regarding editors’ opinion about whether SRs are original studies and what is an original study anyway. These additional data provide rich insight into the reasoning of editors and may provide inspiration for further studies and actions in this field. Certain tangible benefits can depend on whether a SR is regarded as original research or not. Consequently, it would be beneficial if relevant organisations, such as learnt societies and associations of editors, would address the issue and provide some guidance.
Our study had several limitations. We focused on the editors of PubMed Core Clinical Journals, which is a limited sample to begin with. It is possible that different responses would have been obtained if we had surveyed a broader sample of editors. Our decision to use this cohort of journals was guided by the preceding study by Meerpohl et al,3 to serve as a historical control.
Although 9 years have passed between the two studies, it is possible that some involved editors did not change over time and that they were included both in the first and in the second study. In our study, we did not include any questions to verify this. The list of Core Clinical Journals was similar between the two analysed years; there were 13 journals that were replaced by another one compared with the first study. However, none of the contacted editors mentioned that they participated in the first study.
We used PubMed to search for published SRs and MAs. If those were not indexed correctly in PubMed, we could have missed some. For this reason, we hand-searched all journals for which we did not find any SRs or MAs.
Another relevant consideration is the definition of a SR. We did not give editors any a priori definition of what could be considered a SR, and not a single editor asked which definition of a SR we used in the study. Presently, there is no consensus definition of a SR, and which methodology, or characteristics a study should have in order to be considered a SR. For example, it has been suggested that a study should not be called a SR if the authors searched only one database or if there is only one author.15
In conclusion, compared with results obtained in 2009, we found a similar proportion of editors of core clinical journals who consider that SRs are original research and who had conditional acceptance of SRs’ originality. Interviews with editors revealed that there is no uniform approach to defining what makes a SR (or any study) a piece of original research and that these concepts of originality of research are evolving. Editorial organisations, which set standards of publishing, should address this issue.
Footnotes
Contributorship Statement Study design: JJM, EvE, FH, AM, LP. Data collection: MKM, LP. Data analysis: MKM, JJM, EvE, FH, AM, LP. Manuscript writing: JJM, EvE, FH, AM, LP. Approval of the final version of the manuscript: MKM, JJM, EvE, FH, AM, LP. Agree to be accountable for the work: MKM, JJM, EvE, FH, AM, LP.
Funding statement This study was conducted as a part of the research grant ‘Professionalism in Health Care’, funded by the Croatian Science Foundation (Grant No. IP-2014-09-7672).
Competing interests None declared.
Patient and public involvement statement Not required.
Ethics approval The study protocol was approved by the Ethics Committee of the University of Split School of Medicine. Potential participants received information about the study via an email and they were informed that their response to the email will be considered an informed consent to participate in the survey. Consent for interviews was also obtained via email.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement All data generated within this study are available from the corresponding author on request.