Article Text
PDF
Abstract
Introduction One of the most conflicting methodological issues when conducting an overview is the overlap of primary studies across systematic reviews (SRs). Overlap in the pooled effect estimates across SRs may lead to overly precise effect estimates in the overview. SRs that focus on exercise-related interventions are often included in overviews aimed at grouping and determining the effectiveness of various interventions for managing specific health conditions. The aim of this systematic methodological review is to describe the strategies used by authors of overviews focusing on exercise-related interventions to manage the overlap of primary studies.
Methods and analysis A comprehensive search strategy has been developed for different databases and their platforms. The databases to be consulted will be MEDLINE (Ovid), Embase (Ovid), The Cochrane Database of Systematic Reviews (Cochrane Library) and Epistemonikos. Two reviewers will independently screen the records identified through the search strategy and extract the information from the included overviews. The frequency and the type of overlap management strategies of the primary studies included in the SRs will be considered as the main outcome. In addition, the recognition of the lack of use of any overlap management strategy and the congruence between planning and conducting the overview focusing on overlap management strategies will be assessed. A subgroup analysis will be carried out according to the journal impact factor, year of publication and compliance with the Preferred Reporting Items for Overviews of Reviews statement.
Ethics and dissemination This study will not involve human subjects and therefore does not require ethics committee approval. However, the conduct and reporting of the findings of this review will be conducted in a rigorous, systematic and transparent manner, which relates to research ethics.
The findings of this review will be presented at scientific conferences and published as one or more studies in peer-review scientific journals related to rehabilitation or research methods.
- REHABILITATION MEDICINE
- STATISTICS & RESEARCH METHODS
- SPORTS MEDICINE
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
This methodological review will use a systematic approach to describe the strategies used to manage the overlap of primary studies in exercise-related overviews.
This review will conduct a sensitive search of MEDLINE (Ovid), Embase (Ovid), The Cochrane Database of Systematic Reviews (Cochrane Library), Epistemonikos databases and registers of evidence synthesis study protocols to identify exercise-related overviews.
This review will be one of the first to assess the quality of synthesis reports using the recently published Preferred Reporting Items for Overviews of Reviews statement.
A potential limitation of this review is that the overviews identified do not report in detail the methodology used to deal with the overlap.
Introduction
The number of published primary studies covering a similar research question has grown exponentially,1 limiting the possibility of keeping current on a specific topic.2 It is in this context that systematic reviews (SRs) with and without meta-analyses (MAs) of interventions can offer a solution,3 as in addition to synthesising the available evidence, they use reproducible methods to assess the risk of bias in the primary studies included.4
However, the number of published SRs and MAs has increased steadily in recent years despite repositories of SRs and MAs protocol registries5–7 seeking to reduce duplication or redundancy of SR research.8 9
The growth in research evidence makes it difficult for clinicians to stay current and use interventions based on the best available evidence.10 11 Overviews, also known as umbrella reviews, can help clinicians make sense of duplicated SRs on the same topic. Overviews synthesise information and data from similar SRs to guide health decision-making.12
When conducting an overview, one of the most conflicting methodological issues is the overlap of primary studies across SRs with or without MAs.13 When one or more primary studies are included in two or more SRs with or without MAs, the results and conclusions of the overviews may be biased. Overlapping data from the same primary studies may include overlapping in risk of bias and certainty of evidence assessments (eg, Grading of Recommendations, Assessment, Development and Evaluations (GRADE)) or overlapping in the determination of the effect of a specific intervention and other MA outcomes such as heterogeneity (eg, I2).14 15 Overlapping pooled effect estimates across SRs may lead to overly precise effect estimates in the overview.16
Methodological studies from different medical fields reported that authors of overviews rarely assess the overlap of primary studies.13 17 However, these studies have not conducted an exhaustive search of overviews oriented to a specific health problem, specialty or discipline,13 17 as they have only searched an electronic database17 and included heterogeneous overviews concerning the research questions addressed.13 17
SRs that focus on exercise-related interventions are often included in overviews aimed at grouping and determining the effectiveness of various interventions to manage of specific health conditions. Assessing the application of overlap management strategies in overviews focused on exercise-related interventions could contribute to identifying specific or differentiating aspects. This could be because the concept of exercise needs to be understood.18 In addition, the existence of multiple interventions related to exercise due to their different modalities (eg, continuous aerobic, intervallic aerobic, resistance exercise) and dosage (eg, frequency, intensity, time and type) could result in a particular need to manage the overlapping of primary studies data.
Considering the recently published Preferred Reporting Items for Overviews of Reviews (PRIOR) statement, which incorporates the need to report on the handling of overlapping primary studies, both in the data collection phase and in the presentation of results, to improve and standardise the reporting of overviews,19 this systematic methodological review aims to find out how often strategies for handling overlapping data from primary studies are used in SRs considered by syntheses focusing on exercise-related interventions in different health conditions. Second, it aims to describe the overlap strategies used, the authors’ acknowledgement of not using any overlap management strategies as a methodological weakness and the congruence between the protocol and the final published summary in terms of overlap management. These findings are intended to be analysed according to the impact factor of the journal in which the overviews were published, the year of publication of the overview and compliance with the PRIOR statement.
Materials and methods
The protocol of this methodological review is reported following the Preferred Reporting Items for Systematic Review and Meta-Analysis (PRISMA) Protocols20 (see checklist in online supplemental file 1). The start of this study with the preliminary design of the search strategies began in June 2022, and this methodological review is expected to be finalised in April 2023.
Supplemental material
Eligibility criteria
Studies will be eligible if they meet the following inclusion criteria for study design and population. Given the purpose of this methodological review, the intervention and outcomes will not determine the inclusion of studies, and the comparator or control intervention will not be considered as it is not applicable.
Study design
We will include overviews that consider SRs with or without MAs, without distinction of the methodological design of the primary studies included. The definition of SR adopted by the authors of the overviews21 will not be considered an eligibility criterion. Overviews that include primary studies not considered in the selected SRs will not be excluded.
For this review, an overview will be understood as any study22 that:
Synthesises general information, methods and outcome data from SRs.
Makes explicit the inclusion and exclusion criteria for SRs.
Includes an explicit search strategy for the studies.
Examines the effectiveness of health interventions.
Overviews that are conducted using a ‘rapid review’ methodology23 will be excluded, as the time frame in which they are conducted to answer urgent questions will likely not consider the overlap of the primary studies included in the SRs. In addition, overviews published only as abstracts in conference proceedings will be excluded.
Population
Overviews include SRs that have considered primary studies that have studied any exercise-based intervention, where exercise is understood as a subcategory of physical activity that is planned, structured, repetitive and purposefully focused on improving or maintaining one or more components of physical fitness,18 will be included. These overviews may include only SRs related to exercise-based interventions or other non-exercise interventions as well.
Overviews that consider exercise training-based interventions that are applied both preventively and in the recovery phase and that are delivered either as a stand-alone intervention, as part of a comprehensive rehabilitation programme, or as an adjunct to other medical interventions, in which exercise is the main component, will be included.
Furthermore, the inclusion of overviews will not be limited to the context in which the exercise-based interventions were applied (eg, primary care, specialised care) or whether they were delivered face to face, remotely or mixed.
Overviews that include SRs that consider physical activity as an intervention, understood as ‘any bodily movement produced by skeletal muscles that require energy expenditure’ according to the WHO,24 will be excluded. Therefore, to differentiate between exercise-based and physical activity-based interventions, it will be considered that the exercise, together with its structure and dosage (frequency, intensity, time and type), must be prescribed or delivered by a professional related to physical training/rehabilitation.
Intervention
Our goal is to identify the strategies used to manage data from overlapping primary studies selected by SRs included in overviews. Strategies should be specified in the main text of the overviews and may be in the methods or results section, taking all possible methodological strategies that address overlap in the primary study data into consideration. Strategies addressing overlap can address different objectives,16 such as quantifying the overlap13 25 (eg, corrected covered area (CCA)), visually presenting overlap26 (eg, matrix, Venn and Euler diagrams) and avoiding duplicate information by using one or more decision algorithms27 (eg, quality of SRs, comprehensive SRs, up-to-datedness of SRs, statistical methods).
Outcomes
The presence and the type of overlap management strategies of the primary studies included in the SRs will be considered as the main outcome.
In addition, two aspects will be regarded as secondary outcomes:
Acknowledgement of the limitation in the conducting of the overview: we will assess whether the overview’s authors that did not include any strategy for managing primary study overlap considered this limitation in their discussion or conclusion.
Congruence between planning and conducting the overview: we will review available registry entries (eg, PROSPERO) or published protocols in scientific journals (eg, BMC Systematic Reviews Journal, BMJ Open) of all overviews included in this SR to determine whether management of primary study overlap had been considered in the planning phase of the overviews and to determine the congruence between the methods proposed in the protocols and those ultimately used.
Search strategy
A search strategy translated to different databases and their platforms will be developed using a controlled vocabulary (MeSH and Emtree) and text words. The search strategy will include a search filter published in 2016 by Lunny et al,28 which is validated to identify overviews in MEDLINE-Ovid with 93% sensitivity (95% CI 87% to 96%). The search strategy constructed for this database and platform is shown in table 1, which will be used as a basis for adapting the search strategies of the other databases and search platforms.
Search strategy for MEDLINE using the Ovid platform
The databases to be consulted will be MEDLINE (Ovid), Embase (Ovid), The Cochrane Database of Systematic Reviews (Cochrane Library) and Epistemonikos. In addition, we will search protocol registries of SRs such as the International Platform of Registered Systematic Review and Meta-analysis Protocols (INPLASY) (https://inplasy.com/), PROSPERO (https://www.crd.york.ac.uk/PROSPERO/) and OSF Registries (https://osf.io/registries), and follow-up protocols published in scientific journals (eg, BMC Systematic Reviews Journal, BMJ Open). All search resources will be reviewed from inception to June 2022.
We will also review the references of the studies included in this review to identify overviews that may not have been identified by our electronic search strategy.
We will include all languages in our search and will not be limited by the date of publication/indexing in databases.
Study selection
Two reviewers (RG-A and RT-C) will independently and blindly screen the records identified through the search strategy. In the first instance, the titles and abstracts will be evaluated for inclusion. Then the full texts of the records qualified as potentially eligible, and those that did not present sufficient information to be excluded, will be checked for compliance with all eligibility criteria. A pilot test will be conducted with 50 studies to adjust the clarity of the eligibility criteria.
The Rayyan application29 will be used for this stage. Disagreements will be resolved by consensus, or ultimately by a third-party reviewer (RA-E or PS).
Data extraction
The extraction of information from the included overviews will also be carried out independently and blindly by two reviewers (RG-A and RT-C). For this, a standardised extraction form will be used which will contain data related to the basic information of the overviews:
Title.
Journal name.
Year of publication.
Name of the authors.
Objectives of SRs.
Number of SRs included
Number of primary studies included
Methodological aspects: databases consulted, date of search, type of synthesis of results (narrative, MA or both) and instruments for assessing the risk of bias/methodological quality of the SRs included.
Data will be extracted to respond to the findings of this methodological review:
Type of overlap management strategy:
Quantifying overlap: for example, CCA.
Visual presentation of the overlap: for example, matrix, Venn or Euler diagrams.
Strategies to avoid duplicate information: for example, algorithms based on the quality of SRs, comprehensive SRs, up to datedness of SRs, statistical methods such as sensitivity analyses, or a combination of two or more criteria: for example, Jadad algorithm.30
Step in the conducting of the overview where the strategy has been deployed or used: for example, data extraction step, synthesis step.
Level at which the strategies were applied: that is, whether it was at the level of SR or reported outcomes.16
In addition, the impact factor of the journal at the time of publication of the overviews will be recorded. This will be extracted from the journals official websites or from Web of Science (https://www.webofscience.com/).
If more than one record or publication exists for an overview, the most recent version will be considered for analysis. The data extraction form will be tested with 10 studies to assess its completeness and adjusted if necessary. Disagreements will be resolved by consensus or ultimately by a third-party reviewer (RAA-E or PS).
Risk of bias and reporting quality assessment
This methodological review assesses one aspect that may affect the methodological quality or risk of bias of the overviews. The assessment of the overall risk of bias of the overviews is not an objective of this study.
Two independent reviewers will assess the quality of the overviews’ reporting by considering compliance with the PRIOR statement.19 Disagreements will be resolved by consensus, or ultimately by a third reviewer.
Strategy for data synthesis
The results of the study selection will be schematised through a PRISMA-type flow chart.31 In addition, the characteristics of the overviews included, as well as data related to the primary and secondary outcomes, will be presented in narrative form and through tables and figures.
Descriptive statistics will be used to quantify the number of overviews using overlap strategies, whether the strategies were used at the level of the SRs or the level of each reported outcome. In addition, these results will be organised by the type of strategy used.
We will also assess whether the overlapping strategy successfully resolved overlap at the following steps: risk of bias assessment, the certainty of the evidence (eg, GRADE) and the synthesis step. The resolution of the overlap will be considered to have been achieved when the authors manage to avoid double/multiple counting of information from the primary studies.
Analysis of subgroups
Differences in the percentage of overviews that include overlap management strategies, the type of strategies used, the recognition of the weakness of not using any strategy and the congruence between the protocols and the methodology finally used among journals with and without impact factor will be assessed. In addition, this analysis will be repeated for impact factor journals, considering the median or quartiles of the impact factor of the journals at the time of publication of the overviews to form 2 and 4 groups, respectively, depending on the number of overviews included in this methodological review.
In addition, analysis will be carried out by subgroup according to the year of publication of the overviews, compliance with the items considered in the PRIOR statement and whether or not the overviews were published in the Cochrane Database of Systematic Reviews.
Patient and public involvement
Because this protocol is about conducting a methodological review, both patients and the public were not involved. This methodological review is intended to be of use to researchers of evidence synthesis studies.
Discussion
This methodological review will provide a comprehensive and exhaustive summary of the frequency of use of strategies for managing primary study overlap across SRs included in overviews focused on exercise-related interventions in different health conditions. It will also provide insight into the strategies used to quantify and visualise overlap, as well as those used to avoid duplicate data.
On the other hand, the findings of this review will tell us whether the authors of the overviews recognised the failure to include some strategy for handling overlap as a methodological weakness, taking into account that the greater the degree of overlap, the more falsely precise the estimates of the effects of the interventions.16 In addition, the congruence between the strategies used by the published overviews and their respective protocols will be revealed. To our knowledge, the latter two aspects have not been addressed at the overview level by other studies before.
Finally, all analyses will be performed by subgroup of overviews, considering the impact factor of the journal and the year of publication. Although the PRIOR statement was recently published,19 assessing compliance in the reporting of overviews, and its relation to the use of strategies for the management of overlapping primary studies, could expose the shortcomings and weaknesses that have been committed so far.
Future research
To continue this line of research, different overlapping data management strategies should be applied to all, or a representative sample, of the overviews identified by this methodological review. This could empirically test the benefits and limitations of using any strategy.
Ethics and dissemination
This study will not involve human subjects and therefore does not require ethics committee approval. However, the conduct and reporting of the findings of this review will be conducted in a rigorous, systematic and transparent manner, which relates to research ethics.
The findings of this review will be presented at scientific conferences and published as one or more studies in peer-review scientific journals related to rehabilitation, healthcare or methodological aspects associated with evidence synthesis.
Ethics statements
Patient consent for publication
Acknowledgments
This work has been carried out within the framework of the PhD programme Metodología de la Investigación Biomédica y Salud Pública of the Universitat Autònoma de Barcelona.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Twitter @ruvistay, @PieperDawid, @carole_lunny, @klgorodtorres, @pame_seron
Correction notice This article has been corrected since it was first published. The acknowledgements section has been updated.
Contributors RG-A, DP, CL and PS contributed to the conception and design of the study. RG-A developed the search strategies, which were reviewed by RT-C and RA-E. RG-A, DP, CL and PS designed the data analysis. RG-A drafted the manuscript, and all authors read it critically.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.