Article Text
PDF
Abstract
Introduction Observational studies are fraught with several biases including reverse causation and residual confounding. Overview of reviews of observational studies (ie, umbrella reviews) synthesise systematic reviews with or without meta-analyses of cross-sectional, case-control and cohort studies, and may also aid in the grading of the credibility of reported associations. The number of published umbrella reviews has been increasing. Recently, a reporting guideline for overviews of reviews of healthcare interventions (Preferred Reporting Items for Overviews of Reviews (PRIOR)) was published, but the field lacks reporting guidelines for umbrella reviews of observational studies. Our aim is to develop a reporting guideline for umbrella reviews on cross-sectional, case-control and cohort studies assessing epidemiological associations.
Methods and analysis We will adhere to established guidance and prepare a PRIOR extension for systematic reviews of cross-sectional, case-control and cohort studies testing epidemiological associations between an exposure and an outcome, namely Preferred Reporting Items for Umbrella Reviews of Cross-sectional, Case-control and Cohort studies (PRIUR-CCC). Step 1 will be the project launch to identify stakeholders. Step 2 will be a literature review of available guidance to conduct umbrella reviews. Step 3 will be an online Delphi study sampling 100 participants among authors and editors of umbrella reviews. Step 4 will encompass the finalisation of PRIUR-CCC statement, including a checklist, a flow diagram, explanation and elaboration document. Deliverables will be (i) identifying stakeholders to involve according to relevant expertise and end-user groups, with an equity, diversity and inclusion lens; (ii) completing a narrative review of methodological guidance on how to conduct umbrella reviews, a narrative review of methodology and reporting in published umbrella reviews and preparing an initial PRIUR-CCC checklist for Delphi study round 1; (iii) preparing a PRIUR-CCC checklist with guidance after Delphi study; (iv) publishing and disseminating PRIUR-CCC statement.
Ethics and dissemination PRIUR-CCC has been approved by The Ottawa Health Science Network Research Ethics Board and has obtained consent (20220639-01H). Participants to step 3 will give informed consent. PRIUR-CCC steps will be published in a peer-reviewed journal and will guide reporting of umbrella reviews on epidemiological associations.
- protocols & guidelines
- statistics & research methods
- epidemiologic studies
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
STRENGTHS AND LIMITATIONS OF THIS STUDY
This protocol for reporting guidance of umbrella reviews of epidemiological associations is needed to address specific reporting challenges of observational studies.
This protocol follows the guidance for reporting checklist, which is standard in the field.
This study is limited from attrition rates, which is typical of Delphi studies.
Introduction
There is evidence that the number of systematic reviews and meta-analyses in the literature has increased geometrically over the past two decades.1 2 Due to the increasing number of systematic reviews and meta-analyses on a given topic over the years, the field of knowledge synthesis has developed systematic reviews of systematic reviews, also called ‘reviews of reviews’, ‘overviews of (systematic) reviews’, ‘meta-reviews’ or ‘umbrella reviews’.3–8 Umbrella reviews can include interventional studies or observational studies.9 Overviews of reviews and umbrella reviews ideally aim to provide a comprehensive and systematic synthesis following the steps of a systematic review (ie, literature search, methodological quality appraisal, quantitative analysis where feasible and appropriate, etc) with systematic reviews as the unit of analysis.2 9 10 The field has seen a sharp increase in the number of published overviews of reviews and umbrella reviews over the past decade. For example, just limiting to umbrella reviews, approximately 56 were published in 2010, while 560 (10 times increase in yearly publications) were published in 2021 (PubMed (umbrella review)).
There is important variability between and within the approach of overviews of reviews and umbrella reviews, making results hardly comparable.11–13 This heterogeneity is not surprising, considering the large heterogeneity in the conception and implementation of both systematic reviews and meta-analyses.14 15 The methodology used to conduct systematic reviews and meta-analyses, as well as the quality of reporting, can affect conclusions and potentially lead to misleading interpretation of findings, misinforming policy makers, professional organisations and regulatory bodies, practitioners, patients, the public and other stakeholders. The quality and credibility of evidence synthesis efforts is also largely based on the quality of the credibility of their unit of inclusion (ie, individual studies). There is consensus that randomised controlled trials start from a higher credibility in the evidence-based medicine pyramid, and lower credibility is assigned to observational studies, which are more prone to bias. Interventional studies are not free from limitations, but in general experimental designs such as randomised controlled trials can protect from a number of biases, such as confounding by indication, or reverse causality. By contrast, observational evidence is prone to these and other biases, including excess of significance bias.16 Among observational studies, different study designs are adapted depending on the research question to be answered. For instance, studies measuring prognostic accuracy or prediction models are typically cohort studies that need internal development of the model, internal and external validation, calibration and accuracy measures.17 Cross-sectional studies can instead be used to investigate biomarkers or diagnostic accuracy of a given construct/test, or the prevalence of a disease. Other research questions, and typically epidemiological associations between two factors are generally explored with cohort, case-control and cross-sectional studies.18 19 More specifically, among studies investigating epidemiological associations, cross-sectional studies are typically used to measure the association between two factors, neglecting the direction of such association, while case-control and cohort studies are frequently used to measure associations between a construct of interest, and putative risk factors,1 14 20–23 or its outcomes,24 25 with the exposure occurring before the outcome. Research questions for which umbrella reviews of observational studies can be used are reported in table 1.
Frequent research questions for which overviews of reviews of observational studies are used
Reporting guidelines, defined as ‘ checklist, flow diagram or explicit text to guide authors in reporting a specific type of research, developed using explicit methodology’,26 can be useful to improve the transparency, quality and reporting of individual studies, reviews or umbrella reviews. Interventional and observational evidence pose different methodological quality and reporting challenges, which are reflected from different reporting guidance for the two study designs. For observational evidence, several checklists of essential information to be reported in a paper are available for individual studies, systematic reviews and meta-analyses. For observational studies, the Enhancing the QUAlity and Transparency Of health Research (EQUATOR) Network has disseminated Standards for Reporting Diagnostic accuracy studies,27 27 Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis28 and Strengthening the Reporting of Observational Studies in Epidemiology.29 For systematic reviews, the Preferred Reporting Items for Systematic Reviews and Meta-Analyses 2020 (PRISMA 2020)30 is a broad guide inclusive of a range of primary study designs, and more specific statements are available, that is, the PRISMA for Diagnostic Test Accuracy,31 and PRISMA for reviews including harms outcomes,32 or the guidance on conducting systematic reviews and meta-analyses of observational studies on aetiology33 (table 2). The different checklists addressing different study designs and research questions well reflect their different methodological challenges, from original research to evidence synthesis.
Key reporting guidelines across different research questions that can be addressed with observational studies
Regarding overview of reviews and umbrella reviews, virtually all reporting checklists proposed so far have focused on interventional evidence. The first proposed checklist34 was based on A MeaSurement Tool to Assess systematic Reviews quality assessment tool (AMSTAR)35 and Cochrane guidance.36 The same year, one further checklist for overviews of reviews37 merged evidence from PRISMA for abstracts,38 the Preferred reporting items for overviews of systematic reviews (PRIO) checklist39 and the Overview Quality Assessment Questionnaire.40 Then, a checklist for overviews of reviews41 was developed based on Cochrane recommendations,36 and the older versions of PRISMA42 (and not PRISMA 202030), and AMSTAR35 43 (and not AMSTAR 244). Later, checklist was developed for systematic reviews of reviews including harms, called PRIO,39 based on an older version of PRISMA,42 PRISMA harms32 and PRISMA for systematic review protocols.45 The same group also published a checklist for abstract of overviews of reviews of healthcare interventions.46 Recently, the Preferred Reporting Items for Systematic Reviews of Systematic Reviews/Meta-analyses (PRIOR) statement has been published to guide reporting of overviews of reviews of health interventions,3 adhering to EQUATOR guidance.26 In PRIOR protocol,3 authors acknowledged that several relevant sources exist that provide guidance on overviews of reviews or umbrella reviews, but they did not adhere to guidance endorsed by the EQUATOR Network.
Regarding umbrella reviews (ie, observational evidence investigating epidemiological associations), to the best of our knowledge no EQUATOR-adherent guidance has been developed, registered with or disseminated by EQUATOR group, nor any specific checklist has been previously proposed. Given that PRIOR focuses on interventional evidence, and that different reporting guidelines are needed for observational evidence (cross-sectional, case-control, cohort studies) on epidemiological associations versus interventional evidence, the aim of this project is to develop evidence-based and agreement-based guideline PRIOR-extension for reporting umbrella reviews (ie, cross-sectional, case-control, cohort studies testing epidemiological associations), adhering to established guidance26 and building on PRIOR statement.3 Specifically, this project will yield a Preferred Reporting Items for Umbrella Reviews of Cross-sectional, Case-control and Cohort studies (PRIUR-CCC), which will be published in a peer-reviewed journal and available via a dedicated website.
Methods and analysis
Transparency statement
We have also submitted this protocol to The Ottawa Health Science Network- Research Ethics Board and have obtained consent (20220639-01H). All participants to the Delphi survey will give written informed consent, which they will be able to withdraw at any time (yet anonymous responses cannot be withdrawn). This protocol is publicly available at medrxiv.org (MEDRXIV/2022/283572).
All study data and materials will be publicly available.
Study design
This study will follow EQUATOR guidance for developing reporting checklists,26 and will be composed on four steps, namely project launch, literature review, Delphi survey and guideline statement preparation (figure 1).
Development of PRIUR-CCC statement flow diagram. EQUATOR, Enhancing the QUAlity and Transparency Of health Research.
Project launch
Project launch will consist of reaching an agreement on roles and responsibilities of core team members (eg, identifying stakeholders that will participate in the Delphi study). A core group of researchers have prepared the protocol of the project. These authors will be the core team of the project, and have extensive record of umbrella reviews on cross-sectional, case-control and cohort studies investigating epidemiological associations. The project’s day-to-day steps and its finalisation will be responsibility of shared first authors, and last author. All other authors will contribute to the protocol, literature review, initial set of checklist items, Delphi survey and final PRIUR-CCC statement. The identification and involvement of stakeholders will follow the Practical Guidance for Involving Stakeholders in Health Research from the Multi Stakeholder Engagement Consortium.47 Reporting of involvement of patients and public will adhere to Guidance for Reporting Involvement of Patients and the Public 2 (GRIPP2).48
Literature narrative review
We will review the systematic reviews conducted in the context of PRIOR development regarding available guidance to conduct umbrella reviews. Then, since two systematic reviews of umbrella reviews have already been conducted by the members of this project,2 12 those two systematic reviews will be used to assess methodology and reporting of included umbrella reviews. Given that PRIUR-CCC will not focus on diagnostic test accuracy and prediction models, we will not cover those study designs as they are out of scope for PRIUR-CCC. After having reviewed PRIOR documentation and having reviewed the two previous systematic reviews of umbrella reviews, we will extract key methodological factors from identified umbrella reviews, and publish a review of methodology and reporting of previous umbrella reviews. The findings of this review will be used to identify PRIOR domains and items where changes are needed, and to produce an initial PRIOR checklist to start the Delphi study with.
Delphi survey
We will conduct a Delphi study. Delphi studies use social science survey techniques to structure communication between participants in order to drive consensus and make a collective decision.49 Typically, Delphi studies use several rounds of surveys in which participants, vote on issues. Between rounds, results of voting are aggregated and anonymised. They are then presented back to participants along with their own individual scores, and feedback on why others voted as they did.50 51 Among others, two strengths of this method include allowing for effective communication, limiting direct confrontation between individuals and giving participants the opportunity to consider the group’s thoughts and to compare and adjust their own score in the next round. Participants: we will aim to include 100 participants in Delphi study round 1, as done in PRIOR development.3 Participants to the Delphi study will meet any of the following criteria: (a) they have publication track of umbrella reviews in various fields, or (b) have publication track in Delphi surveys or (c) have publication track of reporting checklists of cross-sectional, case-control and cohort studies, or editors of peer-reviewed journals that published umbrella reviews on observational evidence or that have interest in umbrella reviews, or (d) funders of research or meta-research, or (e) practitioners, (f) policy makers, (g) evidence synthesis associations (eg, Cochrane, Campbell collaboration, Joanna Briggs Institute, others).3 Participants will be recruited via a two-step process. First, we have established a core group of participants based on criteria (a) or (b) or (c) described above (project launch), and past solid collaboration track record, that are authoring the present protocol. Second, additional stakeholders will be invited, according to (a) to (h) criteria above. Delphi study methods: this Delphi study will consist of three rounds. For the first two Delphi rounds, we will ask participants to complete an online survey which will be administered using Google Forms, structuring forms based on purpose-built platform for Delphi survey development and management.52 The third round will consist of a facilitated online consensus group meeting (using Zoom53).
Round 1
Age, gender, geographical area (https://www.who.int/countries) and stakeholders group will be collected anonymously. Similar to what has been done for scoping reviews,54 we will build on existing PRIOR statement, which parallels PRISMA 202030 statement, for consistency and continuity with existing established reporting guidance for evidence syntheses. All participants will be asked what items of PRIOR3 will have to (i) remain unchanged, (ii) what will have to be changed and how, (iii) what will have to be removed (three answer options, one possible choice). In addition, participants will have the possibility to propose new items. As a starting point, the core team composed of authors of this protocol will provide a set of suggested items for participants to vote on. This starting set of items will be built based on experience in umbrella reviews. Participants will respond indicating their agreement/disagreement to have each item included in the PRISMA-CCC reporting guideline. Participants will be provided with a free-text box to fill in with additional comments to explain why they voted how they did or to propose wording amendments to the item, or new items. To decide to keep or remove a PRIOR item, or to add a new item, will require a minimum of 80% consensus among participants (based on findings from a systematic review of Delphi studies55). Items where an 80% consensus has not been reached, as well as changes to PRIOR items and new items will be voted on again in round 2. We anticipate the survey will take about 20 min to complete and will provide participants with a 3-week window to take part, with reminders sent after 1 and 2 weeks, respectively. We will pilot test the survey among the authors of the present protocol.
Round 2
All participants who completed round 1 of the Delphi will be re-invited to take part in round 2. Items which achieved consensus in round 1 will be shared with the participants. We will then ask participants to re-vote on any items that did not reach consensus or that were newly suggested. When re-voting on items that appeared in round 1, participants will also be provided with all comments provided by participants to justify their responses. Again, an 80% of consensus will be used to determine what items to include/exclude from the PRIUR-CCC guideline. If a new proposed item will be overlapping with existing items, Delphi moderator will add a comment to the new proposed item pointing to existing overlapping item.
Round 3
The core group and a purposeful selection of participants will be invited to round 3. We will aim to ensure representation from each of our geographically diverse institutions, each of the stakeholder groups and demographic variables including gender. Based on previous Delphi surveys and feasibility, we will invite no more than 30 participants in total to ensure feasibility to round 3.56 57 Round 3 will be moderated by core group members authoring the protocol, who will rotate every two items. We will present all participants with the results (ie, frequency of responses for each item, comments, changes and new items) of round 2 of the Delphi prior to the meeting and summarise these again briefly at the start of the consensus meeting. Participants will have the opportunity to discuss outstanding items one-by-one. They will then be asked to vote anonymously using real-time voting technology available via Zoom on each of these items. While diverse time zones will present challenges in other ways a virtual meeting may foster equity, diversity and inclusion of participants who might otherwise not have had funding or capacity to travel to an in-person event.
Analysis
For completers and drop-outs, demographics and responses will be presented using descriptive statistics using SPSS28. We will identify which items have and have not reached consensus for inclusion or exclusion based on our definition of 80% agreement and report this information for each Delphi round. The list of items identified for inclusion in PRIUR-CCC will be collated after round 3 and we will report the outcomes of participants ranking of these items in a table.
Timeline
The project has not started yet, and will be started in January 2024 and completed by December 2024.
Guidance statement
Co-first and last authors of this protocol will prepare the first draft of final PRIUR-CCC statement, that will then be approved after reiteration with other authors. This PRIUR-CCC statement will include the report of the whole project, the PRIUR-CCC checklist, the PRIUR-CCC flow diagram and an explanatory document that will inform on how to use PRIUR-CCC, with examples. PRIUR-CCC statement will be published in peer-reviewed journals, and on dedicated platforms. The Delphi process reporting will be informed by the Conducting and REporting of DElphi Studies checklist58 (table 3).
CREDES checklist for survey studies
Ethics and dissemination
PRIUR-CCC has been approved by The Ottawa Health Science Network-Research Ethics Board and have obtained consent (20220639-01H). Participants to step 3 will give informed consent. PRIUR-CCC steps will be published in a peer-reviewed journal and will guide reporting of umbrella reviews on epidemiological associations.
Patient and public involvement
Reporting of involvement of patients and public will adhere to GRIPP2.48 All authors completing all three rounds of Delphi process and the core group will be invited to review and finally coauthor the publication.
Discussion
This protocol for reporting guidance of umbrella reviews of epidemiological associations is needed to address specific reporting challenges of observational studies. PRIUR-CCC will provide a reporting framework to guide future umbrella reviews of observational studies assessing epidemiological associations, that can be used from researchers, reviewers, funders and editors to evaluate the transparency and quality of reporting of umbrella reviews, across different research questions. It will follow the guidance for reporting checklist, which are standard in the field. We propose that having reporting guidelines is of crucial relevance when included studies follow an observational design, with high baseline risk of confounding factors and numerous sources of bias potentially guiding misleading results.11 59 We aim to publish the report of this project including the final PRIUR-CCC guidelines. The main limitation of this study will be inherent in the Delphi study design, where only completers of all rounds fully contribute to the project.
Ethics statements
Patient consent for publication
References
Footnotes
X @dmoher, @CorteseSamuele, @eggersnsf
MS and KC contributed equally.
Contributors MS, KC, LH and AFC designed the protocol, drafted the first version of the protocol and finalised the final version. DM, SE, ED, JIIS, JR, SC, BS, NV, MP, ME, SP and JPI approved the initial study design, and revised multiple times the protocol and approved the final version.
Funding LH is supported by a Canada Research Chair in Knowledge Synthesis and Translation.LH is supported by a Canada Research Chair in Knowledge Synthesis and Translation.LH is supported by a Canada Research Chair in Knowledge Synthesis and Translation.
Competing interests MS received honoraria/has been a consultant for Angelini, Lundbeck, Otsuka.
Patient and public involvement Patients and/or the public were involved in the design, or conduct, or reporting, or dissemination plans of this research. Refer to the 'Methods' section for further details.
Provenance and peer review Not commissioned; externally peer reviewed.