Article Text
PDF
Abstract
The COVID-19 pandemic has seen an increase in rapidly disseminated scientific evidence and highlighted that traditional evidence synthesis methods, such as time and resource intensive systematic reviews, may not be successful in responding to rapidly evolving policy and practice needs. In New South Wales (NSW) Australia, the Critical Intelligence Unit (CIU) was established early in the pandemic and acted as an intermediary organisation. It brought together clinical, analytical, research, organisational and policy experts to provide timely and considered advice to decision-makers. This paper provides an overview of the functions, challenges and future implications of the CIU, particularly the Evidence Integration Team. Outputs from the Evidence Integration Team included a daily evidence digest, rapid evidence checks and living evidence tables. These products have been widely disseminated and used to inform policy decisions in NSW, making valuable impacts. Changes and innovations to evidence generation, synthesis and dissemination in response to the COVID-19 pandemic provide an opportunity to shift the way evidence is used in future. The experience and methods of the CIU have potential to be adapted and applied to the broader health system nationally and internationally.
- COVID-19
- Health policy
- PUBLIC HEALTH
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Introduction
The COVID-19 pandemic has seen an increase in rapidly disseminated scientific evidence. This evidence varies in quality and is often pre-peer review. The pandemic has been described as ‘break(ing) the evidence pipeline’ and traditional evidence synthesis methods, such as systematic reviews, may not be successful; many reviews are out of date before publication.1 As a result, there has been an increased need for timely and responsive synthesised evidence to inform decision-making. Internationally, different jurisdictions have adopted new approaches to rapid synthesis of evidence to meet this demand and collaborate to reduce duplication.2 There have been calls to retain and strengthen what has worked during the pandemic and develop both the international and national evidence ecosystem.3
The publicly funded New South Wales (NSW) Health comprises the NSW Ministry of Health, specialist health organisations and pillar organisations (including the Agency for Clinical Innovation (ACI)), local health districts and specialty networks.4 The Critical Intelligence Unit (CIU) was established by NSW Health early in the pandemic as an intermediary organisation to support knowledge translation. Intermediary organisations have an important role to play in evidence ecosystems, ensuring evidence-based policy and practice are promoted and supported, and diverse stakeholder perspectives and viewpoints are taken into account.3 5 6 The CIU brings together clinical, analytical, research, organisational and policy experts to provide timely and considered advice to decision-makers.7 The unit reports directly to the health secretary/incident controller and prepares ‘tailored knowledge products’5 by summarising evidence and applying it to the NSW context. The CIU structure has been described elsewhere7 and includes a steering committee of senior executives from NSW Health Organisations; a Clinical Intelligence Group (CIG); Evidence Integration, Research Intelligence and Data Intelligence teams and a stakeholder advisory group. Experts in the CIG were selected for their expertise and experience in using data and evidence in a clinical and policy context.
This paper focuses on one function of the CIU—the Evidence Integration Team— and its rapid reviews and evidence synthesis. The Evidence Integration Team provides rapid, ‘good enough’ evidence and is transparent about the limitations of current and rapidly changing information.7 A key feature of this work is balancing responsiveness with maintaining quality.
Products and methods
The rapidly evolving landscape of COVID-19 required new approaches to evidence synthesis. To inform policy, reliable, agile and timely sources of evidence-based information were needed. The Evidence Integration Team produces three key COVID-19 products: daily evidence digest, rapid evidence checks and living evidence tables. All products are available on the CIU website.
The work of the Evidence Integration Team was completed by the ACI Evidence directorate, by suitably qualified colleagues from across the ACI8 and staff seconded from other NSW Health organisations.
Evidence digest
The evidence digest is a two-page product published daily (weekdays only) from 26 March 2020 to 27 May 2022 and weekly thereafter. It compiles new evidence and reports on COVID-19 from key international journals and organisations. The digest is a horizon-scanning service that communicates information from a variety of sources.9 While the concept of horizon scanning is not novel, its application in NSW in the context of COVID-19 was identified as a gap early in the pandemic.
A small team (a minimum of five staff on average which expanded with peaks in the pandemic) scans key scientific journals, organisation websites and databases daily and enters results into a spreadsheet. Journals and organisations were identified via the NSW Health Clinical Information Access Portal (CIAP) COVID-19 resources page and college organisation webpages through a rapid evidence review. Automated daily searches on PubMed were monitored to capture additional journal articles. Inclusion criteria for the digest were defined at the onset. Due to the rapidly changing nature of the pandemic, these inclusion criteria were assessed and updated as required.
The digest results spreadsheet is a key resource for CIU products, collating publications from journals from the beginning of the pandemic to the present. New spreadsheet entries for the day are reviewed by the Synthesis manager or Evidence director and included or excluded for the next digest based on relevant criteria. The digest is produced using a macro to autopopulate a Word document sorted by publication type. The measures to ensure responsiveness and quality are outlined in table 1.
Evidence digest—measures for responsiveness and quality
Over 500 evidence digests have been published since March 2020. The evidence digests are distributed through the CIU website and emailed to over 2000 subscribers.
Rapid evidence checks and in briefs
Evidence checks and in briefs follow a simplified rapid review methodology to synthesise evidence on COVID-19-related topics or questions. Evidence includes peer-reviewed and grey literature (eg, government reports, policy statements, professional organisation guidelines and preprint articles). The product is generally a one-to-two-page ‘in brief’ narrative synthesis. It is not intended to be an exhaustive systematic review (ie, with a comprehensive list of databases searched and a formal quality appraisal of included studies), but a ‘good enough’ summary of current evidence (see table 2).
Evidence checks and briefs—measures for responsiveness and quality
Topics and questions are requested by the CIU, NSW Health decision-makers, clinicians and COVID-19 communities of practice. Requests were emailed to the CIU or came directly from decision-makers to the CIU Executive. Topics were prioritised based on whom the request was from and the purpose of the request. For example, requests from the NSW Health Secretary to inform policy decisions were given the highest priority. The Evidence Integration Team met weekly to discuss, prioritise and allocate requests for completion. Evidence checks and in briefs take as little as 8 hours and up to 2 weeks to complete. The degree of urgency affects the number of team members allocated to a review or the time taken to produce an evidence check or in brief. The simplified methodology for review includes defining the question (using a scoping document if needed), literature search, results screening against inclusion criteria, simplified data extraction tables or evidence overview, narrative synthesis, expert review and copy editing (see figure 1). The measures to ensure responsiveness and quality are outlined in table 2.
Evidence checks and in briefs follow standard rapid review methods.
The CIU evidence checks cover a range of topics including epidemiology and transmission; symptoms, diagnosis and treatment; clinical models of care; and system capacity and evaluation. Since March 2020, over 200 rapid evidence checks and briefs have been published on the CIU website. Analytics are tracked using Google Analytics. The most viewed topics include face masks, modes of transmission, ivermectin, healthcare worker infections, workforce reconfigurations and extended or reuse of personal protective equipment. Where evidence was rapidly evolving, evidence checks were revisited and updated as required. Often, these evidence checks formed the evidence basis for clinical guidance or policy documents within NSW Health. For example, the evidence checks and in briefs on monoclonal antibodies and antiviral medications informed the guidance for the use of these anti-SARS-CoV-2 agents as prophylaxis or to prevent severe infection from COVID-19 in NSW. Evidence checks that are out of date but on topics identified as not requiring update are marked as archived.
Living evidence tables
Living evidence tables are web-based resources that summarise the latest studies and emerging evidence on key topics. Tables are reviewed regularly and updated as new evidence is published. The CIU established methods for living evidence in the context of COVID-19 including:
Defining timing of updates.
Defining a process for updates, for example, sources to search.
Defining criteria for when an update is required, for example, a systematic review or study higher in the evidence hierarchy is published on a topic; a study goes from pre-peer review to peer reviewed; or a new study has a larger sample size.
Two levels of review before updates go live.
Highlighting new evidence.
Periodic review and revision.
Living evidence tables are updated through screening results of automated PubMed searches and the daily evidence digest spreadsheet. Changes to the living tables are highlighted each week on the website so users can easily identify new evidence. The measures to ensure responsiveness and quality are outlined in table 3.
Living evidence tables—measures for responsiveness and quality
The CIU has developed a total of seven living evidence tables on topics including: COVID-19 vaccines, SARS-CoV-2 variants, transmission, postacute sequelae (long COVID), surgery, rapid testing and risk mitigation. The topics covered in the living tables are responsive to the stages of the pandemic and as topics become less dynamic, the corresponding living tables are retired. As a result, four living tables (risk mitigation, surgery, transmission and rapid testing) were retired in 2022 and two living tables (vaccines and variants) were retired in 2023. The vaccines, variants and long COVID living evidence tables are the most visited with over 70 000, 40 000 and 12 500 page views, respectively, in an 8-month time period.
Methods
The Evidence Integration Team responded to requests from the CIU using a flexible team approach. Capacity building on the job was critical as there was no time for extensive training or shadowing.10 Some briefs were completed by a single person while others had input from two or more people if the time frame was tight and/or if the question was larger. ‘Parallelisation of tasks’2 was used to streamline processes, for example, one team member screened peer reviewed and grey literature while another completed data extraction. The team used collaborative technology, including Microsoft Teams and SharePoint, to facilitate responses to requests.
The Evidence Integration Team also had the benefit of working with the data intelligence function within the CIU. This collaboration allowed the development of new products, such as the COVID-19 monitor, which pulled together state and international level data indicators, complemented with the latest evidence. The team was also able to integrate learnings from experiential evidence, often in the form of international surveys or meetings, bringing together like countries to discuss approaches that worked for different situations throughout the pandemic.
Challenges unique to COVID-19
The rapid publication of articles during the COVID-19 pandemic means database searching alone cannot be relied on, as there is a lag between studies appearing on databases and indexing terms. The pandemic has also seen a surge in pre-peer-review articles. This led to some concerns around the quality and trustworthiness of articles; however, their value in highlighting and disseminating preliminary findings on evolving topics has been useful in contributing to rapid evidence. The inclusion of pre-peer-review articles in CIU products, especially the living tables, requires periodic and thorough checking, and updating links when they are published in a peer-reviewed journal.
The rapidly changing nature of the pandemic and the time pressure of evidence requests at times made it difficult to clearly articulate and finalise research questions.2 As decision-makers and the CIU were so busy, the Evidence Integration Team often needed to interpret and prioritise requests with limited input or guidance. Occasionally what was produced was no longer required or did not meet the needs of the decision-makers.
The inclusion of low-level evidence, such as letters and commentaries, is another change in usual review practices. This was necessary given the scarce empirical evidence on emerging topics during the pandemic. Additionally, the huge volume of published literature means reviews can become outdated soon after publication and some evidence products require frequent updates.11
Searching and monitoring grey literature has its own unique challenges. Searches often targeted English-speaking countries only or English-language guidance and reports. International policy evolved rapidly and required constant monitoring as information from these sources might become outdated within days.
Other challenges included unpredictability of demand and consequent workforce planning. Frequent workforce mobilisation was required within the organisation as well as temporary new recruitment. Weekend and holiday workforce planning was required to deliver urgent requests from decision-makers.
Impact
Rapid evidence informs different decision-making processes during a pandemic, including policy recommendations.10 Living evidence, including living systematic reviews, has been identified as suitable for a pandemic context.12 Much of the success of the CIU can be attributed to new ways of working with evidence and information, including living evidence and rapid approaches to evidence synthesis.13
While the CIU evidence synthesis products summarise existing evidence and do not provide recommendations, they are used to support decision-makers and inform policy. The CIU also provides evidence summaries and advice to COVID-19 Communities of Practice,14 clinician groups that publish guidelines and advice to the healthcare system. Examples of evidence in this context is a suite of evidence checks produced by the CIU on resuming elective surgery; and evidence briefs on monoclonal antibodies and rapid access clinic. These informed rapidly developed and up-to-date guidance for NSW clinicians on timing of surgery after post-COVID-19 infection and management of stable acute respiratory infections during winter surges.15 16
Google Analytics has been used throughout the pandemic to monitor access to the evidence. Most visitors to the CIU webpages are from NSW, however, all states have accessed content, with Victoria being the next most active. Internationally, people from the USA, Canada, New Zealand and others have visited the webpages. There are high visitor spikes at certain times, generally aligning with COVID-19 outbreaks in NSW. For example, the biggest spike in visitors was during the Omicron outbreak in NSW in early 2022 (increasing from under 5000 weekly page views to over 25 000 weekly page views), with an earlier spike recorded at the beginning of December 2020 during the Sydney Northern Beaches Christmas outbreak. While the homepage is the most viewed page, the living evidence tables, particularly those on vaccines and variants, are the most visited individual pages (online supplemental appendix 1). The CIU’s outputs, mainly evidence checks and living evidence tables, were also widely cited in both the peer-reviewed and grey literature. As of April 2023, the CIU was cited nearly 70 times based on Google Scholar searches. The daily evidence digest was also cited as a key source for identifying the latest evidence on COVID-19 in peer-reviewed literature.17 18
Supplemental material
The CIU was a finalist in the 2021 NSW Premier’s Awards for developing innovative approaches to evidence and data collation and establishing a flexible, agile and low-cost operating model able to respond to changes in circumstances and urgency.
Interpretation and discussion
Using a rapid approach to evidence synthesis is not novel. A study comparing the findings of rapid reviews versus systematic reviews on the same topic found the conclusions may not be fundamentally different.19 In addition, knowledge translation platforms supporting evidence-informed policy-making have been demonstrated in the context of COVID-19 by using evidence synthesis, engaging with key stakeholders, disseminating evidence, promoting trust and countering misinformation, monitoring and evaluating policy response and using technology to enable living evidence.6 20
What is novel in the CIU approach is the emphasis on peer review and transparency to ensure quality products. All CIU work is reviewed and produced with involvement from subject matter experts, clinicians, policy-makers and other stakeholders, including a dedicated group of experts established for this purpose.
Limitations
Limitations to the approach include capacity issues in the team requiring workforce mobilisation, particularly during disease outbreaks. Some products, such as the living evidence tables, are resource intensive to update and constant review is required. Understanding which topics should be living is challenging and as tables expand, they can become unmanageable or unclear. To produce rapid results, often only PubMed, Google and Google Scholar were searched. PubMed has been recommended as the preferred database if only one database is used, as rapid reviews not using PubMed are more likely to differ from systematic reviews.21 Studies were screened by one person only; therefore, despite clearly defined inclusion and exclusion criteria, falsely excluding eligible studies due to individual error or selection or interpretation bias was possible. Not completing quality appraisal on included studies meant studies of varying quality were included and synthesised together. The impact and performance of the CIU has not been formally evaluated using quantitative or qualitative measures, which may warrant future research.
The methods used for CIU products had to be adjusted as priorities changed throughout the pandemic, such as inclusion/exclusion criteria for the Evidence digest. For example, one update to the criteria excluded studies in areas where the science was settled, such as hydroxychloroquine. Setting criteria was challenging; blanket exclusions such as excluding case reports risked missing the first reports of an event. As such, the team exercised judgement alongside the inclusion criteria to ensure key papers and developments were captured. Efforts were made to address the risk of bias through consultation with the CIG. The CIU is transparent in their methods for all products, noting that the Evidence digest is a compilation of sources, with no quality assessment or endorsement of any articles. While Google Analytics was used to measure the impact of these resources, there are limitations in interpreting this data and the impact these resources had on public opinion is not clear. While not undertaking quality appraisal allowed the CIU to respond rapidly, it limits the robustness of the results. Other organisations such as the Australian National Clinical Evidence Taskforce have published methods on rapidly assessing the quality of evidence as one possible approach to increasing this robustness.22
The future of rapid, ‘good enough’ evidence
One recommendation from the Global Commission on Evidence to Address Societal Challenges is commitment and support for the use of research evidence in making decisions including through a central agency.23 The CIU worked as a central unit to coordinate evidence, including synthesis, data and experiential evidence to support the COVID-19 response in NSW. In a bid to reduce duplication, the ACI was part of an international collaboration, COVID-End.
The success of an intermediary organisation in promoting knowledge translation ‘rests on the credibility and legitimacy of the intermediary’.24 Given the political nature of decision-making and the highly charged environment of COVID-19, the CIU had to develop trust and strong communication pathways for the ‘good enough’ evidence to be requested and used on an ongoing basis. Establishing a strong governance process to underpin the responsiveness and accountability of the unit was key. As described in other jurisdictions, creating strategic roles within government to embed research within and across teams builds capacity for the use of evidence to better inform actions.25 Embedding the CIU within NSW Health structures (with a direct reporting line to the health secretary/incident controller), rather than outsourcing to an academic unit, may have contributed to its success.
The approach to rapid and responsive evidence has shown its value and the methods could be applied beyond COVID-19. For some areas of evidence synthesis, such as living evidence, there are learnings published in the literature. Four considerations identified as key for living evidence include: reducing unit costs, such as by using technology to aid in the systematic review process; preventing research waste; optimising publishing; and facilitating implementation.1 While living evidence is an innovative and successful method of evidence synthesis for some topics, it is not feasible or necessary for all clinical areas. The areas that would benefit most from living evidence are those where research is emerging rapidly, current evidence is uncertain, and new research might change policy or practice.1
Other areas of evidence synthesis outlined in this paper have already been applied outside COVID-19. The ACI includes over 40 clinical networks, and rapid evidence checks have been undertaken on a variety of health conditions to inform clinical practice guides and models of care. In March 2023, the last edition of the COVID-19 evidence digest was sent and a new digest was released in May 2023. This digest moves beyond COVID-19 and focuses on advances in healthcare. Adaptations to the process outlined for COVID-19 include using experts in the clinical networks, as well as interstate and international colleagues for peer review, rather than having a centralised group of experts. By maintaining the principles of balancing rapidity and responsiveness with ensuring quality and transparency, this approach could be applied to informing clinical guides and models of care ongoing for NSW and other jurisdictions.
Acknowledgments
We thank both the current and past members of the Critical Intelligence Unit and Evidence Integration Team for their support and contribution.
Footnotes
Twitter @jfredlevesque, @kimlsutherland3
Contributors All authors contributed to the key functions of the Critical Intelligence Unit and have been involved in development and production of key products. LW led the writing of the first draft. EM, HD, J-FL and KS contributed to further writing and critical review of the first draft. All authors contributed to the interpretation of the analysis, critically revised and approved the final draft. LW is the guarantor and attests that all authors meet authorship criteria.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.