You are here

Article Text

Article menu
Evidence based practice
Protocol
Economic evaluations of scaling up strategies of evidence-based health interventions: a systematic review protocol
  1. Francesca Brundisini1,2,
  2. Hervé Tchala Vignon Zomahoun1,3,
  3. France Légaré3,4,
  4. Nathalie Rhéault1,4,
  5. Claude Bernard-Uwizeye1,4,
  6. José Massougbodji1,3,
  7. Amédé Gogovor1,3,
  8. Sébastien Tchoubi1,3,
  9. Odilon Assan1,5,
  10. Maude Laberge2,4
  1. 1Knowledge Translation and Implementation component of the Quebec SPOR-SUPPORT Unit, CIUSSS de la Capitale-Nationale, Quebec, Quebec, Canada
  2. 2Operations and Decision Systems, Université Laval, Quebec, Quebec, Canada
  3. 3Department of Family Medicine and Emergency Medicine, Université Laval, Quebec, Quebec, Canada
  4. 4VITAM, Centre de recherche en santé durable -Université Laval, CIUSSS de la Capitale-Nationale, Quebec, Quebec, Canada
  5. 5Pharmacy, Université Laval, Quebec, Quebec, Canada
  1. Correspondence to Maude Laberge; maude.laberge{at}fsa.ulaval.ca

Abstract

Introduction Scaling science aims to help roll out evidence-based research results on a wide scale to benefit more individuals. Yet, little is known on how to evaluate economic aspects of scaling up strategies of evidence-based health interventions.

Methods and analysis Using the Joanna Briggs Institute guidance on systematic reviews, we will conduct a systematic review of characteristics and methods applied in economic evaluations in scaling up strategies. To be eligible for inclusion, studies must include a scaling up strategy of an evidence-based health intervention delivered and received by any individual or organisation in any country and setting. They must report costs and cost-effectiveness outcomes. We will consider full or partial economic evaluations, modelling and methodological studies. We searched peer-reviewed publications in Medline, Web of Science, Embase, Cochrane Library Database, PEDE, EconLIT, INHATA from their inception onwards. We will search grey literature from international organisations, bilateral agencies, non-governmental organisations, consultancy firms websites and region-specific databases. Two independent reviewers will screen the records against the eligibility criteria and extract data using a pretested extraction form. We will extract data on study characteristics, scaling up strategies, economic evaluation methods and their components. We will appraise the methodological quality of included studies using the BMJ Checklist. We will narratively summarise the studies’ descriptive characteristics, methodological strengths/weaknesses and the main drivers of cost-effectiveness outcomes. This study will help identify what are the trade-offs of scaling up evidence-based interventions to allocate resources efficiently.

Ethics and dissemination No ethics approval is required as no primary data will be collected. The results will be published in a peer-reviewed, international journal and presented at national and international conferences.

  • health economics
  • health services administration & management
  • protocols & guidelines
http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

https://doi.org/10.1136/bmjopen-2021-050838

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Strengths and limitations of this study

    • This is the first systematic review to provide evidence on economic evaluation approaches for the scaling up strategies of evidence-based interventions.

    • We plan a strong, rigorous and reproducible methodology for conducting our systematic reviews of economic evaluations.

    • We follow the Joanna Briggs Institute guidance for conducting systematic reviews of economic evaluations.

    • A comprehensive search strategy will be employed to retrieve both peer-reviewed and grey publications.

    • The review may face some limitations to generalisability due to the highly context-specific nature of economic evaluations.

    Introduction

    Researchers, healthcare professionals and decision-makers are increasingly focusing on filling the gap between knowledge and practice. In recent years, growing efforts to bridge this gap have produced a vast body of knowledge on the efficacy and effectiveness of health interventions and their implementation in practice.1–3 Most of this evidence derives from experimental studies in which interventions are delivered under optimal, or at least ‘best practice’ conditions, generally conducted on relatively small populations and from projects done in given settings. To date, these efforts have produced a wide set of well-documented effective health interventions.1 2 4 5 However, health decision-makers are still not systematically implementing such evidence to benefit more people on a wider scale.1 2 4–8 One way to fill this gap is to develop and implement strategies to scale up effective evidence-based interventions (EBIs) in health.7 9

    While both efficacy and effectiveness are key to the roll out of EBIs on a large scale, other factors—such as costs and cost-effectiveness—are central to the successful scale up of EBIs.8 10–14 As health systems face continuous strains and limited resource availability, economic evaluations can play an important role in informing health decision-makers on the trade-offs in costs health benefits of choosing and defining a scaling up strategy.10 12 14–21 Economic evaluations are a means to both assess the value for money and inform resource allocation decision-making.22 To do so, economic evaluations compare alternative choices in terms of both costs and consequences.22 Alternative choices refer to the different ways in which healthcare resources can be used to improve health. The type of economic evaluations are generally defined by the number of alternatives compared, whether both costs and consequences are examined, and how the consequences are expressed.22

    Little is known on what these evaluations should include to analyse the cost-effectiveness of scaling up strategies, as the cost-effectiveness of EBIs does not necessarily reflect the cost-effectiveness of the scaling up effort.8 13 15–19 21 23 While not many, a small number of studies synthesised the costs and cost-effectiveness of scaling up strategies of EBIs in health. Mostly conducted in low and middle income countries, these reviews show that included studies generally focus, among other interventions, on national immunisation programmes,21 24 25 maternal, infant and children health programmes20 and HIV/AIDS prevention and care interventions.16 26 Despite being conducted in specific geographical areas and having a narrow focus on scaling up strategies of certain health interventions, these reviews provide insights into the economic evaluation research production of scaling up strategies. These reviews reveal a great variability among the included economic evaluation studies. When included, these studies vary in perspectives, scope, approaches, assumptions, cost categories and are often not presented in a way that can be easily comparable and generalised across settings and countries.19–21 26–28

    Oftentimes, the lack of complete availability of scaling up cost data or the use of models leads economic analysts to rely on assumptions that may not reflect the complexity of implementing scaling up strategies.8 16–19 21 26 29 30 For example, economic evaluations of scaling up strategies may posit that scaling up implementation costs are a fixed part of the intervention costs.19 30 31 In reality, scaling up strategies may present additional costs to that of the intervention that can greatly vary across interventions and settings, potentially leading to both economies and/or diseconomies of scale.29 Costs and cost-effectiveness estimates may change according to the type of intervention being expanded, the size of the targeted population, the prevalence/incidence of the disease, the relevant efficacy level of the intervention, the geography and the financial resources available and needed.8 13 15–17 19 29 32 Specific to scaling up strategies, costs and estimates related to infrastructure and available human resources can vary based on the different scaling up strategy operationalisation and management, the cost impacts of change, including the excess cost of service delivery as uptake changes and the opportunity costs to providers and patients participating in the activities.8 13 15–17 19 29 32 Finally, implementation and scale-up theoretical frameworks—that support thinking and interpretation of ‘real world’ complex data—consider economic constructs in scaling up strategies in different ways. For example, some frameworks consider cost (and resource) mobilisation as a key objective,33 34 yet implementation frameworks consider costs as an implementation outcome.35 Frameworks vary also in the ways they consider potential benefit or effectiveness (‘cost–benefit’).36 This variability then results in a wide heterogeneity of studies and approaches when it comes to economically evaluating scaling up strategies. Costs and cost-effectiveness estimates may also vary according to different modelling approaches. For example, ex-ante economic evaluations are often used for informing preimplementation decision-making using available evidence and modelling to simulate the costs and consequences of alternatives.15

    We argue then that little is known on how to evaluate the economic aspects of these strategies to understand what constitutes the trade-offs of scaling up EBIs to allocate resources efficiently. Thus, we seek to identify and describe the methods and issues related to economic evaluations aimed at assessing scaling up strategies of EBIs in health.

    Objectives

    Our goals are to:

    • Identify and describe which economic evaluations methods are used to assess scaling up strategies of EBIs in health.

    • Identify and describe the costs and cost elements adopted in such economic evaluations.

    • Identify and describe environmental factors accounted for in such economic evaluations.

    • Discuss the strengths and limitations of each approach and explain reasons for variation in the reporting of economic evaluations of scaling up strategies of EBIs in health.

    Methods

    Study design

    We are conducting a systematic review following Joanna Briggs Institute (JBI) guidance for conducting systematic review of evidence from all (ie, partial and full) economic evaluations addressing a question(s) about scaling up health intervention strategies’ cost-effectiveness.37 38 We adopted Preferred Reporting Items for Systematic Reviews and Meta-Analyses protocols (PRISMA-P) guidelines for reporting of systematic reviews protocols39 (online supplemental additional file 1). We registered the protocol on Open Science Framework database (registration number osf.io/fsq84).

    Eligibility criteria

    Studies included in the review must adhere to the eligibility criteria described below following the Population, Intervention, Comparison, Outcomes (PICOS) as outlined in the PRISMA-P guidelines.39

    Population: We will include studies in which the population of interest is any individual, organisation, or system—directly or indirectly—involved in the delivery or receipt of any health services that was the target of the scale-up.

    Intervention: We will include research studies that investigate strategies for scaling up. Included studies must evaluate a scaling up strategy of an EBI (and not the evidence-based health intervention itself). For the purposes of this systematic review, we consider:

    • A health intervention to be a health service or a package of health services aimed at improving, maintaining, promoting or restoring health.40 41

    • EBIs in health as health interventions that are effective, efficacious and ready for dissemination.42

    • A strategy as one or more initiatives, approaches or activities that directly aim to change the supply or demand of EBIs in health to improve reach, adoption and sustainability of an EBI.

    • Scaling up in healthcare as the ‘deliberate efforts to increase the impact of successfully tested health interventions so as to benefit more people and to foster policy and program development on a lasting basis’.12 34 43 In other words, scaling up strategies are systematic courses of action that aim to roll out successful local health interventions to regional, national or international levels to reach broader populations and settings over time.34 43

    No restrictions will be made on the type of EBI or impact (effectiveness) metric chosen. The scaling up of an EBI can be implemented as a standalone intervention, or as an addition in combination with other interventions.

    Comparator: There are no restrictions on the type of comparator. Included studies may report economic evaluations that compare the studied scaling up strategy to current practice (ie, no scaling up), or to alternative scaling up strategies.

    Outcomes: All reported partial or full economic evaluation outcomes are of interest. Outcomes will include measures related to costs and cost-effectiveness. Partial evaluations focused only on costs will include cost outcomes reported as monetary amounts. Full economic evaluations cost-effectiveness outcomes will include incremental cost-effectiveness ratio, incremental cost-utility ratio, net benefit, cost–benefit ratio. The metric chosen to report the health gain (effectiveness) used in the economic evaluations will not be an inclusion criterion. It can include (but not restricted to) for instance cost/illness averted, cost/quality-adjusted life year gained, or cost/disability-adjusted life year averted. All viewpoints/analytic perspectives will be considered with no restrictions. We expect that a variety of outcomes are used in studies to report on the cost-effectiveness of scaling up EBIs. Studies in which only scaling up strategy’s effectiveness, adoption or health gain was reported will not be included.

    Study design: Any study design that includes any type of empirical economic evaluation, as well as any modelling and methodological considerations will be included. We will include both full economic evaluation designs, such as cost-effectiveness analysis, cost-utility analysis and cost–benefit analysis and partial economic evaluation designs, such as cost minimisation analysis, cost comparison/cost analysis, cost outcome descriptions, cost descriptions and budget impact analysis. Additionally, included modelling studies can be based on a meta-analysis of data from randomised trials or using secondary data from literature and those based on observational studies or analysis of large administrative databases. Both published and unpublished grey literature will be included. We will exclude the following studies: reviews, systematic reviews, qualitative studies, clinical effectiveness studies, critical reviews, editorials, commentaries, abstracts, protocols and academic theses.

    Settings: We will review studies independently of the settings, thus, including any healthcare setting (ie, public health, primary care clinic, hospital, etc.) in both rural and urban areas. We will not restrict the inclusion criteria based on geography. Economic evaluations undertaken within any country context will be included.

    Information sources

    The information sources include a search of the following electronic bibliographic databases from their inception onwards: Medline, Web of Science, Embase, Cochrane Library Database, PEDE, EconLIT and INHATA. Additionally, since economic evaluation studies are often conducted for the government or by government agencies, we will systematically perform an internet search as this has been shown to regularly capture eligible studies not identified by other databases.25 We will perform an extensive search strategy using free text, with no restrictions on date and year of publication. A search of web pages of international organisations, bilateral agencies, non-governmental organisations and consultancy firms involved in the delivery, funding or evaluation of scaling up EBIs. Reports found to have a matching publication in the published literature will be excluded. We will search the following Internet search databases and data sources: Google, Google Scholar, INESSS (Institut national d'excellence en santé et en services sociaux), OpenGrey, Grey Literature Report, GreyNet, Canadian Evaluation Society, EuroScan, databases included in the ‘Grey Matters—A Practical Deep web Search Tool for Evidence based Medicine’ (CADTH) Checklist and region-specific databases (African Index Medicus, Eastern Mediterranean Literature (WHO), Index Medicus for South-East Asia Region, LILACS for Latin America). We will then conduct a webpage search of following organisations/agency/governmental websites: UNICEF, WHO, GAVI Alliance, Programme for Appropriate Technology in Health, Johns Hopkins School of Public Health, World Bank, Global Affairs Canada, UK Department for International Development and US Agency for International Development.

    Search strategy

    Our information specialist (NR) developed a Medline strategy with input from the project team. An iterative process of revision was conducted by the members of the research team. Comments will be integrated for a final version of the search strategy. This final version was approved by the team members. Once validated, the information specialist (NR) translated this search strategy for each electronic database mentioned above. The present protocol only includes the search strategy conducted in Medline on 14 October 2020 (see online supplemental file 2). A hand search will also be performed, and citations and bibliographies of included primary studies and relevant literature reviews will be reviewed for additional relevant articles.

    The search will include a combination of the following two concepts: (1) scaling and (2) Economic Evaluation basic terms. No language restrictions will be applied. The search strategy in Ovid Medline is in the online supplemental materials.

    The following sources were used to find the search terms: (1) previous reviews who used the concept of scaling up7 20 and the concept of economic evaluation;20 21 44 (2) the knowledge of the experts of our multidisciplinary team in scaling up and (3) the thesaurus of the consulted bibliographic databases. All words and expressions found were tested and evaluated by the information specialist before to be integrated or rejected in the search strategy. The search strategy was commented via an iterative process by the other members of the team for the production of the final version.

    The concept Scaling was created to retrieve all the potential expressions for designed the idea of the spreading of an innovation. It is designed to retrieve very used expression like ‘scaling up’, ‘scale up’, ‘spread of technologies’, but also many variations like ‘widespread adoption of the technology’ or ‘rolling out the model of care’. The concept of Economic Evaluation integrated all synonyms like ‘cost evaluation’, ‘economic analysis’ and ‘net benefit’.

    Study records

    Data management

    In this ongoing study, we exported all citations identified from the electronic databases into Endnote X9 (citation manager software). We used EndNote X9 to remove duplicates in addition to manual checking to identify unique citations for the study selection process. Unique records were then exported into Covidence (internet-based screening and data extraction tool).

    Selection process

    All stages of the selection process will be performed independently by two reviewers. One reviewer (FB) developed and tested (after team validation) together with the second reviewer a pilot screening form against the eligibility criteria on a 7.5% random sample of the retrieved citations (title and abstracts) to validate the process of inclusion of articles in the review (see the data extraction codebook form template in the online supplemental file 3). This piloting stage ensured reviewers shared a common understanding of the eligibility criteria. At the title and abstract stage, the reviewers will independently screen the titles and abstracts with regard to the inclusion/exclusion criteria using Covidence. Studies not fulfilling the eligibility criteria will be excluded, and the full texts of the remaining studies will be retrieved for further assessment. Articles with abstracts that do not appear to meet the criteria for exclusion or are ambiguous, or that have a missing abstract, will be retained and reviewed in full. The full text of retained studies will be independently assessed for exclusion against inclusion/exclusion criteria by both reviewers. To resolve eligibility questions, we will contact the authors of the included studies to seek additional information. Discrepancies between reviewers will be solved through discussion, and—if needed—a consultation with a third reviewer. Any reasons for exclusion will be recorded in Covidence at the full-text stage. The results of the identification, screening and inclusion process will be displayed using the PRISMA flowchart.39

    Data collection process

    A standardised data extraction template form will be piloted in duplicate by the reviewers. The extraction form will be informed by the study objectives, eligibility criteria and the JBI-ACTUARI (Joanna Briggs Institute-Analysis of Cost, Technology and Utilisation Assessment and Review Instrument) tool.37 This template form will allow to extract from each study information on the key characteristics, the results for the outcomes of interest and the author conclusions.38 The form will be tested on a 10% random sample of the included studies for data collection. This pilot test will help to identify extraction items that are missing from the template, or likely to be confusing or unnecessary. Authors’ consensus will be required before the form can be modified if deemed appropriate. The investigators will use the finalised revised and agreed on version of the data extraction form to extract data independently.

    Data items

    The data extracted will cover: first descriptive data about (i) the study general characteristics (eg, title, short name, corresponding author name, funding source and conflict of interest), study type (published or grey literature), study population/participants, type of scaled up intervention and authors’ description of intervention (including whether it was a standalone intervention or a combination of interventions), type of scaling up strategy (including scaling up level of implementation) and authors’ description of strategy, its comparator(s) and outcomes; (ii) study methods including evaluation design type, analytic viewpoint(s), prices and currency used for costing, time period of analysis; sensitivity testing; source of effectiveness data, measures of resource use, cost and health effect/clinical and cost effectiveness and (iii) study context (geographical, healthcare and broader service delivery setting); second reported results for the resource use and/or cost and/or cost effectiveness measures; third, when possible author conclusions about factors that promote and limit the cost-effectiveness of scaling up EBIs strategies.

    Quality appraisal

    There is still no consensus among health economic experts on which guidelines to follow when conducting systematic reviews of economic evaluations.45–48 We will be using Drummond and Jefferson checklist, also known as the British Medical Journal (BMJ) checklist, as it was designed for full economic evaluations but also applicable to partial economic evaluations, report and commentaries on economic evaluations, thus aligned to our broad inclusion criteria.49 The BMJ tool is a yes/no, 35 items checklist organised in three sections: study design, data collection and analysis and interpretation of results.49 If items are not applicable to a specific study, a ‘not appropriate’ response can be stated. Critical appraisal will be undertaken independently by two individuals. If any disagreements arise, they will be discussed between the two reviewers and if need be resolved by team consensus or by a third reviewer.

    Data synthesis

    We will use descriptive structured narratives, statistics and tables to identify and summarise the key features of the included economic evaluations of scaling-up strategies and the elements considered in such evaluations. Narrative synthesis will be used to summarise the methods, highlighting important characteristics of the studies when relevant, focusing on differences/similarities and methodological weaknesses and where possible identifying the main drivers of cost-effectiveness outcomes. In particular, the synthesis will focus on:

    • The assumed key theoretical trade-offs (between levels and types of resources, and levels and types of outcome) of scaling up strategies used in the included economic evaluations.

    • The level and configuration of scaling up resources examined in the economic evaluations, how they are related to the levels and types of outcomes observed, and the contextual/environmental factors accounted for in these relationships.

    • The conclusions regarding the relationship between the cost-effectiveness of the scaling up strategy under examination and the economic evaluation approach.

    • Strengths and weaknesses of each approach for evaluating scaling up strategies of EBIs.

    We expect to include a plurality of economic evaluation studies assessing scaling up strategies of EBIs with diverse interventions, populations and settings, thus we anticipate that there will be heterogeneity making difficult to perform a meta-analysis with interpretable results. We will explore this heterogeneity by narratively synthesising the differences, and if possible, the similarities in settings, participants, intervention, comparison and outcomes characteristics across studies. For example, we will perform the data synthesis of economic evaluation methods according to the economic evaluation parameters reported.

    Patient and public involvement

    Patients and the public were not involved in the design of this study.

    Discussion

    The identification and description of the methods and issues related to the economic evaluations for the scaling up strategies of EBIs in health will help understand what constitutes the trade-offs of scaling up EBIs to allocate resources efficiently. It will contribute to both health economic evaluation research in scaling science and its implementation in policy and practice. Large-scale health intervention implementation warrants governmental investment, this will also require demonstrable benefits for the patients, providers and society at large. As our world is currently hitting rock bottom by an unseen pandemic—that is, COVID-19—healthcare systems are in more need than ever to understand how to best reduce waste50 and increase the roll out of what has more benefits than harms at the lowest cost. If deliberate efforts are not taken to efficiently allocate resources on a wide scale, healthcare systems will collapse.

    To the best of our knowledge, this will be the first review that will systematically outline and summarise different economic evaluation approaches used in scaling up strategies of EBIs in health. The science of scale is young and has been too often either completely undermined or clustered with that of sustainability.51 This study will offer a valuable picture of the advancements and gaps in the application of economic evaluation methods in the scaling science. Earlier reviews of economic evaluations considering scaling up strategies were narrower and focused only on scaling up strategies of specific health interventions. As such, we believe that the findings of this study will point to identify valid recommendations for action for future research and decision-makers. First, this study can help guide future research aimed at defining costing tools and models that can be easily used in scaling up frameworks and plans. It will contribute to define the nature and selection of costs that are integral to the successful roll out of EBIs on large scale, as well as the benefits and disadvantages of each economic methodological approaches aimed at evaluating strategies identified in the literature. Second, as scaling science is becoming an increasingly relevant area for research, policy and practice, clarifying how underlying methodological assumptions are based on evidence and on the multifactorial complexity of real-world scaling strategies will advance the quantity and quality of the information extractable from the evidence to inform both research and practice.8 We believe this review will then offer opportunities for improvement in the quality, production, reporting and application in practice of health economic evaluative methods to scaling up strategies.

    Second, we hope that this work will support the use of economic evaluations in policies that aim to successfully implement EBIs on a large scale. While health economic evaluations are a well-established component of health technology assessments, their use in implementation science, and in particular scaling science, remains limited.15 32 Yet, unless there are sufficient resources, not all possible scaling up strategies can be implemented. Health decision-makers need to have a clear, evidence-based understanding of the financial implications of scaling up EBIs to make an informed choice to use resources efficiently. Without systematically examining and reporting cost and cost-effectiveness evidence, the allocation of financial resources to scaling up strategies may be too high or too low. Economic evidence is then crucial for decision-makers to design scaling up strategies that are affordable and that represent an efficient use of current available resources.

    Ethics and dissemination

    Our research project is a systematic review based on existing primary studies and methodological papers and as such it will not be necessary to request ethics approval. Additionally, we follow the Canadian Institute for Health Research (CIHR) Ethics Guidance for Developing Partnerships with Patients and Researchers to guide the active dissemination of our findings.52 As per CIHR guidelines, no ethical approval is required when engaging patients and public for actively disseminating research findings.

    We plan to use passive and active dissemination strategies to disseminate our findings. First, we will publish this study’s protocol and later the results of this project in leading peer-reviewed journals in health implementation and services research. We will also share our findings at local, national and international conferences addressing audiences interested in implementation science, scaling science and health economics. Second, findings from this project will be relevant for health administrators, decision-makers, health professionals and patients. To reach these audiences, we will use our networks with health organisations and health research groups (such as the Quebec Strategy for Patient-Oriented Research (SPOR) Unit). We will tailor the dissemination message to fit each audience and select champions to disseminate our results. Finally, we will use different communication channels, such as newsletters, organisation websites and webinars, to reach all relevant audiences.

    Ethics statements

    Patient consent for publication

    References

    1. ↵
      1. Massoud MR DK,
      2. McCannon CJ
      . Options for Large-scale Spread of Simple, High impact Interventions. : USAID Health Care Improvement Proj. Bethesda, MD: University Research Co, 2010.
    2. ↵
      1. Eaton J,
      2. McCay L,
      3. Semrau M, et al
      . Scale up of services for mental health in low-income and middle-income countries. The Lancet 2011;378:1592–603.doi:10.1016/S0140-6736(11)60891-X
      OpenUrlCrossRef
    3. ↵
      1. Greenhalgh T,
      2. Howick J,
      3. Maskrey N, et al
      . Evidence based medicine: a movement in crisis? BMJ 2014;348:g3725. doi:10.1136/bmj.g3725
    4. ↵
      1. Damschroder LJ,
      2. Aron DC,
      3. Keith RE, et al
      . Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci 2009;4:50.doi:10.1186/1748-5908-4-50pmid:http://www.ncbi.nlm.nih.gov/pubmed/19664226
      OpenUrlCrossRefPubMed
    5. ↵
      1. Shaw J,
      2. Tepper J,
      3. Martin D
      . From pilot project to system solution: innovation, spread and scale for health system leaders. BMJ Leader 2018;2:87–90.doi:10.1136/leader-2017-000055
      OpenUrl
    6. ↵
      1. Whitworth J,
      2. Sewankambo NK,
      3. Snewin VA
      . Improving implementation: building research capacity in maternal, neonatal, and child health in Africa. PLoS Med 2010;7:e1000299. doi:10.1371/journal.pmed.1000299
    7. ↵
      1. Ben Charif A,
      2. Zomahoun HTV,
      3. LeBlanc A, et al
      . Effective strategies for scaling up evidence-based practices in primary care: a systematic review. Implement Sci 2017;12:139.doi:10.1186/s13012-017-0672-ypmid:http://www.ncbi.nlm.nih.gov/pubmed/29166911
      OpenUrlCrossRefPubMed
    8. ↵
      1. Zomahoun HTV,
      2. Ben Charif A,
      3. Freitas A, et al
      . The pitfalls of scaling up evidence-based interventions in health. Glob Health Action 2019;12:1670449. doi:10.1080/16549716.2019.1670449
    9. ↵
      1. Kruk ME,
      2. Yamey G,
      3. Angell SY, et al
      . Transforming global health by improving the science of scale-up. PLoS Biol 2016;14:e1002360. doi:10.1371/journal.pbio.1002360
    10. ↵
      1. Mangham LJ,
      2. Hanson K
      . Scaling up in international health: what are the key issues? Health Policy Plan 2010;25:85–96.doi:10.1093/heapol/czp066
      OpenUrlCrossRefPubMedWeb of Science
    11. ↵
      1. World Health Organization
      . Scaling up health services: challenges and choices. WHO, 2008.
    12. ↵
      1. Simmons R,
      2. Fajans P,
      3. Ghiron L
      . Scaling up health service delivery: from pilot innovations to policies and programmes. Geneva: World Health Organization, 2007.
    13. ↵
      1. Victora CG,
      2. Hanson K,
      3. Bryce J, et al
      . Achieving universal coverage with health interventions. The Lancet 2004;364:1541–8.doi:10.1016/S0140-6736(04)17279-6
      OpenUrl
    14. ↵
      1. Milat AJ,
      2. Newson R,
      3. King L
      . Increasing the scale of population health interventions: A guide. In: Evidence CfEa. North Sydney: NSW Ministry of Health, 2014.
    15. ↵
      1. Roberts SLE,
      2. Healey A,
      3. Sevdalis N
      . Use of health economic evaluation in the implementation and improvement science fields-a systematic literature review. Implement Sci 2019;14:72.doi:10.1186/s13012-019-0901-7pmid:http://www.ncbi.nlm.nih.gov/pubmed/31307489
      OpenUrlPubMed
    16. ↵
      1. Salomon JA
      . Integrating economic evaluation and implementation science to advance the global HIV response. J Acquir Immune Defic Syndr 2019;82 Suppl 3:S314–21.doi:10.1097/QAI.0000000000002219pmid:http://www.ncbi.nlm.nih.gov/pubmed/31764269
      OpenUrlPubMed
    17. ↵
      1. Adam T,
      2. Ebener S,
      3. Johns B, et al
      . Capacity utilization and the cost of primary care visits: implications for the costs of scaling up health interventions. Cost Eff Resour Alloc 2008;6:22.doi:10.1186/1478-7547-6-22pmid:http://www.ncbi.nlm.nih.gov/pubmed/19014524
      OpenUrlPubMed
    18. ↵
      1. Johns B,
      2. Baltussen R,
      3. Hutubessy R
      . Cost effectiveness and resource allocation. Cost Effective Resour Alloc 2003;1:1.doi:10.1186/1478-7547-1-1
      OpenUrl
    19. ↵
      1. Johns B,
      2. Torres TT
      . Costs of scaling up health interventions: a systematic review. Health Policy Plan 2005;20:1–13.doi:10.1093/heapol/czi001
      OpenUrlCrossRefPubMedWeb of Science
    20. ↵
      1. Carroll G,
      2. Safon C,
      3. Buccini G, et al
      . A systematic review of costing studies for implementing and scaling-up breastfeeding interventions: what do we know and what are the gaps? Health Policy Plan 2020;35:461–501.doi:10.1093/heapol/czaa005
      OpenUrl
    21. ↵
      1. Munk C,
      2. Portnoy A,
      3. Suharlim C, et al
      . Systematic review of the costs and effectiveness of interventions to increase infant vaccination coverage in low- and middle-income countries. BMC Health Serv Res 2019;19.doi:10.1186/s12913-019-4468-4
    22. ↵
      1. Drummond MF,
      2. Sculpher MJ,
      3. Claxton K
      . Methods for the economic evaluation of health care programmes. Oxford, United Kingdom: Oxford University Press, 2015.
    23. ↵
      1. Eisman AB,
      2. Kilbourne AM,
      3. Dopp AR, et al
      . Economic evaluation in implementation science: making the business case for implementation strategies. Psychiatry Res 2020;283:112433. doi:10.1016/j.psychres.2019.06.008
    24. ↵
      1. Pegurri E,
      2. Fox-Rushby JA,
      3. Damian W
      . The effects and costs of expanding the coverage of immunisation services in developing countries: a systematic literature review. Vaccine 2005;23:1624–35.doi:10.1016/j.vaccine.2004.02.029
      OpenUrlCrossRefPubMedWeb of Science
    25. ↵
      1. Batt K,
      2. Fox-Rushby JA,
      3. Castillo-Riquelme M
      . The costs, effects and cost-effectiveness of strategies to increase coverage of routine immunizations in low- and middle-income countries: systematic review of the grey literature. Bull World Health Organ 2004;82:689–96.pmid:http://www.ncbi.nlm.nih.gov/pubmed/15628207
      OpenUrlPubMed
    26. ↵
      1. Gomez GB,
      2. Borquez A,
      3. Case KK, et al
      . The cost and impact of scaling up pre-exposure prophylaxis for HIV prevention: a systematic review of cost-effectiveness modelling studies. PLoS Med 2013;10:e1001401. doi:10.1371/journal.pmed.1001401
    27. ↵
      1. Vassall A,
      2. Compernolle P
      . Estimating the resource needs of scaling-up HIV/AIDS and tuberculosis interventions in sub-Saharan Africa: a systematic review for national policy makers and planners. Health Policy 2006;79:1–15.doi:10.1016/j.healthpol.2005.11.005
      OpenUrlCrossRefPubMed
    28. ↵
      1. Marseille E,
      2. Jiwani A,
      3. Raut A, et al
      . Scaling up integrated prevention campaigns for global health: costs and cost-effectiveness in 70 countries. BMJ Open 2014;4:e003987. doi:10.1136/bmjopen-2013-003987
    29. ↵
      1. Turner HC,
      2. Toor J,
      3. Hollingsworth TD, et al
      . Economic evaluations of mass drug administration: the importance of economies of scale and scope. Clin Infect Dis 2018;66:1298–303.doi:10.1093/cid/cix1001pmid:http://www.ncbi.nlm.nih.gov/pubmed/29126255
      OpenUrlPubMed
    30. ↵
      1. Turner HC,
      2. Truscott JE,
      3. Fleming FM, et al
      . Cost-Effectiveness of scaling up mass drug administration for the control of soil-transmitted helminths: a comparison of cost function and constant costs analyses. Lancet Infect Dis 2016;16:838–46.doi:10.1016/S1473-3099(15)00268-6
      OpenUrl
    31. ↵
      1. Kumaranayake L
      . The economics of scaling up: cost estimation for HIV/AIDS interventions. AIDS 2008;22:S23–33.doi:10.1097/01.aids.0000327620.47103.1d
      OpenUrlCrossRefPubMedWeb of Science
    32. ↵
      1. Hoomans T,
      2. Severens JL
      . Economic evaluation of implementation strategies in health care. Implement Sci 2014;9:168.doi:10.1186/s13012-014-0168-ypmid:http://www.ncbi.nlm.nih.gov/pubmed/25518730
      OpenUrlPubMed
    33. ↵
      1. Simmons R,
      2. Shiffman J,
      3. Fajans P, et al
      . Scaling up health service innovations: a framework for action 2007.
    34. ↵
      1. WHOa E
      . Nine steps for developing a scaling-up strategy. Geneva: WHO, 2010.
    35. ↵
      1. Proctor E,
      2. Silmere H,
      3. Raghavan R, et al
      . Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health 2011;38:65–76.doi:10.1007/s10488-010-0319-7
      OpenUrlCrossRefPubMed
    36. ↵
      1. Vicki B,
      2. Huong T,
      3. Miranda B, et al
      . A narrative review of economic constructs in commonly used implementation and scale-up theories, frameworks and models. Health Res Policy Syst 2020;18:115.doi:10.1186/s12961-020-00633-6pmid:http://www.ncbi.nlm.nih.gov/pubmed/32998752
      OpenUrlPubMed
    37. ↵
      1. Gomersall JS,
      2. Jadotte YT,
      3. Xue Y, et al
      . Conducting systematic reviews of economic evaluations. Int J Evid Based Healthc 2015;13:170–8.doi:10.1097/XEB.0000000000000063
      OpenUrlPubMed
    38. ↵
      1. Streak J,
      2. Xue Y,
      3. Lockwood S
      . The Systematic Review of Economic Evaluation Evidence. Joanna Briggs Institute Reviewers’ Manual: The Joanna Briggs Institute, 2014.
    39. ↵
      1. Moher D,
      2. Shamseer L,
      3. Clarke M, et al
      . Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev 2015;4.doi:10.1186/2046-4053-4-1
    40. ↵
      1. WHO
      . Preamble to the constitution of who as adopted by the International health conference. New York: WHO, 1946: 100.
    41. ↵
      1. World Health Organization
      . International classification of health interventions. Geneva: World Health Organization, 2019.
    42. ↵
      1. Flay BR,
      2. Biglan A,
      3. Boruch RF, et al
      . Standards of evidence: criteria for efficacy, effectiveness and dissemination. Prev Sci 2005;6:151–75.doi:10.1007/s11121-005-5553-y
      OpenUrlCrossRefPubMedWeb of Science
    43. ↵
      1. Milat AJ,
      2. Newson R,
      3. King L, et al
      . A guide to scaling up population health interventions. Public Health Res Pract 2016;26:e2611604.doi:10.17061/phrp2611604pmid:http://www.ncbi.nlm.nih.gov/pubmed/26863167
      OpenUrlPubMed
    44. ↵
      1. Glanville J,
      2. Fleetwood K,
      3. Yellowlees A
      . Development and testing of search filters to identify economic evaluations in MEDLINE and EMBASE. Ottawa, ON: Canadian Agency for Drugs and Technologies in Health, 2009.
    45. ↵
      1. Jacobsen E,
      2. Boyers D,
      3. Avenell A
      . Challenges of systematic reviews of economic evaluations: a review of recent reviews and an obesity case study. Pharmacoeconomics 2020;38:259–67.doi:10.1007/s40273-019-00878-2
      OpenUrl
    46. ↵
      1. Wijnen BFM,
      2. Van Mastrigt G,
      3. Redekop WK, et al
      . How to prepare a systematic review of economic evaluations for informing evidence-based healthcare decisions: data extraction, risk of bias, and transferability (part 3/3). Expert Rev Pharmacoecon Outcomes Res 2016;16:723–32.doi:10.1080/14737167.2016.1246961
      OpenUrlPubMed
    47. ↵
      1. Gerkens S,
      2. Crott R,
      3. Cleemput I, et al
      . Comparison of three instruments assessing the quality of economic evaluations: a practical exercise on economic evaluations of the surgical treatment of obesity. Int J Technol Assess Health Care 2008;24:318–25.doi:10.1017/S0266462308080422
      OpenUrlCrossRefPubMedWeb of Science
    48. ↵
      1. Walker DG WR,
      2. Sharma R, et al
      . Best practices for conducting economic evaluations in health care: a systematic review of quality assessment tools. Rockville (MD): Agency for Healthcare Research and Quality (US), 2012.
    49. ↵
      1. Drummond MF,
      2. Jefferson TO
      . Guidelines for authors and peer reviewers of economic submissions to the BMJ. BMJ 1996;313:275–83.doi:10.1136/bmj.313.7052.275
      OpenUrlFREE Full Text
    50. ↵
      1. Moynihan R,
      2. Johansson M,
      3. Maybee A, et al
      . Covid-19: an opportunity to reduce unnecessary healthcare. BMJ 2020;370:m2752.doi:10.1136/bmj.m2752
      OpenUrlFREE Full Text
    51. ↵
      1. Graham ID,
      2. Tetroe JM
      . The knowledge to action framework. models and frameworks for implementing evidence-based practice: linking evidence to action 2010;207:222.
    52. ↵
      1. Canadian Institute for Health Research
      . Ethics guidance for developing partnerships with patients and researchers. Ottawa, 2020.

    Supplementary materials

    Footnotes

    • Twitter @HZomahoun, @SDM_ULAVAL, @JMassougbodji, @AgogovN, @MaudeLaberge

    • Contributors FB, ML, HTVZ and FL conceptualised the idea and developed the design for the systematic review. They developed the research questions which were discussed with NR, CBU, JM, AG, ST and OA and agreed upon by all authors. NR designed the search strategy which was reviewed by all authors. FB, CBU, JM, AG, ST and OA contributed to a preliminary process of article selection, which enabled further clarification of the research question and of eligibility criteria for the studies that would be included. Members of the executive committee (FB, ML, HTVZ, AG, NR and FR) contributed to the conception and design. FB drafted the initial version of the protocol which was critically revised by ML, HTVZ and FL. A revised version of the protocol was shared with coauthors who all provided a critical review of the protocol. All authors read and approved the final protocol.

    • Funding This review is funded by the Quebec Strategy for Patient-Oriented Research (SPOR)—Support for People and Patient-Oriented and Trials (SUPPORT) Unit (Grant number: #SU1-139759). This Unit is supported by the Canadian Institutes of Health Research (CIHR) and provincial partners, including the Ministère de la Santé et des Services sociaux (MSSS) du Québec and the Fonds de recherche du Québec-Santé (FRQ-S).

    • Competing interests None declared.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.