You are here

Article Text

Article menu

Download PDFPDF

XML
Health informatics
Protocol
Type and use of digital technology in learning health systems: a scoping review protocol
  1. Lysanne Lessard1,2,
  2. Agnes Grudniewicz1,2,
  3. Antoine Sauré1,
  4. Agnieszka Szczotka3,
  5. James King4,
  6. Michael Fung-Kee-Fung5,6
  1. 1 Telfer School of Management, University of Ottawa, Ottawa, Ontario, Canada
  2. 2 Institut du Savoir Montfort - Research, Ottawa, Ontario, Canada
  3. 3 University of Ottawa, Ottawa, Ontario, Canada
  4. 4 Children’s Hospital of Eastern Ontario, Ottawa, Ontario, Canada
  5. 5 Departments of Obstetrics-Gynaecology and Surgery, Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada
  6. 6 The Ottawa Hospital – General Campus, University of Ottawa/Ottawa Regional Cancer Centre, Ottawa, Ontario, Canada
  1. Correspondence to Professor Lysanne Lessard; lessard{at}telfer.uottawa.ca

Abstract

Introduction Health systems in North America and Europe have been criticised for their lack of safety, efficiency and effectiveness despite rising healthcare costs. In response, healthcare leaders and researchers have articulated the need to transform current health systems into continuously and rapidly learning health systems (LHSs). While digital technology has been envisioned as providing the transformational power for LHSs by generating timely evidence and supporting best care practices, it remains to be ascertained if it is indeed playing this role in current LHS initiatives. This paper presents a protocol for a scoping review that aims at providing a comprehensive understanding of how and to what extent digital technology is used within LHSs. Results will help to identify gaps in the literature as a means to guide future research on this topic.

Methods and analysis Multiple databases and grey literature will be searched with terms related to learning health systems. Records selection will be done in duplicate by two reviewers applying pre-defined inclusion and exclusion criteria. Data extraction from selected records will be done by two reviewers using a piloted data charting form. Results will be synthesised through a descriptive numerical summary and a mapping of digital technology use onto types of LHSs and phases of learning within LHSs.

Ethics and dissemination Ethical approval is not required for this scoping review. Preliminary results will be shared with stakeholders to account for their perspectives when drawing conclusions. Final results will be disseminated through presentations at relevant conferences and publications in peer-reviewed journals.

  • digital technology
  • learning health systems
  • scoping review

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

https://doi.org/10.1136/bmjopen-2018-026204

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Strengths and limitations of this study

    • The strengths of this scoping review include the timeliness of its topic and the use of systematic methods for identifying, selecting and extracting data from academical and grey literature with support from a management and health sciences librarian.

    • The review is also strengthened by meaningful stakeholder engagement. One knowledge user (MF-K-F) has led the creation of a successful regional learning health system focused on cancer care, while the other (JK) is a health informatics expert who has led multiple medical informatics development and implementation projects . Both stakeholders are also physicians.

    • The limitations of this review are the possibility of missing relevant records as a result of limiting the search strategy to records self-identified as learning health system (LHS), the possibility of including records that do not fully correspond to LHSs and the absence of quality assessment of selected records in scoping review methods.

    Introduction

    Health systems in North America and Europe have been criticised for their lack of safety, efficiency and effectiveness despite rising healthcare costs, seen to arise in large part from a significant lag between biomedical discovery and care implementation.1 2 In response, a number of researchers and healthcare leaders proposed to leverage the huge volumes of data contained within electronic health records (EHRs) by using advanced computing capabilities to enable rapid learning within health systems.3 This vision to transform current health systems into learning health systems (LHSs), also called rapid-learning health systems,2 was first articulated by the Institute of Medicine (IoM) in 2006 as a call for ‘the development of a continuously learning health system in which science, informatics, incentives and culture are aligned for continuous improvement and innovation, with best practices seamlessly embedded in the delivery process and new knowledge captured as an integral by-product of the delivery’.4 This vision, which stands as the most authoritative definition of LHSs was revised in 2013 by integrating patient-clinician partnerships and the identification of both effectiveness and efficiency of care as a result of real-time improvement.5

    Digital health data and related technology are taken to provide the information backbone and transformational power for LHSs by generating timely evidence and supporting best care practices.6 7 As such, a LHS can be understood as a cyber-social ecosystem in which digital technology is a support and catalyst for information processing.7 Digital technology refers to the hardware and software that can process and transmit digital information. Examples of digital technology that can be used in LHSs include EHRs, databases, analytics tools and visual dashboards.

    The IoM vision has fostered the emergence of numerous LHS initiatives at different scales and scope across the world.8 Despite these early innovations, many challenges remain in implementing required changes, including how best to use new technology for comparative effectiveness research, the access to and safe storage of patient-level data and full interoperability among EHR systems and third-party apps and software.7 9 Moreover, the type of digital technology used in LHS initiatives can vary widely, from existing operational and clinical systems,10 11 to data warehouses and analytics interfaces developed specifically for the initiative.12 As a result, it is difficult to assess if digital technology is being leveraged to develop and implement an infrastructure for LHSs as it had been envisioned.6

    There have been recent attempts at synthesising knowledge about LHSs through various types of literature reviews. However, most of them do not focus on digital technology and instead examine core competencies of LHS researchers,13 ethical issues raised for stakeholders14 or characteristics of and gaps in region-specific rapid-learning health systems.15 Other reviews present information related to digital technology, but they do not provide the detailed and comprehensive information needed to understand its role in current LHSs. For example, a systematic literature review that summarises attempts at implementing the LHS paradigm and evaluating its impacts on medical practice identifies three approaches at implementing LHSs that rely at least in part on digital technology (eg, clinical data reuse), but only provides limited information on the technologies being used within each approach.16 A similar example is a comprehensive review resulting in an LHS taxonomy in which a research approach such as comparative effectiveness research is categorised on par with a digital technology such as clinical decision support systems.17 Another study combines a literature review with expert interviews and focus groups to assess the impact of different types of LHSs on quality within healthcare providers18; as part of its results, the study identifies six types of LHSs that combine technology-based and methods-based categories (eg, clinical decision support and comparative effectiveness research).

    This paper presents a protocol for a scoping review that examines the extent and use of digital technology within self-identified LHSs. The purpose of this scoping review is to identify gaps in the literature as a means to guide future research on this topic. It is thus limited to self-identified LHSs since literature that does explicitly refer to LHSs is unlikely to conceive of outputs and structures in line with the IoM vision. The research aim will be answered by addressing the following objectives: (1) Identify the types of digital technology used or proposed for use in LHSs, (2) Summarise their development and context of use and (3) Describe the ways in which digital technology supports LHSs. The first objective serves to clarify the scope of the review by anchoring the concept of ‘digital technology’ in a classification of health information technology and systems derived from the literature.19–23 The second objective will provide an overview of how digital technology is developed and used within LHSs. The third objective will allow for mapping the use of digital technology in terms of the types of LHSs and the phases of learning being or proposed to be supported by digital technology.17 24 Other uses of digital technology within LHSs may emerge through data charting, resulting in a framework providing a ‘best fit’ to evidence in the literature.25

    Methods

    A scoping review is ideal for the research aim because it allows for a synthesis of information from both grey and academical literature, thereby ensuring that the latest research initiatives are captured and accounted for.

    We follow Arksey and O’Malley’s framework for undertaking scoping reviews,26 as well more recent recommendations on how to improve it.27–29 We follow these methods to articulate the stages of the review process: (1) Identifying purpose and research question, (2) Identifying relevant records, (3) Selecting records, (4) Charting the data, (5) Collating, summarising and reporting the results and (6) Consulting with relevant stakeholders to interpret findings.26 These stages are meant to be applied in an iterative manner, where researchers may revisit initial decisions and repeat some of the steps as they become more familiar with the literature.26 27 The results of applying the records identification and selection methods will be communicated using a Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow diagram.30

    Records identification

    The search strategy was developed by an experienced health sciences and management librarian (ASz) in consultation with review authors, including two knowledge users (MF-K-F and JK). In line with guidelines for conducting scoping reviews,29 the search strategy was piloted and refined by executing an initial limited search. Specifically, searches using broad terms (‘learning health’ AND (digital OR technolog*)) were executed in MEDLINE and Scopus, and the titles and abstracts of the most relevant results were analysed to identify any additional terms to be added. However, given the variation in terms used to refer to digital technology and the manageable size of the body of literature on LHSs, only variations of the term LHSs will be used for the final search strategy. Synonyms and variations of the term ‘learning health system’ were derived from seminal literature on this topic.2 6 7 The search strategy used in MEDLINE is available as an online supplementary file 1. The search terms will be adapted for use with other bibliographical databases as appropriate.

    Supplemental material

    Since relevant literature may be published in health, informatics and information technology publications, we will search the following electronic bibliographical databases: MEDLINE (OVID), Cochrane Database of Systematic Reviews (OVID), Health Technology Assessment (OVID), ABI/INFORM (Proquest) and Scopus. Two reviewers will independently hand-search the journal ‘Learning Health Systems’, manually applying the same eligibility criteria to the titles and abstracts of each published volume. The reference lists of included records and published systematic or scoping reviews on LHSs will be independently screened by two reviewers. The search strategy was validated by ensuring that a preselected key set of relevant articles were retrieved when the search was executed. Five hundred and forty-nine articles were identified when the search was first piloted in MEDLINE (OVID).

    The search strategy for grey literature follows guidelines for applying systematic review search methods to the grey literature31 32 and documenting the results in a transparent manner.33 We will use five different search strategies for grey literature: (1) grey literature databases, (2) the Grey Matters checklist created by the Canadian Agency for Drugs and Technology in Health34, (3) customised searches in Google, (4) targeted websites and (5) consultation with experts. Grey literature databases include Open Grey, Proquest Dissertations & Thesis Global and New York Academy of Medicine’s Grey Literature Report.32 Targeted websites include the American Medical Informatics Association (AMIA) knowledge centre and the websites of identified agencies and organisations known to advance LHS research (eg, Cancer Care Ontario, BORN Ontario, BC Cancer Agency, Cancer UK, the Agency for Healthcare Research and Quality, Kaiser Permanente, UnitedHealth Group, InterMountain Healthcare, PCORnet, the TRANSFoRm Project EU, the UK Learning Healthcare Project). To manage the scope of searches through Google, we will follow recommendations to review the first ten pages of each search’s hits, so as to capture the most relevant hits while maintaining feasibility.31 Experts in the field of learning health systems will be contacted to help identify additional published or unpublished work. Grey literature will be searched after academical literature has been selected for inclusion. Doing so will ensure that reviewers have become well-acquainted with the type of content relevant for the review.

    The search will be restricted to English-language publications records published from 1 January, 2007, up to the search date; this limit reflects the year in which the IoM published its seminal report on LHS35 and has been used in other systematic or comprehensive reviews of LHSs.17 18 We will not restrict by study type. All decisions made during the search process, including any decision to limit the scope of the search for reasons of feasibility, will be duly recorded, justified and communicated with the results of the review. Endnote software will be used to store and manage the bibliographical information of identified articles and report.

    Records selection

    Records selection will follow a two-phase process.29 In the first phase, two reviewers will independently apply inclusion and exclusion criteria to the title and abstract of identified articles and reports (or to the introduction sections of identified records when an abstract is not provided). Records that describe the use or proposed use of any digital technology in the context of any LHSs will be included. Records that mention one or more terms related to LHSs, but where LHSs are not the context or focus of the study, will be excluded. Each identified record will be marked as ‘include’, ‘exclude’ or ‘uncertain’ by each reviewer.

    In the second phase, records marked as ‘include’ and ‘uncertain’ will be reviewed independently by each reviewer in full text and assessed against the same inclusion and exclusion criteria. An additional exclusion criterion to be applied at this stage will concern the presence of sufficient information on digital technology and its context of use or proposed use. This is an important dimension for this review, since mere mentions of a digital technology in an article or report on LHSs would not allow charting the data in an appropriate manner. Records will thus be assessed in terms of the presence of: (a) a description of at least one digital technology and (b) a description of how that technology is or could be used in a LHS. Based on these criteria, records will be marked as ‘include’ or ‘exclude’ by each reviewer.

    A pilot-tested screening form developed for this review will be used throughout the process. Record selection will be supported by the software package DistillerSR, which is an online software that supports the screening process of systematic reviews (https://www.evidencepartners.com/products/distillersr-systematic-review-software/). Reviewers will meet regularly during the record selection process to discuss any challenges and disagreements over the eligibility of particular records in the first or second phase. The inter-rater reliability of their screening decision at each screening phase will be calculated using Cohen’s kappa and communicated with the results of the review. Discrepancies, hence records for which reviewers reach different decisions, will be resolved through discussions and, if necessary, reviewers will consult a third reviewer to resolve any ambiguity or disagreement. Discussions could lead to modifications to inclusion and exclusion criteria. Any change thereof will be duly recorded and communicated with the results of the review.

    Data charting

    Data charting refers to the extraction and organisation of qualitative data from selected records according to key issues and themes, a process called data extraction in systematic literature reviews.26 A data charting form will be created in order to extract the following information from each eligible paper:

    1. Bibliographical information (eg, authors, year of publication, source).

    2. Type of content (eg, theoretical, empirical).

    3. Type of digital technology (eg, EHR, decision support system, computerised provider order entry, electronic results reporting, laboratory information system, wearable devices19–23 36).

    4. Activities related to the development process of each identified digital technology (eg, planning, designing, developing, testing, etc).

    5. Types of data inputs for each identified digital technology (eg, clinical data, health service operations and financial data, public health programmes, patient-reported data37).

    6. Context of use of each identified digital technology (eg, location, scale of health system, disease, health and patient-reported outcomes, number of users, user roles).

    7. Consequences of use of each identified digital technology within a LHS (eg, intended,38 unintended39).

    8. Degree of implementation of each identified digital technology (eg, none, concept, prototype, pilot implementation, full implementation).

    9. Presence and method of evaluation of the effectiveness of each identified digital technology.

    10. Type of LHS supported or proposed to be supported by each identified digital technology (eg, deviance identification, risk modelling, clinical decision-making, intelligent automation16–18).

    11. Phase(s) of learning supported or proposed to be supported by each digital technology (eg, decision to study, data aggregation, data analysis, data interpretation, feedback to decision-makers, practice change, measurement of change impact24).

    The use of a pre-defined form for charting data facilitates subsequent summarising of extracted information28; specifically, we use a directed approach where initial categories are defined from extant literature or theory, but where additional categories can be identified through examination of a corpus.40 The charting form will be piloted on five of the selected records and improved as needed prior to full data charting. At this stage, the ability to answer the scoping review’s questions from extracted data will be assessed by the research team to ensure that the evidence base enables achieving the scoping review’s objectives. Changes to the form or a revision of the research question and research objectives may follow.

    Following best practices for charting data,27 28 two reviewers will then independently and in duplicate review a small sample of selected articles to determine whether their approach to data extraction is consistent, discussing any discrepancies and making necessary changes to the charting form. Data charting of the remaining selected articles and reports will not be executed in duplicate once the form has been finalised.

    Synthesis, presentation and discussion of results

    Results will first be synthesised through a descriptive numerical summary and a narrative account of the literature.26 The numerical summary will characterise included records in terms of their year of publication, type of record, context of use, etc. The literature will also be organised thematically in terms of the type of digital technology identified, their degree of implementation and the evaluation of their effectiveness. A framework-based synthesis approach will be used to map digital technology use onto known and emerging dimensions of LHSs, helping to understand the way in which digital technology is being leveraged for LHSs.25

    While the exact format for reporting the numerical and narrative summaries cannot be fully decided before data extraction, we envision creating an overview table to communicate the numerical summary and data charts to communicate the results of the thematic analysis. A visual representation of LHS dimensions will most likely be used to report the results of the mapping exercise.

    This review will identify research gaps and avenues for future research and may recommend practices for reporting on the design and use of digital technology in LHSs. Implications for practice and policy will be ascertained once results are finalised, in consultation with the study’s knowledge users (MF-K-F and JK) and additional stakeholders, as described below.

    Consultation and knowledge mobilisation

    We use two strategies for stakeholder engagement. The first strategy focuses on involving two physician knowledge users as active members of the research team. One knowledge user has led the creation of a successful regional learning health system focused on cancer care (MF-K-F), while the other is both a physician and a health informatics expert and has many years of experience leading medical informatics development and implementation projects (JK). These members are and will continue to be involved throughout the review process from the articulation of research objectives to the interpretation of the results. The second strategy focuses on consulting with additional stakeholders to add methodological rigour to the review process, and to integrate additional sources of perspectives, meaning and applicability to the scoping review.26 In this phase, preliminary results will be shared with stakeholders to support interpretation of preliminary findings for practice and policy, and to inform future research. We will try to identify stakeholders with varied expertise and playing a variety of roles within healthcare organisations, including information technology specialists, physicians and clinical managers. This will ensure that research findings account for perspectives related to the different dimensions of LHSs (organisational, technical, clinical, etc). Additional knowledge translation is planned through the dissemination of review findings at relevant conferences and the publication of results in peer-reviewed journals. The process of engaging stakeholders is meant to support knowledge mobilisation, in particular to build partnerships for the development of LHS initiatives.28

    Despite its contributions, this scoping review will have a number of limitations. The most important limitation is records selection bias that will result from limiting the search strategy to records self-identified as LHS. While this approach follows other systematic and rapid reviews,15 16 it could exclude sources that describe LHSs using related terms such as quality improvement.14 This choice is justified because the relationship between LHSs and related approaches remains to be clearly articulated despite a number of recent articles exploring the links among comparative effectiveness research, quality improvement and LHSs.41 42 Future research could focus on clarifying the characteristics of LHSs in order to differentiate between, for example, quality improvement projects of local scope and continuous quality improvement initiatives implemented across health organisations that may indeed qualify as LHSs. This issue leads to a second limitation, namely that self-identification as a basis for identification and selection of records could result in the inclusion of records that do not fully correspond to the IoM definition of LHSs. We mitigate this issue in part by the eligibility criteria used for screening, which will result in the exclusion of records that mention one or more terms related to LHSs but where LHSs are not the context or focus of the study. Nevertheless, we acknowledge that including records not truly referring to LHSs is a risk of the review. This risk is, again, partly mitigated through data charting items 10 and 11; indeed, these points will allow us to report on the number and content of records that either do not correspond to recognised types of LHSs, or that use or recommend digital technology that do not comprehensively support learning phases within an LHS. A third limitation is the absence of quality assessment of selected records, a limitation inherent to scoping reviews. An important implication of this limitation is that research gaps identified through a scoping review may reflect gaps in reporting methods rather than gaps in research practices.26 This limitation will be mitigated by clearly articulating if findings and conclusions are derived from missing data (thus potentially from reporting issues) or from data themselves.

    Patient and Public Involvement

    There was no involvement of patients or the public in the creation of this protocol.

    Conclusion

    This paper presents the protocol for a scoping review of academical and grey literature on the topic of digital technology for self-identified LHSs. It will provide a detailed yet holistic understanding of if and how digital technology is being used to make the LHS vision a reality, as envisioned by the individuals and organisations that articulated it.4 6 Doing so, it may identify research or reporting gaps related to the type of digital technology currently leveraged for LHSs, the transparency of their development methods, their degree of implementation and use and the support they provide to LHSs. The review also has policy implications as governments around the world look to improving the efficiency and outcomes of their health organisations and systems. Understanding how technology can support LHSs may contribute to funding planning and policies designed to stimulate local LHSs. Healthcare providers may also be interested in the results as digital technology is likely to change their practice over the years to come. Provider engagement will be essential to the successful implementation of LHSs and the digital technology that support them. Lastly, these findings will be used to articulate recommendations for future research.

    References

    1. 1.↵
      2. Friedman CP ,
      3. Wong AK ,
      4. Blumenthal D
      . Achieving a nationwide learning health system. Sci Transl Med 2010;2:57cm29.doi:10.1126/scitranslmed.3001456
      OpenUrlFREE Full Text
    2. 2.↵
      2. Etheredge LM
      . A rapid-learning health system. Health Aff 2007;26:w107–w118.doi:10.1377/hlthaff.26.2.w107
      OpenUrlAbstract/FREE Full Text
    3. 3.↵
      Rapid Learning: Getting Technology Into Practice. Health Aff 2007;26:w106.doi:10.1377/hlthaff.26.2.w106
      OpenUrlFREE Full Text
    4. 4.↵
      Institute of Medicine. Roundtable on Value & Science-Driven Health Care. Washington, DC: Institute of Medicine, 2006.
    5. 5.↵
      2. Smith M ,
      3. Saunders R ,
      4. Stuckhardt L , et al.
      , eds. Best care at lower cost; the path to continuously learning health care in America: National Academies Press, 2013.
    6. 6.↵
      2. Grossman C ,
      3. Powers B ,
      4. McGinnis JM
      . eds. Digital Infrastructure for the Learning Health System: The Foundation for Continuous Improvement in Health and Health Care. Washington, DC: National Academies Press, 2011.
    7. 7.↵
      2. Friedman C ,
      3. Rubin J ,
      4. Brown J , et al
      . Toward a science of learning systems: a research agenda for the high-functioning Learning Health System. J Am Med Inform Assoc 2015;22:43–50.doi:10.1136/amiajnl-2014-002977
      OpenUrlCrossRefPubMed
    8. 8.↵
      2. Lessard L ,
      3. Michalowski W ,
      4. Fung-Kee-Fung M , et al
      . Architectural frameworks: defining the structures for implementing learning health systems. Implement Sci 2017;12:78.doi:10.1186/s13012-017-0607-7
      OpenUrlPubMed
    9. 9.↵
      2. Etheredge LM
      . Rapid learning: a breakthrough agenda. Health Aff 2014;33:1155–62.doi:10.1377/hlthaff.2014.0043
      OpenUrlAbstract/FREE Full Text
    10. 10.↵
      2. Fung-Kee-Fung M ,
      3. Maziak DE ,
      4. Pantarotto JR , et al
      . Regional process redesign of lung cancer care: a learning health system pilot project. Curr Oncol 2018;25:59.doi:10.3747/co.25.3719
      OpenUrl
    11. 11.↵
      2. Lowes LP ,
      3. Noritz GH ,
      4. Newmeyer A , et al
      . ’Learn From Every Patient': implementation and early results of a learning health system. Dev Med Child Neurol 2017;59.doi:10.1111/dmcn.13227
    12. 12.↵
      2. Amin W ,
      3. Tsui FR ,
      4. Borromeo C , et al
      . PaTH: towards a learning health system in the Mid-Atlantic region. J Am Med Inform Assoc 2014;21:633–6.doi:10.1136/amiajnl-2014-002759
      OpenUrlCrossRefPubMed
    13. 13.↵
      2. Forrest CB ,
      3. Chesley FD ,
      4. Tregear ML , et al
      . Development of the Learning Health System Researcher Core Competencies. Health Serv Res 2018;53:2615–32.doi:10.1111/1475-6773.12751
      OpenUrl
    14. 14.↵
      2. McLennan S ,
      3. Kahrass H ,
      4. Wieschowski S , et al
      . The spectrum of ethical issues in a Learning Health Care System: a systematic qualitative review. Int J Qual Health Care 2018;30:161–8.doi:10.1093/intqhc/mzy005
      OpenUrl
    15. 15.↵
      2. Lavis JN ,
      3. Gauvin F-P ,
      4. Reid R , et al
      . Rapid Synthesis: Creating a Rapid-learning Health System in Ontario: McMaster University, 2018:59.
    16. 16.↵
      2. Budrionis A ,
      3. Bellika JG
      . The Learning Healthcare System: Where are we now? A systematic review. J Biomed Inform 2016;64:87–92.doi:10.1016/j.jbi.2016.09.018
      OpenUrl
    17. 17.↵
      2. McLachlan S ,
      3. Potts HWW ,
      4. Dube K , et al
      . The Heimdall Framework for Supporting Characterisation of Learning Health Systems. J Innov Health Inform 2018;25:77–87.doi:10.14236/jhi.v25i2.996
      OpenUrl
    18. 18.↵
      2. Foley TJ ,
      3. Vale L
      . What role for learning health systems in quality improvement within healthcare providers? Learn Health Syst 2017;1:e10025.doi:10.1002/lrh2.10025
      OpenUrl
    19. 19.↵
      2. Chaudhry B ,
      3. Wang J ,
      4. Wu S , et al
      . Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med 2006;144:742–52.doi:10.7326/0003-4819-144-10-200605160-00125
      OpenUrlCrossRefPubMedWeb of Science
    20. 20.↵
      2. Coiera E
      . Guide to Health Informatics. 3rd edn: CRC Press, 2015:710.
    21. 21.↵
      2. Shortliffe EH ,
      3. Cimino JJ
      , eds. Biomedical Informatics: Computer Applications in Health Care and Biomedicine. 3rd edn. New York, NY: Springer, 2006.
    22. 22.↵
      2. Wright A ,
      3. Sittig DF ,
      4. Ash JS , et al
      . Development and evaluation of a comprehensive clinical decision support taxonomy: comparison of front-end tools in commercial and internally developed electronic health record systems. J Am Med Inform Assoc 2011;18:232–42.doi:10.1136/amiajnl-2011-000113
      OpenUrlCrossRefPubMed
    23. 23.↵
      2. Yusof MM ,
      3. Papazafeiropoulou A ,
      4. Paul RJ , et al
      . Investigating evaluation frameworks for health information systems. Int J Med Inform 2008;77:377–85.doi:10.1016/j.ijmedinf.2007.08.004
      OpenUrlCrossRefPubMed
    24. 24.↵
      2. Friedman CP ,
      3. Omollo K ,
      4. Rubin J , et al
      . Learning Health Sciences and Learning Health Systems. 2015b.
    25. 25.↵
      2. Dixon-Woods M
      . Using framework-based synthesis for conducting reviews of qualitative studies. BMC Med 2011;9:39.doi:10.1186/1741-7015-9-39
      OpenUrlCrossRefPubMed
    26. 26.↵
      2. Arksey H ,
      3. O’Malley L
      . Scoping studies: towards a methodological framework. Int J Soc Res Methodol 2005;8:19–32.doi:10.1080/1364557032000119616
      OpenUrlCrossRef
    27. 27.↵
      2. Daudt HM ,
      3. van Mossel C ,
      4. Scott SJ
      . Enhancing the scoping study methodology: a large, inter-professional team’s experience with Arksey and O’Malley’s framework. BMC Med Res Methodol 2013;13:48.doi:10.1186/1471-2288-13-48
      OpenUrlCrossRefPubMed
    28. 28.↵
      2. Levac D ,
      3. Colquhoun H ,
      4. O’Brien KK
      . Scoping studies: advancing the methodology. Implement Sci 2010;5:69.doi:10.1186/1748-5908-5-69
      OpenUrlCrossRefPubMed
    29. 29.↵
      2. Peters MD ,
      3. Godfrey CM ,
      4. Khalil H , et al
      . Guidance for conducting systematic scoping reviews. Int J Evid Based Healthc 2015;13:141–6.doi:10.1097/XEB.0000000000000050
      OpenUrlCrossRefPubMed
    30. 30.↵
      2. Moher D ,
      3. Liberati A ,
      4. Tetzlaff J , et al
      . Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ 2009;339:b2535.doi:10.1136/bmj.b2535
      OpenUrlFREE Full Text
    31. 31.↵
      2. Godin K ,
      3. Stapleton J ,
      4. Kirkpatrick SI , et al
      . Applying systematic review search methods to the grey literature: a case study examining guidelines for school-based breakfast programs in Canada. Syst Rev 2015;4:138.doi:10.1186/s13643-015-0125-0
      OpenUrl
    32. 32.↵
      Searching the Literature: A Guide to Comprehensive Searching in the Health Sciences, 2019. https://guides.library.utoronto.ca/c.php?g=577919&p=4123572 [Accessed 21 Jan 2019].
    33. 33.↵
      2. Stapleton J ,
      3. Fuller K ,
      4. Lenton E
      . How to Find & Document Grey Literature Toronto. Canada: University of Waterloo, University of Toronto, 2018. Available: https://guides.library.utoronto.ca/c.php?g=577919&p=4123572 [Accessed 10 Jan 2019].
    34. 34.↵
      Grey Matters: a practical tool for searching health-related grey literature: CADTH (Canadian Agency for Drugs and Technology in Health), 2015;8.
    35. 35.↵
      2. Olsen L ,
      3. Aisner D ,
      4. McGinnis J
      . eds. The Learning Healthcare System: Workshop Summary. Washington, DC: National Institutes of Health, 2007.
    36. 36.↵
      Institute of Medicine. Digital Infrastructure for the Learning Health System: The Foundation for Continuous Improvement in Health and Health Care: Workshop Series Summary. Washington, DC: The National Academies Press, 2011.
    37. 37.↵
      2. Robert JR
      . Embedding Research in the Learning Health System. Healthc Pap 2016;16:30–5.
      OpenUrl
    38. 38.↵
      2. Buntin MB ,
      3. Burke MF ,
      4. Hoaglin MC , et al
      . The benefits of health information technology: a review of the recent literature shows predominantly positive results. Health Aff 2011;30:464–71.doi:10.1377/hlthaff.2011.0178
      OpenUrlAbstract/FREE Full Text
    39. 39.↵
      2. Campbell EM ,
      3. Sittig DF ,
      4. Ash JS , et al
      . Types of unintended consequences related to computerized provider order entry. J Am Med Inform Assoc 2006;13:547–56.doi:10.1197/jamia.M2042
      OpenUrlCrossRefPubMed
    40. 40.↵
      2. Hsieh HF ,
      3. Shannon SE
      . Three approaches to qualitative content analysis. Qual Health Res 2005;15:1277–88.doi:10.1177/1049732305276687
      OpenUrlCrossRefPubMedWeb of Science
    41. 41.↵
      2. Ginsburg GS ,
      3. Kuderer NM
      . Comparative effectiveness research, genomics-enabled personalized medicine, and rapid learning health care: a common bond. J Clin Oncol 2012;30:4233–42.doi:10.1200/JCO.2012.42.6114
      OpenUrlAbstract/FREE Full Text
    42. 42.↵
      2. Chambers DA ,
      3. Feero WG ,
      4. Khoury MJ
      . Convergence of Implementation Science, Precision Medicine, and the Learning Health Care System: A New Model for Biomedical Research. JAMA 2016;315:1941–2.doi:10.1001/jama.2016.3867
      OpenUrlCrossRefPubMed

    Footnotes

    • Contributors LL, AG and AS conceived the study. LL drafted the manuscript and is its guarantor. LL, AG, AS, MF-K-F and JK contributed to the protocol design and plan. ASz developed and tested the search strategy. All the authors worked collaboratively to revise the manuscript, and read and approved the final version. All the authors made substantive intellectual contributions to the development of this protocol.

    • Funding This work was supported by a research grant from the Telfer School of Management, University of Ottawa. This funding will support the search, selection and data charting processes of the review. The Telfer School of Management is not involved in any other aspect of the project, such as the design of the project’s protocol and data collection and analysis. The funder will have no input on the interpretation or publication of the study results.

    • Competing interests None declared.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Patient consent for publication Not required.