You are here

Article Text

Article menu
Health informatics
Protocol
Human factors applications in the design of decision support systems for population health: a scoping review
  1. Holland Marie Vasquez1,
  2. Emilie Pianarosa2,
  3. Renee Sirbu2,
  4. Lori M Diemert2,
  5. Heather V Cunningham3,
  6. Birsen Donmez1,
  7. Laura C Rosella2
  1. 1Mechanical & Industrial Engineering, University of Toronto Faculty of Applied Science and Engineering, Toronto, Ontario, Canada
  2. 2Dalla Lana School of Public Health, University of Toronto, Toronto, Ontario, Canada
  3. 3Gerstein Science Information Centre, University of Toronto, Toronto, Ontario, Canada
  1. Correspondence to Dr Laura C Rosella; laura.rosella{at}utoronto.ca

Abstract

Introduction Public health professionals engage in complex cognitive tasks, often using evidence-based decision support tools to bolster their decision-making. Human factors methods take a user-centred approach to improve the design of systems, processes, and interfaces to better support planning and decision-making. While human factors methods have been applied to the design of clinical health tools, these methods are limited in the design of tools for population health. The objective of this scoping review is to develop a comprehensive understanding of how human factors techniques have been applied in the design of population health decision support tools.

Methods and analysis The scoping review will follow the methodology and framework proposed by Arksey and O’Malley. We include English-language documents between January 1990 and August 2021 describing the development, validation or application of human factors principles to decision support tools in population health. The search will include Ovid MEDLINE: Epub Ahead of Print, In-Process and Other Non-Indexed Citations, Ovid MEDLINE Daily and Ovid MEDLINE 1946–present; EMBASE, Scopus, PsycINFO, Compendex, IEEE Xplore and Inspec. The results will be integrated into Covidence. First, the abstract of all identified articles will be screened independently by two reviewers with disagreements being resolved by a third reviewer. Next, the full text for articles identified as include or inconclusive will be reviewed by two independent reviewers, leading to a final decision regarding inclusion. Reference lists of included articles will be manually screened to identify additional studies. Data will be extracted by one reviewer, verified by a second, and presented according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for scoping reviews.

Ethics and dissemination Ethics approval is not required for this work as human participants are not involved. The completed review will be published in a peer-reviewed, interdisciplinary journal.

  • PUBLIC HEALTH
  • Health informatics
  • Information technology
http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

http://dx.doi.org/10.1136/bmjopen-2021-054330

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Strengths and limitations of this study

    • Our methodology was devised in consultation involving several multidisciplinary experts who advised on the rigour and feasibility of this review.

    • A librarian specialising in health science developed the search strategies for the health science databases and a librarian specialising in engineering consulted on the development of search strategies for the engineering databases.

    • Our review is limited to articles written in English and as such, presents a bias to Western research applications.

    • This scoping review does not include book chapters, theses, short papers, editorials, systematic reviews and conference abstracts, which may limit our findings.

    Background

    Human Factors Engineering (also referred to as Ergonomics, Cognitive Ergonomics, Engineering Psychology or Cognitive Engineering), an interdisciplinary field at the intersection of psychology and engineering, seeks to improve the design of systems by providing the best match between the characteristics of users (eg, physical, cognitive and perceptual abilities) and the operation of the tools they use.1 The discipline of human factors is generally considered to have originated during World War II within aviation during which more sophisticated systems were being developed, and pilot error in using such systems led to an increased interest in human capability.2 Of particular concern was how the design of controls and displays within the cockpit could better match the pilot’s physical, cognitive and perceptual abilities. Since the involvement of human factors engineers in the design of these systems, aviation has become the safest mode of transportation. Beyond aviation, human factors engineering methods have been applied to the design of complex systems in other domains, including ground transportation,3 the military,4 nuclear process control5 and healthcare.6 7

    Human factors engineers use a systematic approach to ensure that a given system meets the needs of the human, rather than forcing the human to adapt to the system. Accordingly, this allows the human to perform to the best of his or her abilities, make the best decisions possible, reducing physical and mental workload, and providing personal satisfaction. In doing so, human factors engineers employ a user-centred design methodology or design thinking, which aims to: (1) understand the people and the system that they interact with, (2) create a solution that meets the needs of the user, and (3) evaluate how well the solution meets the needs of the user and achieves the human factors objectives. Interviews, shadowing of human operators, task analysis, work domain analysis, in addition to a variety of other methods, are employed for the first stage, which lead to a set of design requirements. An iterative design and evaluation process is then followed, for example, by the creation of prototype designs and their evaluation in usability studies. In evaluating the system design, human factors engineers may measure user’s decision-making errors, efficiency (eg, task completion times), user stress or mental workload, as well as user satisfaction.

    Humans make decisions every day in a variety of domains, from piloting an aircraft, to diagnosing a patient, or determining whether to close in-person classes to slow the spread of the COVID-19 pandemic. Generally, these decisions will depend on one’s understanding of the situation by integrating multiple sources of information, determining what the information means and selecting the best course of action while considering the risks associated with each alternative.1 While normative decision theory models describe what decisions people should make (ie, the optimal decision), descriptive decision-making models account for how people actually make decisions. Real-life decision-making is complex in ways that normative decision models cannot account for. Real-world settings can include dynamic, uncertain, and continually changing conditions, and can require real-time decisions in high-stakes situations with significant consequences for mistakes. Limitations in human cognition and perception can contribute to decision errors. Decision support systems have the potential to support the user making better decisions and reduce decision errors. For example, clinical decision support tools have the potential in improving patient safety by improving the clinician’s diagnostic decisions.6 7 However, the success of decision support tools in clinical settings has been limited, in part due to human factors such as poor usability and workflow integration.8–11 Indeed, if human factors perspectives are not considered in their design (eg, how people make decisions, their expertise, their information needs), users may not leverage the tool.

    While human factors methods have been applied to the design of decision support tools to aid clinicians in decision-making tasks in healthcare settings,6 7 applications of human factors to support public health professionals in improving population health outcomes are limited. Population health can be defined as ‘the health outcomes of a group of individuals, including the distribution of such outcomes within the group’.12 The important distinction from clinical applications is that population health applications employ broader determinants that work across populations, such as social, economic, biology, early childhood development and health services.13 Accordingly, population health has a broad scope and ranges from physical and mental health to environmental health within a population, all encompassed within the public health sector.12 Public health professionals in provincial and local health departments engage in complex cognitive tasks to make the best possible decisions for resource allocation and public health planning. They do this based on their understanding of the current health status of their population, the factors that influence the health of the population and assess which interventions will work to address the health issues within the population based on available data.12 14 15 Evidence-based decision support tools, which use objective data to support the expertise of a decision-maker have been employed in many domains. Such tools have the potential to support public health professionals by answering complex questions, such as, what makes certain demographic groups within a population healthier than others.14

    There has been a proliferation of evidence-based decision support tools in population health, particularly since the onset of the COVID-19 pandemic. For example, Afzal and colleagues developed a visual analytics platform for public health professionals to forecast COVID-19 cases and explore the effects of various interventions (eg, school closures, stay at home orders) on cases.16 However, few studies have employed human factors methods to the design of these tools and evaluated their efficacy in supporting public health decision-making. For example, Afzal and colleagues focus on the development of the user interface but did not discuss how public health professionals’ needs were determined and factored into the design of the tool.16 Moreover, the proposed tool was not evaluated with public health professionals and as such, how users interpret the COVID-19 modelling scenarios, the quality of their decision-making, workload and satisfaction with the tool were not considered. Human factors methods would ensure that the tool met public health practitioner needs (eg, information to understand modelling assumptions or uncertainty) and facilitated optimal decision-making.

    Given that the focus is on populations and unique functions in public health, evidence-based decision support tools for public health professionals will have distinct user needs and requirements compared with other domains.15 17 Human factors engineers can apply user-centred design methodology in creating these decision support tools and can leverage other human factors methods in evaluating their efficacy in supporting public health professionals. By doing so, human factors can improve the design of these tools to better support public health professionals in decision-making efforts. For example, Pike and colleagues used an iterative user-centred design process to develop a decision support tool for child and youth injury surveillance and prevention.18 Injury prevention practitioners and policymakers were involved in an evaluation of the tool during which they were presented with a series of fictional planning problems to solve using the tool (eg, determine the trends for suicide and homicide for children aged 10–19 years old from 2007 to 2010). Following this exercise, users were interviewed during which they were asked to provide feedback on the tool (eg, the dashboard, indicators and specific visualisations) as well as provide feedback pertaining to the ease of navigating the dashboard and overall satisfaction. Results from the evaluation underscored the utility of the dashboard in injury surveillance and prevention, and highlighted painpoints and opportunities to improve on the dashboard’s design.

    De Lima and colleagues developed a decision support tool for aiding public health professionals in planning and decision-making processes in the context of infectious disease.19 Public health professionals completed a focus group session during which they interacted with the tool to build and run models for dengue fever. After interacting with the tool, users were asked to complete a questionnaire providing additional feedback. Overall, the results suggested that public health professionals could effectively use the tool for building and running models and scenarios. However, the authors noted that this process could be improved by providing users with documentation for how the model was developed and a guide on how to use the tool. Importantly, this evaluation was used to iterate on the design of the tool.

    These examples exemplify how human factors methods can be employed in the design and evaluation of evidence-based decision support tools to ensure that they meet the needs of public health professionals. The objective of the present scoping review is to build on this and provide a foundational understanding of the current landscape of human factors applications in the design of evidence-based decision support tools within population health.

    Research question

    We identified the research question following extensive discussions among the protocol authors to clarify the concept and the purpose of the review: How are human factors considered in the design and development of evidence-based decision support tools for population health applications?

    Methods

    This study question adheres to the population, concept and context framework used for scoping reviews.20 21 In this case, the population is general and not defined. We propose conducting a scoping review of how human factors methods have been applied to evidence-based decision support tools in the context of population health. This scoping review will follow the methodological framework described by Arksey and O’Malley20 with refinements proposed to the framework by Levac and colleagues.21 The reporting of this protocol and search have been informed by the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P and PRISMA-S) reporting guidelines, respectively, to facilitate understanding and transparency.22 23

    Registration and review stage

    The present research protocol will be registered with BMJ prior to beginning the study. The study is expected to commence in May 2021 with an anticipated completion date in March 2022.

    Operationalising population health, human factors and decision support tools

    One major challenge for this scoping review was operationalising the concepts of population health, human factors and decision support tools for the search strategy. As such, to aid in the codification of these concepts, our team includes a librarian specialising in health science. Additionally, we consulted with a librarian specialising in engineering.

    Population health was operationalised to encompass all aspects of public health in the broadest sense and is not limited to any specific aspects, such as chronic or infectious disease. Search terms included “population health”, “public health”, “community health”, “community medicine”, “health promotion”, “epidemiology” and “disease prevention”.

    Human factors was operationalised to encompass all aspects of human factors in the broadest sense and is not limited to a particular method or tool. Search terms included “human factors”, “ergonomics”, “cognitive ergonomics”, “cognitive analytics”, “usability”, “human engineering”, “human computer interaction”, “human-centered design”, “interface design”, “user interface”, “user evaluation”, “usability evaluation”, “user friendly”, “user experience” and “human machine interface”.

    Decision support tool was operationalised to encompass any electronic system to aid decision-making. Search terms included “decision support”, “decision support systems”, “decision support tool”, “information systems”, “data visualizations”, “visual analytics”, “informatics”, “data display” and “dashboard”.

    Search strategy

    The search strategy includes indexed databases of peer-reviewed literature and manual searches. We discuss each of these in turn:

    Peer-reviewed literature

    The published literature search will include Ovid MEDLINE: Epub Ahead of Print, In-Process and Other Non-Indexed Citations, Ovid MEDLINE Daily and Ovid MEDLINE 1946–present; EMBASE (on Ovid), Scopus, PsycINFO (Ovid), Compendex (Engineering Village), IEEE Xplore and Inspec (Engineering Village). Comprehensive literature searches were developed in collaboration with two librarians: one who specialised in health science and another who specialised in engineering. The search strategies used a combination of keywords, and subject headings relevant to each database for each concept. The databases were selected based on subject area coverage and functionality (see the online supplemental file 1 for our search strategy for each database).

    Results prior to 1990 were excluded in the search strategy. We do not expect articles related to human factors to the design of digital evidence-based decision support tools in population health as human factors applications in healthcare began to emerge in the 1990s. By including articles from 1990, we are capturing the potential evolution of the application of human factors in the public health domain. This research study only included primary studies, limited to the English language. A modified version of the systematic review filter developed by the Scottish Intercollegiate Guidelines Network was applied to exclude systematic reviews, scoping reviews, meta-analyses as well as editorials, guidelines, letters and patient education handouts. The MEDLINE search strategy was validated against a key set of eight articles15 18 19 24–28 predetermined by the authors and was peer reviewed using PRESS29 by another librarian, not associated with this study to ensure accuracy and comprehensiveness.

    Grey literature

    Our grey literature search strategy is guided by our research question. While government or organisation websites may contain dashboards and interfaces used by public health professionals to inform their population assessment and planning, information about the development and assessment of public health professional interactions with the interfaces or dashboards will not be available. In other words, the information will be about tools, but not the development or evaluation, which is needed for the human factors aspect of this review. As such, our grey literature search will be focused on capturing full-text conference proceedings papers identified through Compendex (Engineering Village), IEEE Xplore and Inspec (Engineering Village) to counter the positive reporting bias of the published article literature, ensuring the review is thorough and balanced.

    Manual searches

    Reviews of human factors and population health discovered in the formal peer-reviewed literature search will be identified and their references will be manually searched to identify additional articles for inclusion. Reference lists of included articles will also be manually screened to identify additional studies.

    Integration of results

    The results from the two search types will be integrated into Covidence, a systematic review management software, and duplicates will subsequently be removed. Screening for article inclusion will be completed using Covidence and will consist of two phases. First, the title and abstract of all identified articles will be screened independently by two reviewers on the research team and will be categorised as ‘include’, ‘exclude’ or ‘inconclusive’.21 Such judgements will be informed by the inclusion and exclusion criteria (see box 1) and will be documented using a piloted standardised relevance form. Disagreements will be resolved through team discussion and may include a third, independent reviewer, if necessary. Articles identified in the title and abstract screening will undergo full-text screening by two independent reviewers, which will lead to a final decision regarding inclusion.

    Box 1

    Exclusion criteria for literature search

    Exclusion criteria

    1. Articles whose end-user is not within population health.

    2. Articles not related to human factors.

    3. Articles that do not include a digital evidence-based decision support tool.

    4. Conference abstracts, as these may not include sufficient details on the methodology.

    5. Reviews, including commentaries and discussion pieces.

    6. Articles written in languages other than English.

    Inclusion and exclusion criteria

    We sought to limit the scope of our challenge by developing a priori eligibility criteria for the literature, detailing the types of literature to be included. The inclusion and exclusion criteria are presented in box 1.

    All documents created since 1990 in English that describe the development, validation or application guided by human factors principles to any study design in the field of population health will be included. Examples of studies not related to population health include clinical applications, such as studies that discuss patient safety, monitoring of an individual’s health or clinical decision support tools. Documents that have described the application of human factors in terms of the population of study and sample size, method, analysis, prototype and iteration process, end-user and intended setting will be included. Examples of studies not related to human factors may include studies that describe a tool as user-friendly but do not discuss the engagement of users in the design process or evaluate the tool with users (eg, in determining design requirements through interviews or focus group or in evaluating the tool with users).

    Data abstraction and synthesis

    A data abstraction form will be developed and pilot-tested using two researchers, working independently of each other. The data form will be tested on five to seven articles for consistency and comprehensiveness for capturing relevant study data. Changes will be made in a team meeting during which the team will compare pilot test results and discuss discrepancies. Following the article screening, data will be extracted from each article included in the review by one reviewer using the data extraction form and will be verified by a second reviewer. The data will be abstracted and synthesised according to three themes: study characteristics, study methods and human factors characteristics (see box 2).

    Box 2

    Data abstraction themes and items

    Study characteristics

    1. Authors.

    2. Academic discipline of authors.

    3. Year of publication.

    4. Type of publication (eg, peer-reviewed article, conference proceeding, dissertation, other).

    5. Publication venue (eg, journal or conference name).

    Study methods

    1. Study location (eg, country).

    2. Study design (eg, cross-sectional, cohort, case–control, qualitative, other).

    3. Description of the evidence-based decision support tool.

    4. Population health subject area (eg, infectious disease, non-communicable disease).

    5. Description of the subject area.

    6. Study goal (eg, development, validation, application, other).

    Human factors characteristics

    1. Study sample size.

    2. Human factors study method (eg, semistructured interviews, focus groups, Delphi process, survey, experiments, shadowing).

    3. Human factors analysis method (eg, descriptive, inferential, thematic).

    4. Prototyping/iteration (ie, did the study involve prototype or iterative design, yes or no).

    5. Decision-maker (ie, who is the tool being designed for).

    6. Setting (eg, hospitals, federal public health, regional public health, local public health, community health centre, other).

    Results will be synthesised, and summarised study characteristics will be presented using tables and figures. We will discuss key lessons learnt from the use of human factors in the design of decision support tools for public health. Additionally, the scoping review will discuss implications and future research directions for human factors applications in population health.

    Patient and public involvement

    This is a scoping review protocol and as such it was not appropriate or possible to involve patients or the public in the design, conduct, reporting or dissemination plans of our research.

    Ethics approval is not required for this knowledge synthesis, as we are not involving human participants. The completed scoping review will be submitted for publication to a peer-reviewed, interdisciplinary journal in addition to conferences on population health and human factors.

    Ethics statements

    Patient consent for publication

    Acknowledgments

    We would like to acknowledge Glyneva Bradley-Ridout, Liaison & Education Librarian at the University of Toronto, for peer reviewing the MEDLINE search strategy; Tracy Zahradnik from the Engineering Library at the University of Toronto for consulting on the engineering database search strategy; and Patricia Ayala from Gerstein Library at the University of Toronto for her guidance on article screening.

    References

    1. ↵
      1. Lee JD,
      2. Wickens CD,
      3. Liu Y
      . Designing for people: an introduction to human factors engineering. CreateSpace, 2017.
    2. ↵
      1. Salas E,
      2. Maurino D,
      3. Curtis M
      . Human factors in aviation: an overview. In: Hum. factors Aviat, 2010: 3–19.
    3. ↵
      1. Donmez B,
      2. Boyle LN,
      3. Lee JD
      . Safety implications of providing real-time feedback to distracted drivers. Accid Anal Prev 2007;39:581–90.doi:10.1016/j.aap.2006.10.003pmid:http://www.ncbi.nlm.nih.gov/pubmed/17109807
      OpenUrlCrossRefPubMed
    4. ↵
      1. Hollands JG,
      2. Spivak T,
      3. Kramkowski EW
      . Cognitive load and situation awareness for soldiers: effects of message presentation rate and sensory modality. Hum Factors 2019;61:763–73.doi:10.1177/0018720819825803pmid:http://www.ncbi.nlm.nih.gov/pubmed/30768371
      OpenUrlPubMed
    5. ↵
      1. Carvalho PVR,
      2. dos Santos IL,
      3. Gomes JO, et al
      . Human factors approach for evaluation and redesign of human–system interfaces of a nuclear power plant simulator. Displays 2008;29:273–84.doi:10.1016/j.displa.2007.08.010
      OpenUrlCrossRef
    6. ↵
      1. Salwei ME,
      2. Carayon P,
      3. Hoonakker PLT, et al
      . Workflow integration analysis of a human factors-based clinical decision support in the emergency department. Appl Ergon 2021;97:103498.doi:10.1016/j.apergo.2021.103498pmid:http://www.ncbi.nlm.nih.gov/pubmed/34182430
      OpenUrlPubMed
    7. ↵
      1. Carayon P,
      2. Hoonakker P,
      3. Hundt AS, et al
      . Application of human factors to improve usability of clinical decision support for diagnostic decision-making: a scenario-based simulation study. BMJ Qual Saf 2020;29:329–40.doi:10.1136/bmjqs-2019-009857pmid:http://www.ncbi.nlm.nih.gov/pubmed/31776197
      OpenUrlAbstract/FREE Full Text
    8. ↵
      1. Karsh B-T
      . Clinical practice improvement and redesign: how change in workflow can be supported by clinical decision support. Agency Healthc. Res. Qual 2009.;200943.
    9. ↵
      1. Bates DW,
      2. Kuperman GJ,
      3. Wang S, et al
      . Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc 2003;10:523–30.doi:10.1197/jamia.M1370pmid:http://www.ncbi.nlm.nih.gov/pubmed/12925543
      OpenUrlCrossRefPubMed
    10. ↵
      1. Sittig DF,
      2. Wright A,
      3. Osheroff JA, et al
      . Grand challenges in clinical decision support. J Biomed Inform 2008;41:387–92.doi:10.1016/j.jbi.2007.09.003pmid:http://www.ncbi.nlm.nih.gov/pubmed/18029232
      OpenUrlCrossRefPubMedWeb of Science
    11. ↵
      1. Kilsdonk E,
      2. Peute LW,
      3. Jaspers MWM
      . Factors influencing implementation success of guideline-based clinical decision support systems: a systematic review and gaps analysis. Int J Med Inform 2017;98:56–64.doi:10.1016/j.ijmedinf.2016.12.001pmid:http://www.ncbi.nlm.nih.gov/pubmed/28034413
      OpenUrlPubMed
    12. ↵
      1. Kindig D,
      2. Stoddart G
      . What is population health? Am J Public Health 2003;93:380–3.doi:10.2105/AJPH.93.3.380
      OpenUrlCrossRefPubMedWeb of Science
    13. ↵
      Public health agency of Canada, 2012. Available: https://www.canada.ca/en/public-health.html
    14. ↵
      1. Park S,
      2. Bekemeier B,
      3. Flaxman A, et al
      . Impact of data visualization on decision-making and its implications for public health practice: a systematic literature review. Inform Health Soc Care 2021:1–19.doi:10.1080/17538157.2021.1982949pmid:http://www.ncbi.nlm.nih.gov/pubmed/34582297
      OpenUrlPubMed
    15. ↵
      1. Revere D,
      2. Fuller S
      . Building a Customizable knowledge management environment to support public health practice: design strategies. 252. IEEE Computer Society, 2008.
    16. ↵
      1. Afzal S,
      2. Ghani S,
      3. Jenkins-Smith HC
      . A visual analytics based decision making environment for COVID-19 modeling and visualization. Proceedings - 2020 IEEE Visualization Conference, VIS, 2020:86–90.
    17. ↵
      1. Manuel DG,
      2. Rosella LC,
      3. Stukel TA
      . Importance of accurately identifying disease in studies using electronic health records. BMJ 2010;341:c4226.doi:10.1136/bmj.c4226pmid:http://www.ncbi.nlm.nih.gov/pubmed/20724404
      OpenUrlFREE Full Text
    18. ↵
      1. Pike I,
      2. Smith J,
      3. Al-Hajj S, et al
      . The Canadian atlas of child and youth injury: mobilizing injury surveillance data to launch a national knowledge translation tool. Int J Environ Res Public Health 2017;14:982.doi:10.3390/ijerph14090982pmid:http://www.ncbi.nlm.nih.gov/pubmed/28867782
      OpenUrlPubMed
    19. ↵
      1. de Lima TFM,
      2. Lana RM,
      3. de Senna Carneiro TG, et al
      . DengueME: a tool for the modeling and simulation of dengue spatiotemporal dynamics. Int J Environ Res Public Health 2016;13. doi:doi:10.3390/ijerph13090920. [Epub ahead of print: 15 09 2016].pmid:http://www.ncbi.nlm.nih.gov/pubmed/27649226
      OpenUrlPubMed
    20. ↵
      1. Arksey H,
      2. O'Malley L
      . Scoping studies: towards a methodological framework. Int J Soc Res Methodol 2005;8:19–32.doi:10.1080/1364557032000119616
      OpenUrlCrossRef
    21. ↵
      1. Levac D,
      2. Colquhoun H,
      3. O'Brien KK
      . Scoping studies: advancing the methodology. Implement Sci 2010;5:1–9.doi:10.1186/1748-5908-5-69pmid:http://www.ncbi.nlm.nih.gov/pubmed/20854677
      OpenUrlCrossRefPubMed
    22. ↵
      1. Moher D,
      2. Shamseer L,
      3. Clarke M, et al
      . Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev 2015;4:pp. 1–9.doi:10.1186/2046-4053-4-1
      OpenUrl
    23. ↵
      1. Rethlefsen ML,
      2. Kirtley S,
      3. Waffenschmidt S, et al
      . PRISMA-S: an extension to the PRISMA statement for reporting literature searches in systematic reviews. Syst Rev 2021;10:1–19.doi:10.1186/s13643-020-01542-z
      OpenUrlCrossRef
    24. ↵
      1. Sedig K,
      2. Parsons P,
      3. Dittmer M, et al
      . Beyond information access: support for complex cognitive activities in public health informatics tools. Online J Public Health Inform 2012;4. doi:doi:10.5210/ojphi.v4i3.4270. [Epub ahead of print: 19 12 2012].pmid:http://www.ncbi.nlm.nih.gov/pubmed/23569645
      OpenUrlPubMed
    25. ↵
      1. Yuan M,
      2. Powell G,
      3. Lavigne M
      . “Initial usability evaluation of a knowledge-based population health information system: The Population Health Record (PopHR). In: AMIA . Annu. Symp. proceedings. AMIA Symp, 2017: 1878–84. http://ovidsp.ovid.com/ovidweb.cgi?T=JS&PAGE=reference&D=emed18&NEWS=N&AN=625552211
    26. ↵
      1. Al-Hajj S,
      2. Pike I,
      3. Riecke BE
      . Visual analytics for public health: supporting knowledge construction and decision-making. Proceedings of the Annual Hawaii International Conference on System Sciences, 2013. p.:2416–23.
    27. ↵
      1. Scotch M,
      2. Parmanto B,
      3. Monaco V
      . Usability evaluation of the spatial OLAP visualization and analysis tool (SOVAT). J Usability Stud 2007;2:76–95 http://ovidsp.ovid.com/ovidweb.cgi?T=JS&PAGE=reference&D=pmnm2&NEWS=N&AN=26613012pmid:http://www.ncbi.nlm.nih.gov/pubmed/26613012
      OpenUrlPubMed
    28. ↵
      1. Harris J,
      2. Hinyard L,
      3. Beatty K, et al
      . Evaluating the implementation of a twitter-based foodborne illness reporting tool in the city of St. Louis department of health. Int J Environ Res Public Health 2018;15:833.doi:10.3390/ijerph15050833
    29. ↵
      1. McGowan J,
      2. Sampson M,
      3. Salzwedel DM, et al
      . PRESS peer review of electronic search strategies: 2015 Guideline Statement. J Clin Epidemiol 2016;75:40–6.doi:10.1016/j.jclinepi.2016.01.021pmid:http://www.ncbi.nlm.nih.gov/pubmed/27005575
      OpenUrlCrossRefPubMed

    Supplementary materials

    • Supplementary Data

      This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

    Footnotes

    • Twitter @ReneeSirbu, @LauraCRosella

    • Contributors LCR and BD conceived the study. HMV, EP, LMD, RS, LCR and BD jointly developed the research questions and HMV drafted the paper. HVC provided input on the search strategy and operationalisation of study concepts. All authors further revised the paper and approved of the final text.

    • Funding This study was funded by a grant from the X-Seed programme at the University of Toronto, jointly held by BD and LCR. BD holds a Canada Research Chair in Human Factors and Transportation and LCR holds a Canada Research Chair in Population Health Analytics.

    • Competing interests None declared.

    • Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.