Eligibility criteria

Studies reporting on the development or application of frameworks (ie, theoretical or interpretive) to guide implementation or evaluation of DHIs in healthcare will be included. We will use the WHO definition of health which encompasses physical, mental and social well-being and spans across multiple disciplines such as psychology, sociology or medical sciences.11 DHI was defined as any health service or treatment delivered using technology that aims to facilitate, capture or exchange knowledge (formally or informally).5 DHI definition was generated from a search of the literature and consultations with digital health knowledge users, including policymakers, researchers, clinicians and developers. Implementation frameworks will be operationalised according to MRC guidance, as frameworks that aim to guide research into practice, which can include development, feasibility and dissemination frameworks.3 Evaluation frameworks will be defined as frameworks that focus on determining the effectiveness of DHIs, which includes measuring outcomes and understanding processes or mechanisms of action.3 No limitations will be placed on user population, comparators, study design, publication status or geographic region to enhance the comprehensiveness of our results and avoid unintended exclusion of relevant studies. Conference abstracts/proceedings and white papers will be included. We will include studies reported in other languages and use appropriate tools (ie, Google translate, translation services, contact author) to assess inclusion. Commentaries and studies examining mathematical or statistical frameworks will be excluded.

Information sources

An experienced information specialist developed the literature search in consultation with the multidisciplinary research team. The search will be peer reviewed by a second information specialist using the Peer Review of Electronic Search Strategy (PRESS) checklist to ensure the search is comprehensive and maximises appropriate search terms.12

We will search MEDLINE, EMBASE, CINAHL and PsychINFO using keywords such as ‘digital health’ and ‘framework’. Additional search terms were drawn from multiple disciplines such as psychology, nursing, sociology and medicine to ensure comprehensiveness. The databases will be searched from inception to present and the search strategy is presented in online supplementary appendix 1. We chose not to use the BeHEMoTh (behaviour of interest, health context, exclusions and models or theory) approach13 as specified in our OSF registration. Although this approach has been successful in identifying frameworks in knowledge translation,4 it did not prove to be a feasible approach in our scoping review as it yielded a vast number of citations with limited specificity related to our objectives. We used a simplified heuristic, which included identifying DHIs in various healthcare contexts, adding terms for frameworks and removing exclusions such as animal studies (online supplementary appendix 1).

The search strategy will be supplemented by a search for grey literature using the checklist suggested by the Canadian Agency for Drugs and Technologies in Health.14 Specifically, we will search for white papers or benefit evaluation studies through health technology assessment agencies such as Agency for Healthcare Research and Quality and National Institute for Health and Care Excellence, Canada Infoway and other relevant organisations involved in providing guidance on delivery of healthcare services. We will use keywords such as ‘digital health’, ‘frameworks’ and ‘benefits evaluation’ to refine our supplementary search. In addition, we will also scan reference lists of included studies and conduct a forward citation search (ie, examine studies that reference the included studies) in Web of Science using the cited reference search feature. This will ensure our approach is comprehensive.

Eligibility screening process

Citations obtained from the literature search will be uploaded to Covidence,15 a systematic review software program which organises citations, enables screening of citations by multiple reviewers and identifies discrepancies. We will apply a two-step process for identifying relevant citations. At level 1, titles and abstracts will be assessed using the eligibility criteria (online supplementary appendix 2). Studies with abstracts fulfilling criteria will be passed to level 2 where the eligibility criteria will be applied to the full text articles.

Prior to screening, a pilot test will be completed using a random sample of 10% of citations or full text articles, with the expressed purpose of assessing agreement between reviewers at each level. Specifically, percent agreement will be used to assess agreement among reviewers (inter-rater reliability ≥80% will be considered adequate). If agreement is not reached, a second pilot will be conducted with another random sample of 10%. A third reviewer will mediate any disagreements. Citations and full text articles will be screened in duplicate by two reviewers.

Data items and abstraction process

Studies fulfilling the eligibility criteria will be abstracted in Excel. We will extract the following study characteristics for the identified frameworks: name, reference, theory associated with framework (if applicable), description of its components or constructs and its application in research (or stage of research to which it was applied, if applicable). For studies outlining the application of a framework, additional characteristics will be abstracted such as the type of DHI, healthcare setting, method of application and nature and directionality of the results. We will abstract information such as name of the framework, the role of framework in study (ie, development, feasibility/pilot testing, implementation, evaluation), components of the framework that were used, type of DHI, the objective of the study (if applicable) and healthcare setting from included studies.

Methodological appraisal

We will not assess the quality of included articles in the scoping review (consistent with Joanna Briggs Institute reviewer’s manual7 as our purpose is to gain an overview of frameworks used in relation to DHIs and not to assess the quality of their application.

Ethics and dissemination

This scoping review is focused on published reports and studies of DHI and does not involve patients or primary data collection; as such, no formal ethics approval is required.

The dissemination plan will be tailored to end-users and will include passive and interactive strategies such as peer-reviewed publications, conference events and other network events with digital health knowledge users. To ensure broader reach, we will also disseminate our findings through social media platforms, and public-facing communications such as one-page briefs released on the Women’s College Institute for Health Innovation website at Women’s College hospital.

Patient and public involvement

We employed an integrated knowledge translation strategy to engage digital health knowledge users in the review process to ensure the scope of the project met the needs of various end-users. Knowledge users are defined as individuals who are likely to use the findings to inform health decision making.2 A priori, we decided to engage senior leaders and policymakers at organisations that promote or support implementation of digital health solutions, as well as researchers, clinicians and developers evaluating DHIs in real-world settings. An advisory panel of digital health knowledge users was established to provide input at strategic phases of the scoping review.

Potential panelists were identified through organisational networks and were invited to participate via email. Six members agreed to participate (CSG, TS, HCW, JZ, SM, DL) on the advisory panel. Panelists and the research team convened a meeting and discussed the strategic steps and opportunities for involvement and input in the review. Specifically, the advisory panel will support refinement of inclusion criteria, prioritisation of data abstraction elements, assist in interpretation of findings and develop dissemination strategies. Panelists have national and international networks that will ensure the scope of the review reflects the knowledge needs of a diverse audience, which is directly in line with the stated aim of providing practical guidance on the selection and application of frameworks for DHIs. As the intended audience of this paper does not include patients and members of the general public, they were not included as part of the advisory panel. The perspectives of patients and the general public will be incorporated through their participation and involvement in the respective studies included as part of this review.

Analysis

Included studies will be summarised using qualitative description, an approach that seeks to create an understanding of phenomenon through accessing the meanings ascribed by authors.16 Descriptions of individual frameworks will be organised by key categories, including study design, report type (published vs non-published), methodological approach (ie, how the framework is intended to be applied) and application papers (ie, how the framework has been applied in practice). We will then synthesise findings by mapping core components of the frameworks and examining how research objectives and type of DHIs are linked to the framework. Categorisation will use language directly from included studies, where possible, and authors will be contacted when information is not present or unclear. The advisory panel will guide the synthesis of findings by providing input on the level of detail abstracted from included articles and provide input on categorisation of frameworks, where appropriate.

Strengths and limitations

To our knowledge, this is the first scoping review to examine the use of frameworks to guide implementation or evaluation of DHIs on a broad scale. The protocol was generated using established methods for the conduct of scoping reviews and informed by input from digital health knowledge users to define scope and ensure the relevance of the project. A clear understanding of which frameworks can be used for development, feasibility, implementation and evaluation of DHIs will facilitate decision making by making evidence-based approaches available to policymakers, clinicians and developers. Additionally, this guidance will support researchers in identifying appropriate frameworks with the goal of establishing consistency across studies, minimising duplication and accelerating scientific progress.

Given the breadth of this scoping review, we anticipate a few key challenges. The first relates to the inconsistent and often ill-defined nature of DHIs and frameworks. To be inclusive, we have defined DHIs broadly as any health intervention that can be delivered using technology to ensure we capture frameworks that are currently being used across formal (eg, care delivered within the walls of a healthcare organisation) and informal settings (eg, direct to consumer technologies). Moreover, use of the term framework itself also creates challenges. For the purposes of this scoping review, we have defined a framework as a tool to systematically organise and link research questions or constructs, but a range of terms are often used synonymously (eg, models or processes). To account for this variability, we will include studies reporting on ‘models’ and work closely with the advisory panel to confirm whether the reported framework aligns with our a priori definition, as well as the needs of relevant digital health knowledge user groups.

Second, we also anticipate challenges searching the literature as a product of inconsistent terminology outlined above. We have constructed our search to balance comprehensiveness and specificity, working closely with an information specialist to ensure the number of citations are focused and feasible. Several iterations of the literature search were conducted, specifically, we added in keywords and removed them in a stepwise fashion to understand the impact on specificity and sensitivity of our search. Through this iterative process, we developed our search strategy, which was then peer reviewed using the PRESS checklist; however, we anticipate additional challenges when screening.

Third, we anticipate challenges arising from poor reporting or limited description, as evidenced by previous studies.17 18 Authors may not provide sufficient details on the frameworks they use or their method of application.19 To mitigate this, we will contact authors to obtain additional information whenever information is missing or unclear.

Finally, we anticipate that some included frameworks will have a dual purpose of addressing implementation and evaluation or may contain components that lend themselves to both constructs. We will convene with the advisory committee on a quarterly basis to discuss these issues as they arise and will devise the most appropriate plan for analysis through group consensus.

Overall, identification of frameworks will serve as a guide for researchers, clinicians, policymakers and developers of DHIs by providing practical guidance on which frameworks may be most appropriate for which objectives (ie, implementation or evaluation). In parallel, the results will contribute to a more nuanced understanding of how to evaluate and implement DHIs, including the identification and understanding of key constructs.