Article Text
PDF
Abstract
Objectives Delivery of virtual care increased throughout the COVID-19 pandemic and persisted after physical distancing measures ended. However, little is known about how to measure the quality of virtual care, as current measures focus on in-person care and may not apply to a virtual context. This scoping review aims to understand the connections between virtual care modalities used with ambulatory patient populations and quality measures across the Quintuple Aim (provider experience, patient experience, per capita cost, population health and health equity).
Design Virtual care was considered any interaction between patients and/or their circle of care occurring remotely using any form of information technology. Five databases (MEDLINE, Embase, PsycInfo, Cochrane Library, JBI) and grey literature sources (11 websites, 3 search engines) were searched from 2015 to June 2021 and again in August 2022 for publications that analysed virtual care in ambulatory settings. Indicators were extracted, double-coded into the Quintuple Aim framework; patient and provider experience indicators were further categorised based on the National Academy of Medicine quality framework (safety, effectiveness, patient-centredness, timeliness, efficiency and equity). Sustainability was added to capture the potential for continued use of virtual care.
Results 13 504 citations were double-screened resulting in 631 full-text articles, 66 of which were included. Common modalities included video or audio visits (n=43), remote monitoring (n=11) and mobile applications (n=11). The most common quality indicators were related to patient experience (n=58 articles), followed by provider experience (n=25 articles), population health outcomes (n=23 articles) and health system costs (n=19 articles).
Conclusions The connections between virtual care modalities and quality domains identified here can inform clinicians, administrators and other decision-makers how to monitor the quality of virtual care and provide insights into gaps in current quality measures. The next steps include the development of a balanced scorecard of virtual care quality indicators for ambulatory settings to inform quality improvement.
- Quality in health care
- Telemedicine
- Patient-Centered Care
- Health Equity
Data availability statement
No data are available.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
The methodology used for this review was broad in scope, seeking to collate a comprehensive list of quality indicators of virtual care and included both published and grey literature.
Used established scoping review methods for literature search, study selection, data collection, coding and reporting to the quality indicators were mapped and categorised into established quality frameworks including the Quintuple Aim and the National Academy of Medicine quality domains.
Literature search was limited to articles published after 2015 and available in English language only.
Categorisation of indicators was complicated by the lack of reported detail.
Introduction
Virtual care is defined as any interaction between patients and/or caregivers and their healthcare providers (or ‘circle of care’) that occurs remotely and is facilitated through digital communication or other information technologies.1 Prior to the COVID-19 pandemic, virtual care was largely underused for delivering healthcare services in Canada.2 The COVID-19 pandemic changed the global healthcare landscape and accelerated the implementation and access to virtual care for many patients.2 Although utilisation of virtual care decreased after pandemic restrictions eased, the demands for the option of virtual care persist with many patients and/or caregivers appreciating the convenience that virtual care affords.3 4 Healthcare providers also see the value of virtual care in specific situations due to its potential to improve patient access to care especially for those with mobility issues or living in rural/remote areas, and for maintaining connections with patients in between in-person visits.5 The future of healthcare delivery will require the integration of both virtual and in-person modalities across the continuum of care.6 To meet these needs, virtual care needs to be accessible and high quality; however, there is little understanding of what constitutes a ‘quality’ encounter through virtual care for both patients and providers. Continued use and integration of virtual care into standard practice, in part, depends on its impact on the quality of care and the experiences of patients, caregivers and healthcare providers.
Quality indicators are standardised, evidence-based measures that can be used to track and compare health outcomes and performance over time and across different organisations.7 8 Measurement of these indicators can identify gaps in care delivery and inform quality improvement (QI) efforts within an organisation, across health systems, and across geographical boundaries. However, traditional indicators to assess quality in healthcare primarily focus on in-person care and may have limited applicability to care delivered virtually. While some literature on virtual care indicators has been published, most studies focus solely on a specific clinical area (eg, obstetrics, cardiology) or a few domains of interest (eg, acceptability, satisfaction), limiting the scope and applicability for healthcare organisations to evaluate what constitutes ‘quality’ in virtual care.9–11
This scoping review was conducted with the aim to characterise existing quality indicators used to evaluate modalities of virtual care and categorise the indicators across the Quintuple Aim framework and National Academy of Medicine (NAM) quality framework (safety, effectiveness, patient-centredness, timeliness, efficiency and equity). Sustainability was added to capture the potential for continued use of virtual care. Our review focused on ambulatory patients as virtual care has a considerable role in access to care. Hospitalised in-patients have unique characteristics with higher acuity rendering a need for in-person care and therefore were not included in this review.
This work will provide the foundation for the identification and categorisation of quality indicators that can inform clinicians, healthcare managers and other decision-makers on how best to monitor the quality of virtual care, identify performance gaps, and target areas for future improvement efforts.
Methods
The scoping review methodology used in the study was guided by the Joanna Briggs Institute (JBI) Manual for Evidence Synthesis.12 Reporting of methods and results was developed in accordance with the Preferred Reporting Items for Systematic review and Meta-Analysis (PRISMA) extension statement for scoping reviews (see online supplemental file 1 for the PRISMA reporting checklist and see online supplemental file 2 for inclusions/exclusion protocols).13
Supplemental material
Supplemental material
Information sources and literature search
A comprehensive literature search was developed in consultation with an academic librarian that included published academic and grey literature sources, as well as hand searches of relevant journals. Initial literature searches were conducted in June 2021 and were updated in August 2022. Databases searched included MEDLINE, Embase, PsycInfo, The Cochrane Library and the JBI Evidence-based practice database (online supplemental file 3 – table 1). Keyword searches were also carried out in selected websites, Google Scholar, medRxiv and Open Grey to identify grey literature including policy documents, organisational strategies and unpublished academic literature (online supplemental file 3 – table 2). Specialty journals focused on virtual/digital care including Journal of Medical Internet Research, Lancet Digital Health and Nature Digital Medicine, were hand searched from 2015 to June 2022.
Supplemental material
Eligibility criteria
The eligibility criteria were structured based on the ‘Population, Context, Concept’ (PCC) research framework12 as follows:
Population: adults (over 18 years of age) receiving ambulatory/outpatient care through healthcare organisations for chronic or acute/subacute conditions.
Context: participation in virtual care, is defined as ‘any interaction between patients and/or members of their circle of care, occurring remotely, using any forms of communication or information technologies’.1
Concept: within the Quintuple Aim framework (provider experience, patient experience, per capita cost, population health and health equity),14 indicators are based on the NAM’s quality framework (safety, effectiveness, patient-centredness, timeliness, efficiency and equity) and sustainability.15
All modalities of virtual care were included such as video conferencing, remote monitoring and patient portals. Included papers must have addressed multiple domains within the Quintuple Aim, or a domain with the Quintuple Aim and NAM quality framework. Studies that focused on only one domain in a specific population were excluded. Disease-specific publications were only included if they focused on two or more quality domains. Due to changes in technology and the rapid increase in virtual care, only studies published in English from 2015 to 2022 were included.
Study selection process
The literature search results were uploaded into the Covidence review manager and de-duplicated prior to screening.16 Screening questions based on the eligibility criteria (online supplemental file 2 – figure 1) were developed for both title/abstract and full-text screening and pilot-tested with rounds of randomly selected citations until satisfactory agreement (>75%) was reached between reviewers. Double screening of title/abstracts was conducted (by CL, PR, VK and AS), followed by full-text screening by a single reviewer for a streamlined approach. Conflicts were resolved through group discussion with the reviewers and confirmed by (GM). After full-text screening, all excluded citations or articles were re-reviewed by an independent second reviewer (PR) to ensure accuracy. A similar process was followed for the 2022 update with SP, PR and AS, as reviewers.
Data collection and management
Due to the large heterogeneity across the types of papers, and the lack of standardisation in reporting of quality indicators and domains, the team opted for a rigorous approach with two reviewers (two of CL, PR, VK, SP and AS) independently extracting each article. Any discrepancies in extracted data were reviewed and resolved by an independent third reviewer. A standardised data extraction form was developed to summarise and record the reviewed studies within the Covidence review manager. Extraction was piloted in two rounds using three studies per round, and the extraction form was revised accordingly. Data items included publication characteristics (year of publication, country, study design, funding source and sample size), intervention details (care setting, virtual care modality, primary condition/health concern) and details on reported quality indicators (definition, measurement/scoring tools, etc).
Data synthesis
Individual indicators were used to demonstrate the connections between virtual care modalities and the Quintuple Aim14; patient experience and provider experience were further subcoded into the NAM quality domains15 along with sustainability. These connections were visualised through an alluvial (also known as Sankey) diagram. Although these individual indicators will be further refined through a future Delphi consensus process to narrow a subset of indicators across quality domains, the proportions provide valuable information regarding the types of quality domains and their corresponding modalities.
Risk of bias
As this was a scoping review intended to capture a collection of quality indicator items, rather than evaluate their effectiveness or appropriateness, risk of bias was not assessed based on current published guidelines.12
Patient and public involvement statement
People with lived experience were involved in the conception and development of the scoping review as well as the presentation and dissemination of results through the Patient Experience Advisors programme and the Community Liaison Advisory Council at Women’s College Hospital. We intend to solicit further patient participation as well as general public involvement in future work developing a balanced scorecard based on the literature identified in this review. Working with patients helped to ensure quality indicators that were extracting from the literature were relevant to patient concerns and provided guidance on the mapping of indicators to the Quintuple Aim and NAM quality framework. Patient advisors also helped to identify gaps, including the lack of focus on equity, and will further input to addressing these gaps through the development of the scorecard.
Results
The literature search resulted in a total of 18 395 citations from databases and grey literature searching that were imported into Covidence; no additional citations were identified through hand-searching journals. A total of 4891 duplicate citations were removed leaving 13 504 citations for title and abstract screening. After title and abstract screening, 814 full-text articles were identified for retrieval and 183 reports, mostly conference abstracts and commentaries, were further excluded. After assessing the reports for eligibility, 66 articles were included for data extraction (figure 1).
Preferred Reporting Items for Systematic review and Meta-Analysis study flow diagram.
Characteristics of included studies
The included studies comprised 45 primary research studies (eg, observational studies) and 21 secondary research articles (eg, systematic reviews). table 1 summarises the main characteristics of the included articles. The most frequent study designs were observational studies (n=32), followed by systematic reviews (n=11), qualitative studies (n=5), and interventional trials (n=5). The majority of included studies were published between 2018 and 2020 (n=25) in North America (n=29) or Europe (n=22). Funding sources were mainly from public institutions (n=30), but a large proportion of studies also reported no external funding for their study (n=24).
Characteristics of included studies (n=66)
Interventions examined in the included studies covered a range of virtual care modalities (table 2) including video and audio visits with clinicians (n=44), remote monitoring programmes (n=13), mobile applications (n=12), self-monitoring and wearable devices (n=9), synchronous or asynchronous messaging with healthcare providers (n=13), or patient portals (n=10). Care settings (table 2) ranged from ambulatory and outpatient clinics (n=41) to primary care (n=13), and home and community care settings (n=8). Patient populations (table 2) included individuals with chronic noncommunicable conditions (n=19), cancer (n=7), and mental health conditions (n=5).
Summary of intervention and setting characteristics (n=66)
Connections between virtual care modalities within the Quintuple Aim framework and NAM quality domains are visualised in figure 2. Within the Quintuple Aim, the most reported category related to patient experience (n=200 indicators), followed by provider experience (n=52), population health outcomes (n=47), health system costs (n=22), and equity (n=7). Virtual visits by phone and/or video (n=96) were the most common modality of virtual care reported, and were strongly connected to patient experience, provider experience, and population health outcomes (figure 2). Virtual visits (video only) (n=60) and remote monitoring (n=59) were the next most common, also connecting strongly to patient experience, provider experience, and population health outcomes.
Alluvial diagram connecting virtual care modalities to the Quintuple Aim and quality domains. Through the middle are the Quintuple Aim domains, with curved bars representing a connection to the virtual care modality (left) and NAM quality domains (right). For example, the 59 mentions of remote monitoring (left) were mostly connected with patient experience, followed by population health outcomes, then provider experience. The thick vertical bars show the frequency compared with the other domains: virtual visits were the most common modality; patient experience was the most common Quintuple Aim domain; and patient-centredness was the most common NAM quality domain.
Within the NAM quality domains, the most common sub-coded domains included patient-centredness (n=66), effectiveness (n=64), sustainability (n=36), and efficiency of care (n=36) (figure 2). Details regarding the frequency of indicators for NAM domains and patient experience and provider experience are provided in online supplemental file 4), with examples of indicators and their stem groupings in table 3.
Supplemental material
Examples of indicator-stem groupings and indicators within the Quintuple Aim, specific to virtual care.
Specific instruments measuring quality
Throughout data extraction, several existing tools were mentioned for measuring the quality of care. For patient experience, tools included Telemedicine/Telehealth usability questionnaires (n=18 studies); Telemedicine Satisfaction and Usefulness Questionnaire (n=3); and Service User Technology Acceptability Questionnaire (n=1). For provider experience the Telemedicine/Telehealth Usability Questionnaire (n=2) was used, while others only mention 5- and 10-point Likert scales with unique or custom question prompts. No specific measurement tools were mentioned for population health outcomes, costs, or health equity.
Discussion
This scoping review on quality measures for ambulatory virtual care found that within the Quintuple Aim, the most reported category assessing the quality of virtual care was related to patient experience, followed by provider experience, and population health outcomes. There was limited mention of health system costs, patient costs, or equity. The most frequently mentioned virtual care modality was virtual visits in video and/or audio, followed by video visits only, remote monitoring programmes, mobile applications, and self-monitoring tools. Within the NAM domains of quality for patient and provider experience, effectiveness was most common, followed by patient-centredness and efficiency, with limited mention of sustainability.
Best practice guidelines for virtual care are required to realise the integration of virtual care across health systems.17 18 This review provides the foundation for the identification and refinement of quality indicators in virtual care and highlights current gaps in the assessment of virtual care performance indicators. Results indicate that further development of evaluation methods specifically analysing the equitable and cost-efficient deployment of virtual care is needed. Measurement of quality of virtual care will support understanding performance gaps, and targets for future quality improvement efforts and benchmarking efforts across organisations.
Limited attention on equity, Sustainability, and cost
It was noted that the majority of quality indicators were concentrated in specific domains such as patient experience and/or satisfaction, or provider experience and/or satisfaction, while the more complex indicators, such as equity, cost, and sustainability, were not as widely reported. The limited attention on equity is consistent with published literature, such as a scoping review of reviews which highlights the lack of focus on equity assessments in virtual care deployments.19 Equity may have been more challenging to measure and integrate into programme evaluation and may have been considered outside the scope of QI projects or time-sensitive deployments (such as during the COVID-19 pandemic).20 This lack of consideration of equity is concerning, as virtual care has the potential to improve access to care for some typically underserved populations and worsen access for others,21 making it an essential consideration in understanding the quality of care.
Our review also noted limited attention in evaluations around the sustainability of virtual care delivery. Sustainability may be considered from multiple perspectives, including if the patient plans to continue to use the virtual options, or if the organisation has the means (infrastructure, IT support, etc) to continue to deliver virtual care after the COVID-19 pandemic restrictions lessened. Further indicators are needed to monitor the continued need for and use of virtual care to assess if it remains a preferred and feasible option, especially within hybrid models of care delivery which includes both virtual and in-person options. Considering sustainability is also important when taking a Learning Health Systems (LHS) approach across healthcare organisations and systems where the focus is on continuous learning and iterative advancement.22 Sustainability needs to be a key part of this systems-based approach, and thus quality indicators of virtual care that consider sustainability are needed.
Indicators within the Quintuple Aim domain of cost were also rarely reported in the literature, for either patient costs or health system costs. Like equity, this limited inclusion may partly be due to the fact that cost can be difficult to measure and quantify in rapid virtual care deployments. This finding is consistent with a 2020 scoping review that analysed the cost-effectiveness of telehealth to the healthcare system, which also highlighted this gap.23 Cost needs to be considered as part of the evaluation of the quality of virtual care, including in connection with the sustainability of the service and the cost implications for patients, which also ties back to the topic of health equity and access to virtual care.
Evidence synthesis of virtual care
There has been a considerable amount of work published about virtual care, and extensive synthesis of that work. About a third of the literature included in this scoping review were reviews themselves, however, these reviews usually were limited and focused on a specific service or modality of virtual care such as patient portals,24 telehealth/phone visits25 26 and video consultations.27 28 Few reviews assessed approaches or models for evaluating virtual care, or tools that specifically measured quality across multiple domains as identified by the Quintuple Aim and NAM frameworks. To the authors’ knowledge, this is the first review that specifically evaluates quality in virtual care with a comprehensive approach to defining quality indicators and assessing their use against established quality frameworks.
A report by the Canadian Agency for Drugs and Technology in Health (CADTH) which studied evaluations of virtual care was consistent with our findings.29 The CADTH report stated that evaluations should focus on ensuring the scope of the virtual care visits was appropriate for the care being administered, be conducted systematically, and that cost-analysis or cost-benefit evaluations be conducted with a specific viewpoint in mind – such as a patient or provider.29 The findings of this scoping review support these assertions from a quality perspective. A quality virtual care programme is clear in its scope, has dedicated resources for ongoing evaluations, and specifically investigates the impact of the programme from the patient perspective, provider perspective, or both.
Leveraging review results: next steps
This review provides the foundational knowledge base of existing quality indicators of virtual care across domains of quality and organised within established quality frameworks. This knowledge base will be further refined and distilled through a modified Delphi approach with key knowledge users and decision-makers, including patients, healthcare professionals, virtual care experts, and policymakers, to help add indicators to address gaps, and rate and rank existing indicators based on feasibility, relevance, and utility. A subset of high-ranked indicators across quality domains will then be translated into a balanced scorecard for evaluating the quality of virtual care.
An understanding of how to define quality in virtual care and measure performance is key to the development and growth of a potential LHS, which aims to enable cycles of continuous learning and improvement to be routinised and embedded across the healthcare system.30 An LHS is the combination of a health system and research system that, at all levels, is anchored on patient’s needs, perspectives, and aspirations; driven by timely data and evidence; supported by appropriate decision supports, aligned governance, financial and care delivery arrangements; and enabled with a culture of, and competencies for, rapid learning and improvement.31 Further, it is a system in which science, informatics, incentives, and culture are aligned for continuous improvement and innovation, and thus our domains of quality for virtual care can be integrated into this approach. Ideally, best practices, including best practices in virtual care, are seamlessly embedded in the delivery process, patients and families are active participants in all elements, and new knowledge capture is an integral by-product of the delivery experience.30 Ultimately, the deployment of virtual care should follow an LHS approach, with consistent revision and adjustment of virtual care initiatives to reflect dynamic contexts and adapt based on new evidence. Mapping the current knowledge base aims to provide organisations with quick references for improving and iterating their virtual care programme. Working towards true integration of virtual care into an LHS is subject to further research and implementation.
Strengths and limitations
This scoping review used a wide range of published and grey sources, covered a vast breadth of literature, and applied rigorous methods for data extraction and coding. The body of literature included in this review also represents the current state of publications on virtual care, as the search was updated as the project progressed. However, our search strategy was limited to English only, and relying on the past 7 years meant that foundational work on the implementation of virtual care may have been excluded.
A key strength of this work was our categorisation of existing indicators across established quality frameworks (NAM and sustainability, Quintuple Aim) to identify which quality domains of virtual care are present and lacking within the current literature. However, there is considerable overlap between the NAM domains for categorisation, and extensive differences in the way indicators were explained in the published studies, typically with very little detail. Indicators could overlap and routinely fit within more than one quality domain, and the lack of detail provided about each indicator meant that some assumptions were made, leading to subjectivity with current results. For example, patient-centred care was used very differently across studies, with some having it synonymous with patient satisfaction, while others acknowledged the multiple factors and complexity in the delivery of patient-centred care. To address this limitation we dual-coded all indicators, resolved discrepancy with a third reviewer and will be working with interprofessional decision-makers and persons with lived experience as we develop the balanced scorecard. The lack of reporting on how indicators were selected when measuring quality of care, the lack of detail provided and the lack of reported knowledge of user (patient, provider, management, etc) involvement, further highlighted the need for a consensus-based approach to finalise a concise list of virtual care quality indicators across all quality domains.
Conclusion
To ensure delivery of high-quality virtual care, it is imperative to understand what is ‘quality’ in virtual care. By identifying and mapping indicators of the quality of virtual care to the Quintuple Aim and the NAM framework, we aim to deepen our understanding of these indicators and identify gaps. The most reported category assessing the quality of virtual care was related to patient experience, followed by provider experience, and population health outcomes. There was limited mention of health system costs, patient costs, or equity, which limits our understanding of the quality of virtual care initiatives, these gaps highlight the need for the development of new indicators, and a consensus-based process with knowledge users to create a prioritised list. Future work will add and further refine the quality indicators through a modified Delphi approach, with the aim of providing decision-makers with a balanced scorecard for the implementation of quality virtual care initiatives moving forward.
Data availability statement
No data are available.
Ethics statements
Patient consent for publication
Ethics approval
Not applicable.
Acknowledgments
We would like to thank Vanessa Kishimoto for assistance with title and abstract screening and Becky Skidmore for assistance with developing the literature search.
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
SP and CL contributed equally.
Contributors Conceptualisation: GM. Formal Analysis: SP, CL, PR. Funding Acquisition: GM. Investigation: SP, CL, OM, PR, GM, AS. Writing—Original Draft Preparation: SP, CL. Writing—Reviewing and Editing: CL, PR, AS, OM, GM, OB, EB. Guarantor: GM.
Funding This study was supported by a grant from the Innovation Fund of the Alternative Funding Plan for the Academic Health Sciences Centres of Ontario. Funding grant number: WCH-22-004.
Competing interests None declared.
Patient and public involvement Patients and/or the public were involved in the design, or conduct, or reporting, or dissemination plans of this research. Refer to the Methods section for further details.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.