You are here

Article Text

Article menu

Download PDFPDF

Surgery
Original research
An international consensus panel on the potential value of Digital Surgery
  1. Jamie Erskine1,
  2. Payam Abrishami2,
  3. Jean-Christophe Bernhard3,
  4. Richard Charter4,5,
  5. Richard Culbertson6,
  6. Jo Carol Hiatt4,
  7. Ataru Igarashi7,
  8. Gretchen Purcell Jackson8,9,
  9. Matthew Lien8,
  10. Guy Maddern10,
  11. Joseph Soon Yau Ng11,
  12. Anita Patel12,13,
  13. Koon Ho Rha14,
  14. Prasanna Sooriakumaran15,16,
  15. Scott Tackett8,
  16. Giuseppe Turchetti17,
  17. Anastasia Chalkidou18
  1. 1Market Access, Alira Health, Boston, Massachusetts, USA
  2. 2Erasmus School of Health Policy and Management, National Health Care Institute, Rotterdam, The Netherlands
  3. 3University Hospital Centre Bordeaux, Bordeaux, France
  4. 4Health Technology Assessment International, Edmonton, Alberta, Canada
  5. 5CHLOE Healthcare Advisory Group, London, UK
  6. 6Louisiana State University Health Sciences Center, New Orleans, Louisiana, USA
  7. 7The University of Tokyo, Tokyo, Japan
  8. 8Intuitive Surgical Inc, Sunnyvale, California, USA
  9. 9American Medical Informatics Association, Bethesda, Maryland, USA
  10. 10Surgery, The Queen Elizabeth Hospital, University of Adelaide, Woodville, Adelaide, Australia
  11. 11National University Cancer Institute, Singapore
  12. 12Anita Patel Health Economics Consulting Ltd, London, UK
  13. 13Wolfson Institute of Population Health, Queen Mary University of London, London, UK
  14. 14Yonsei University Medical Center, Seodaemun-gu, Seoul, Republic of Korea
  15. 15University College London, London, UK
  16. 16Cleveland Clinic London, London, UK
  17. 17Institute of Management, Scuola Superiore Sant'Anna, Pisa, Toscana, Italy
  18. 18NICE, London, UK
  1. Correspondence to Jamie Erskine; jamie.erskine{at}alirahealth.com

Abstract

Objectives The use of digital technology in surgery is increasing rapidly, with a wide array of new applications from presurgical planning to postsurgical performance assessment. Understanding the clinical and economic value of these technologies is vital for making appropriate health policy and purchasing decisions. We explore the potential value of digital technologies in surgery and produce expert consensus on how to assess this value.

Design A modified Delphi and consensus conference approach was adopted. Delphi rounds were used to generate priority topics and consensus statements for discussion.

Setting and participants An international panel of 14 experts was assembled, representing relevant stakeholder groups: clinicians, health economists, health technology assessment experts, policy-makers and industry.

Primary and secondary outcome measures A scoping questionnaire was used to generate research questions to be answered. A second questionnaire was used to rate the importance of these research questions. A final questionnaire was used to generate statements for discussion during three consensus conferences. After discussion, the panel voted on their level of agreement from 1 to 9; where 1=strongly disagree and 9=strongly agree. Consensus was defined as a mean level of agreement of >7.

Results Four priority topics were identified: (1) how data are used in digital surgery, (2) the existing evidence base for digital surgical technologies, (3) how digital technologies may assist surgical training and education and (4) methods for the assessment of these technologies. Seven consensus statements were generated and refined, with the final level of consensus ranging from 7.1 to 8.6.

Conclusion Potential benefits of digital technologies in surgery include reducing unwarranted variation in surgical practice, increasing access to surgery and reducing health inequalities. Assessments to consider the value of the entire surgical ecosystem holistically are critical, especially as many digital technologies are likely to interact simultaneously in the operating theatre.

  • surgery
  • health economics
  • health informatics
  • quality in health care
  • ehealth

Data availability statement

All data relevant to the study are included in the article or uploaded as online supplemental information. All data are available within the manuscript.

http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

https://doi.org/10.1136/bmjopen-2023-082875

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    STRENGTHS AND LIMITATIONS OF THIS STUDY

    • Using a combination of a modified Delphi process and a series of consensus conferences, this study generates expert consensus on the value of digital surgical technologies (DSTs).

    • This study identifies specific considerations for health technology assessments (HTAs) of DSTs.

    • Although the expert panel assembled for this study represents a range of stakeholders and geographies, the results are subjective and may not represent all relevant viewpoints.

    • The study did not have a designated patient representative; however, it did include consumer health informatics expertise and members who have been surgical patients.

    • This study’s aim is not to provide methodological guidance for completing assessments of DSTs but rather to advise HTA bodies who may be developing frameworks for digital technology to consider the specific nuances and complexities of digital technologies in surgery.

    Introduction

    Digital technologies are being used increasingly in healthcare systems globally, accelerated by the COVID-19 pandemic with the global telehealth market reaching more than US$80 billion in 2021 and expected to reach over US$200 billion in 2025.1 These technologies, known as digital health technologies (DHTs), are extremely diverse. The Food and Drug Administration includes mobile health (mHealth), health information technologies, wearable devices, telehealth, telemedicine and personalised medicine in its definition of digital health2 while the National Institute for Health and Care Excellence simply notes that DHTs ‘comprise a wide range of products used in the health and care system including apps, software and online platforms’.3 Given that DHTs have such a broad range of functionalities, use cases and benefits, understanding and evaluating them are highly complex tasks.

    DHTs differ from other health technologies in several ways. First, many DHTs are frequently updated. Artificial intelligence (AI)-based technologies are changing perpetually as algorithms learn from new data. This rapid pace of development makes evaluating the clinical and economic benefits of these technologies challenging. Further, the evidence supporting DHTs may not be as robust as other health technologies such as medical devices or pharmaceuticals.4 5 DHTs are often highly context-dependent, particularly within surgery or other hospital settings, which could mean that standard randomised trial designs are less applicable in some cases compared with other forms of evidence, such as real-world evidence (RWE).6 Large-scale Randomised Controlled Trials (RCTs), for example, are often performed by clinical trial networks or contract research organisations that operate outside of normal clinical practice. This may be less applicable for some digital health technologies (particularly those that implement AI) as data collected from routine clinical practice are often required for the operation of these technologies. Further, like medical devices,7 there may also be an operator learning curve related to digital technologies.8 To further add to the complexity, DHTs are often used simultaneously or integrated with another technology such as a medical device and can be used across a variety of different indications. These complexities, in addition to the huge range of use cases, level of autonomy and potential risk, make assessing digital technologies for safety, efficacy and cost-effectiveness a uniquely difficult proposition.6

    In surgery, digital technologies are rapidly being developed and adopted, from preoperative planning and intraoperative guidance to postoperative performance assessment.9 Advancements in training and education,10 virtual reality (VR),11 machine learning,9 and telehealth12 are being implemented in surgical practice, either as standalone solutions or alongside other DHTs and devices. This is increasingly true in robot-assisted surgery (RAS), as advancements in digital capabilities are developing in tandem with various robotic platforms.13 14

    Lam et al,15 in a Delphi exercise, aimed to define ‘digital surgery’, agreeing on ‘the use of technology for the enhancement of preoperative planning, surgical performance, therapeutic support or training, to improve outcomes and reduce harm’. The study also reported that there were no clearly defined reimbursement or business models for these technologies. Furthermore, adoption barriers may arise due to difficulties in demonstrating safety and clinical benefits. The authors recommended future research into developing a framework for the introduction and evaluation of surgical AI and establishing a business model with the industry.

    There have been several frameworks for evaluating digital technologies published by various health technology assessment (HTA) bodies in recent years, many of which are still evolving and being refined. A review by San Miguel et al16 at the Belgian Health Care Knowledge Centre reviewed six existing European frameworks (table 1) for evaluating digital technologies as part of the further development of their own procedures.

    View this table:
    Table 1

    Existing frameworks developed by HTA agencies for assessing digital health technologies in Europe, adapted from San Miguel et al16

    The frameworks listed vary in scope considerably. The DiGA procedure in Germany, for example, was designed specifically for health apps.17 The Finnish Digi-HTA framework is slightly broader, including specific considerations for robotics and AI,18 although no assessments have been completed on such technologies used in surgery to date.19 Outside of Europe, guidelines have also been developed in countries like South Korea; however, these guidelines only cover AI for medical imaging and three-dimensional printing and so are limited in scope.20

    Despite the range of efforts to develop assessment frameworks for DHTs, no consistent standards have yet been agreed on, partially due to the diversity in technologies, in the setting of use and in reimbursement models.21 Payers and policy-makers are still challenged when quantifying the clinical and economic utility of these technologies. Unlike health apps, DHTs used in surgery may be high-cost technologies or are used in conjunction with other high-cost surgical devices, such as robotic platforms. Payment and reimbursement models are likely to differ significantly. Applications, for example, may use a subscription model or a population-based payment model. Some digital technologies in surgery are more likely to be purchased as a separate product in the operating theatre but many are integrated into a robotic platform or even into a digital operating theatre.14 Calculating the economic impact of such technologies is potentially very complex.

    In this article, we focus on digital surgical technologies (DSTs), referred to from here onwards as DSTs. Elsewhere, we have developed guidance for the assessment of RAS platforms.22 Existing frameworks for evaluating digital technologies in healthcare, have so far inadequately included specific considerations for digital technologies used in surgery.19 While Lam et al include robotics as part of their definition of digital surgery,15 we consider that DSTs are a broader category of technologies that may be integral to robotic platforms, used alongside these platforms or used standalone. Notwithstanding, they offer distinctive additional potentials, besides the clinical utility of the platforms themselves. We argue that this additional digital capacity in robotic surgery merits specific attention for developing frameworks for the value assessment of DSTs.

    In this article, we describe a modified Delphi study that aims to develop consensus on the value potentials of digital technologies in surgery and highlight important considerations and challenges in assessing these value perspectives.

    Methods

    A modified Delphi approach was used in conjunction with a consensus conference approach.23 An international expert panel was assembled, including 14 panellists from 9 countries and 4 continents. The panel included 11 of the members of a previous international expert panel put together to discuss HTAs of RAS22 but also included other members recommended by the existing panel. A total of four new members were chosen for their knowledge and expertise in digital technologies in surgery (table 2). The panel members represent a wide range of relevant stakeholders, including surgeons, health economists, HTA practitioners and methodologists, policy-makers and industry representatives. The surgeons on the panel all had significant experience in a range of soft-tissue surgeries completed both with and without robotic assistance. The health economists on the panel had significant experience in modelling complex health interventions, including robotic surgery platforms. HTA practitioners on the panel had experience in assessing health technologies including medical devices and robotic surgery platforms at national and regional HTA bodies. Methodologists had previous experience in developing research and assessment methods for health technologies. Policy-makers had experience in making national and regional-level decisions about the implementation of health policy and adoption of health technologies including medical devices. Many of the panel also had specific experience in using digital technologies in surgery or performing HTAs of digital technologies.

    View this table:
    Table 2

    Panellists’ details, listed in alphabetical order

    Three questionnaires were used to develop priority topics for discussion between January and March 2023 (figure 1).

    Figure 1
    Figure 1

    Structure of modified Delphi process.

    An initial scoping questionnaire, developed in Google Forms (Google, California, USA) and sent as a link by email, invited panellists to rate the importance of two questions (see topics 1 and 2 in table 3) identified as potential research priorities during the previous panel discussions22 and to suggest any other questions or issues that the group should address. A further two questions were added at this stage (see topics 3 and 4 in table 3), totalling four. The second questionnaire, also created and distributed in the same format, asked panellists to rate the importance of these four questions from 1 to 9, where 1 indicated ‘not important’ and 9 indicated ‘critical’. The group was also asked to rate the same questions from 1 to 9 based on their perceived ability to contribute to the discussion of these questions, where 1 indicated ‘no knowledge/expertise’ and 9 indicated ‘very high knowledge and expertise’. These rankings are subjective and merely descriptive. The suggested questions were considered priority topics and are summarised in table 3, ranked by their perceived level of importance.

    View this table:
    Table 3

    Proposed priority topics and their perceived importance and panel ability to answer

    It is notable that the panel voted the topic of training and education (topic 4 in table 3) as the topic with the lowest mean level of panel knowledge. To ensure that the panel was well informed during the discussions, the members who were more familiar with this topic provided details of their experiences with using digital technologies in surgical training and education.

    Following the generation of the priority topics shown above, the panel members were asked to provide the three most important issues to tackle within these topics when considering the value of digital surgery.

    These issues were discussed during the first of three consensus conferences, in which eight initial consensus statements were drafted. The consensus statements were further refined during the second consensus conference and finalised during the third conference, with a total of seven reaching consensus.

    Patient and public involvement

    Although patients were not directly involved in the design of the research, the impetus for the research project was informed by the previous work of the panel, which considered how robotic surgery can affect patient outcomes including quality of life and patient satisfaction.

    Results

    The key issues to tackle, as proposed by the panel, were categorised into four topics for discussion: (1) how data are used in surgery, (2) the existing evidence base for DSTs, (3) how digital technologies may be used in surgical training and education and (4) methods for the assessment of these technologies. Given the often-tandem development and integration of DSTs with robotic platforms, many of the topics discussed were in the context of DSTs used in RAS; however, the panel felt the consensus statements would still be broadly applicable to all DSTs.

    Eight consensus statements, two per topic, were developed during the first consensus conference (see table 4). These statements were discussed during the second interactive consensus conference. The panel convened three times (between March and June 2023) for in-depth discussion of the topics. Each of these meetings was ‘hybrid’, with some panel members meeting in person where feasible and others joining through teleconferencing software (Zoom, California, USA). The original list of eight consensus statements was reduced to seven final statements that were agreed on and refined to form the conclusion of the final consensus conference (table 4).

    View this table:
    Table 4

    Priority topics for discussion and draft consensus statements

    Discussion

    This modified Delphi consensus study used a series of questionnaires to identify potential value perspectives of DSTs and potential challenges in assessing this value. An expert panel discussed these challenges and developed a series of consensus statements. The panel agreed that traditional evaluation frameworks for drugs, devices and even new digital technologies may not be applicable for certain types of DSTs used as part of surgical practice. While many DSTs employ applications and software, they are often not used as stand-alone technologies. Rather, they exist as part of an array of technologies used in the operating theatre and wider hospital setting. This rich interaction is particularly true for RAS systems, in which the robotic platform includes a rapidly developing portfolio of digital solutions. Advanced computer imaging for preoperative planning, intraoperative VR assistance, simulation-based surgical training, real-time decision support, data recording, postoperative analytics, performance assessments and AI-based clinical decision support are examples of DSTs integrated with robotic platforms.14 24

    DSTs aim to provide improved surgical outcomes or processes, notably when integrated into robotic surgery platforms.25 26 Additionally, many share complexities such as learning curve issues and multi-indication applications.22 Our panel considered that DSTs also have additional value and intricacies that are unique to specific stakeholders or processes, particularly in terms of training and education, workflow, and efficiency, and in generating RWE (see consensus statements 2 and 7 in table 5). Given that DSTs are often part of a large ecosystem of technologies in the surgical setting, this highlights the need to consider their value holistically. These ecosystems should be assessed according to the specificities of the setting of use and the specific perspectives of the stakeholders the evaluation is relevant to. The World Bank Group framework for the economic assessment of digital health technologies suggested a value aggregation function—where multiple value attributes are weighted based on the preferences of chosen stakeholder groups—as one such method.21 Despite the range of potential values that DSTs may bring, the evidence is still in its infancy and is often not growing at the same rapid pace as the technologies (see consensus statement 5). The panel noted that this lack of evidence represents an opportunity for HTAs and other healthcare decision-makers to provide guidance for evidence generation. This guidance may ensure that appropriate study designs are recommended according to the type of DST, that studies are performed efficiently, and that they capture relevant outcomes.

    View this table:
    Table 5

    Final consensus statements

    The emerging evidence for DSTs has highlighted a wide range of potential use cases and value propositions. Our panel agreed that the organisational and social benefits of digitising surgery are of particular interest due to their novelty from the perspective of policy-makers and HTA bodies. Teleoperated surgery is one example where DSTs may allow for increased access to surgery due to advancements in 5G and telepresence technology, particularly for patients in rural areas.27–29 Such remote care, along with virtual consultations and patient apps or wearables, may have far-reaching sustainability benefits by avoiding the need for travel.30 Additionally, a reduction in unwarranted variation through improved training and performance assessments may address equity of care issues and provide further social benefits.31

    Digital aspects of RAS technology have the capability of delivering greater implementation of minimally invasive surgery than prior known minimally invasive surgical approaches. These enhancements include but are not limited to advanced imaging, simulation, remote proctoring, telepresence, intraoperative guidance, decision support, data analytics, improved standardisation of procedures and reduced variation of care. As such, digital RAS ecosystems have the potential to allow surgeons and care teams to more effectively treat, by providing greater quality, accessibility and availability of minimally invasive surgery to a greater number of patients. The panel discussed that this may have particular importance in regions or countries with less well-developed surgical programmes, such as in low-income and middle-income countries (LMICs). Telepresence further enables access to surgical expertise and knowledge irrespective of geographical location.

    Despite being the topic on which our panel perceived as having the least knowledge, RAS platforms have already caused a shift in surgical training and education, and digital technologies assessing performance may revolutionise surgical learning models. VR,11 simulation32 and objective performance indicators (OPIs)33 are increasingly being used in surgical training. Real-time advice and intervention from expert surgeons can be facilitated remotely.34 Our panel agreed that emerging technologies would not only reduce the time to proficiency for novice surgeons but would also allow for career-long continuous improvement for practising surgeons. The panel agreed emerging digital surgery technologies may also reduce the total time and cost needed to train novice surgeons, which is often a cost borne by institutions. HTAs should look to describe the utility of these DSTs according to their specific perspectives and context, for example, as part of robotic platforms. They can then look to develop methods to quantify this cost impact on their institution or jurisdiction. As is stated in consensus point 2, new DSTs are likely to increase healthcare provider’s ability to train, retrain and retain surgeons. This may be particularly relevant for national health systems facing pressures related to ageing populations and surgeon shortages.22 Panel members reported difficulties in their own practices in recruiting surgeons if robotic platforms were not available. Furthermore, a 2021 study found that 73.8% of surgical trainees valued greater access to robotic surgery training, 73.4% believed that robotic surgery was important for the future of their desired specialty and 77.2% believed it should be incorporated into formal surgical training.35

    A topic of particular interest to the panel was the use of OPIs to evaluate surgeon performance.36 OPIs are quantitative measurements, derived from kinematic and system events data that are automatically captured by (some robot-assisted) surgical systems.36 Automatically capturing objective measures of surgeon performance may allow for a scalable evaluation of certain surgical techniques that have not been possible in the past. This may represent an opportunity to improve or accelerate surgical training and allow for continuous improvement, even among intermediate and expert surgeons.37 OPIs may also provide insights related to operational workflow and efficiency, as well as be linked to postoperative outcomes.38 Early evidence has shown that these metrics may predict outcomes, such as early urinary continence recovery38 and length of stay, particularly when considering confounding patient factors, such as age and body mass index. Surgical platforms equipped to capture OPIs may improve outcomes by a measurable and predictable amount. The panel concurred that whether related to skill assessment and learning, workflow and efficiency, or postoperative outcomes, insights generated by OPIs can have a measurable impact clinically and economically (see consensus statement 4). As DSTs related to OPIs continue to be developed, the panel recommended that HTAs at a national and local level may consider them as proxy of value, or as surrogate outcomes, and try to quantify them in a harmonised way. The evidence base for these measures is growing and if strong links can be established, this may represent a paradigm shift in surgical training and practice. The panel would urge healthcare decision-makers to determine how, and to what extent, these technologies fit within their value assessments and what their implications are for future use and adoption. On the other hand, the panel also discussed how these technologies may also represent a risk to privacy. As Lam et al noted, large-scale recording of operating room data may increase the threat of litigation for surgical teams, many of whom may be reluctant to consent to data collection9 and patients may also have objections to their data being collected.

    The panel felt that increasing automation of data collection may have far-reaching consequences, not least in terms of the ability to generate RWE for measuring surgical outcomes and for healthcare decision-making (see consensus statement 6). RWE is consistently listed as a vital part of how we evaluate technologies.39 Traditionally used for postmarket surveillance, it is increasingly being recognised in regulatory approvals and HTA evaluations.39 A major barrier to RWE generation is the hands-on time required, often by clinical staff, to collect data. Automating and standardising this process may be of great value to the healthcare system by saving staff time and increasing the depth and quality of data available for decision-making and policy development. The panel also considered that all stakeholders in the surgical space must advocate for interoperability of data between different technologies and systems (see consensus statement 3). This may be a barrier to the generation of good-quality RWE. If substantial time and resources are required to ensure that one dataset is compatible with another, this defeats the purpose of automated collection. The panel also noted that it is vital to ensure that studies of DSTs (and particularly early phase studies of AI technologies which involve the training of algorithms) are conducted to high standards, following guidance from Standard Protocol Items: Recommendations for Interventional Trials and Consolidated Standards of Reporting Trials-AI on conducting and reporting trials.4 6 40 Patient safety and equity are paramount, and avoiding unexpected consequences that arise due to using unrepresentative populations should be prioritised.6 It should be noted that RCTs should still be considered the gold standard where they are feasible, and RWE should be considered as having additional value, rather than replacing traditional methods. In addition, all studies should be conducted to the highest possible standard and reported using the aforementioned transparency standards, as well as noting potential biases and limitations.

    Here, we advocate for the development of surgery-specific considerations in evaluation frameworks for digital technologies; however, this requires flexibility and adaptability to new innovations coming at a rapid pace. Existing frameworks provide a starting place, but continued methodological work may be needed to define appropriate processes for specific technologies. A strong horizon scanning protocol would go some way to ensuring that we are ready to evaluate incoming technologies (see consensus statement 1).

    Limitations

    This work aimed to gather expert insights into the evaluation of DSTs. A wide range of expertise from across the globe was gathered; however, the modified Delphi exercise only included the 14 panellists and could have been expanded to include a greater sample size. However, given the novelty of the topic and the requirement for specific expertise, a smaller group of 14 was considered sufficient, similar to the panel described in Erskine et al.22 To make up for the reduced number of responses for the Delphi exercise, the panel met three times to discuss the topics at length, as opposed to a single meeting in many standard Delphi approaches.

    The panel did not have representation from LMICs for this piece of work. This may bias the results towards high-income countries and it should be noted that there are specific benefits of DSTs in their potential to increase access to minimally invasive surgery in lower-income settings.41 A further limitation is the lack of patient representation. While the panel did not have a designated patient representative, it did include consumer health informatics expertise and members who have been surgical patients. In the future, the group may look to convene such a panel to specifically discuss the results of this work and the prior results of Erskine et al.22 In particular, patients’ opinions on the patient-related and social benefits of digital technologies would be valuable. The panel discussed topics that may be potential future considerations for further work, including surgical technologies’ ability to reduce health inequities and increase patient access in rural areas and LMICs, their effect on environmental sustainability, and the patient and public perspective. These value types may be considered broadly ‘societal value’ and could be a future research project for this panel.

    The previous work of this panel aimed to provide guidance for HTA bodies completing assessments of RAS platforms and concluded that digital technologies were an increasingly important consideration in surgery. Here, given that there are no prior assessments to our knowledge, and that the evidence base for these technologies is still limited at this time, the panel’s aim is not to provide methodological guidance for completing assessments of DSTs but rather to advise HTA bodies may be developing frameworks for digital technology to consider the specific nuances and complexities of digital technologies in surgery. While this is an initial step, it is recommended that HTA bodies consider the conclusions of this work when developing evaluation frameworks for DSTs. Most likely, such considerations may be made as an add-on to wider frameworks for DHTs.

    Conclusions

    Evaluating DSTs requires taking into account specific considerations of use within the surgical context that differ from other DHTs. Frameworks and methodologies developed for assessing DHTs should, therefore, consider the unique complexities of the high-stakes surgical environment and increasingly digitally enabled surgical ecosystem. It is unlikely that digital surgery is the only specialty with particular difficulties in assessing value. Digital ecosystems are arising in many areas of healthcare, such as in home care22 and personalised medicine, where digital technologies are radically changing the care models. New capacities are being built into the assessment frameworks to consider the benefits of DHTs in decreasing healthcare inequalities and lowering carbon emissions. As these ecosystems are established, it is vital to ensure that the individual technologies that the systems are composed of are evaluated holistically. In surgery, there are additional value propositions that need to be considered by HTAs, including the value of reducing unwarranted variations in performing surgical procedures, accelerating proficiency-based surgical training and making complex surgical care more accessible to patients in need. Further, OPIs may have substantial impacts on surgical education and training. Lastly, the enormous potential for automated data collection and evidence generation should not be underestimated. All stakeholders should advocate for data interoperability to fully recognise this value.

    Data availability statement

    All data relevant to the study are included in the article or uploaded as online supplemental information. All data are available within the manuscript.

    Ethics statements

    Patient consent for publication

    References

    1. ↵
      1. Negreiro M
      . n.d. The rise of digital health technologies during the pandemic. 2021 Members’ Research Service, European Parliament. PE 690.548 – April 2021. Available: https://www.europarl.europa.eu/RegData/etudes/BRIE/2021/690548/EPRS_BRI(2021)690548_EN.pdf
    2. ↵
      1. Center for Devices and Radiological Health
      . n.d. What is digital health? U.S. Food and Drug Administration. Available: https://www.fda.gov/medical-devices/digital-health-center-excellence/what-digital-health
    3. ↵
      1. NICE
      . Evidence standards framework for digital health technologies (ECD7), 2022. Available: https://www.nice.org.uk/corporate/ecd7
    4. ↵
      1. Liu X,
      2. Rivera SC,
      3. Moher D, et al
      . Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI Extension. BMJ 2020;370:m3164. doi:10.1136/bmj.m3164
    5. ↵
      1. Mumtaz H,
      2. Riaz MH,
      3. Wajid H, et al
      . Current challenges and potential solutions to the use of digital health technologies in evidence generation: a narrative review. Front Digit Health 2023;5:1203945. doi:10.3389/fdgth.2023.1203945
    6. ↵
      1. Park Y,
      2. Jackson GP,
      3. Foreman MA, et al
      . Evaluating artificial intelligence in medicine: phases of clinical research. JAMIA Open 2020;3:326–31. doi:10.1093/jamiaopen/ooaa033
      OpenUrl
    7. ↵
      1. Tarricone R,
      2. Boscolo PR,
      3. Armeni P
      . What type of clinical evidence is needed to assess medical devices? Eur Respir Rev 2016;25:259–65. doi:10.1183/16000617.0016-2016
      OpenUrlAbstract/FREE Full Text
    8. ↵
      1. Motta M,
      2. Groves E,
      3. Schneider A, et al
      . Designing self-tracking experiences: A qualitative study of the perceptions of barriers and facilitators to adopting digital health technology for automatic urine analysis at home. PLOS Digit Health 2023;2:e0000319. doi:10.1371/journal.pdig.0000319
    9. ↵
      1. Lam K,
      2. Chen J,
      3. Wang Z, et al
      . Machine learning for technical skill assessment in surgery: a systematic review. NPJ Digit Med 2022;5:24. doi:10.1038/s41746-022-00566-0
    10. ↵
      1. Paine H,
      2. Chand M
      . Role of digital resources in minimally invasive colorectal surgery training. Clin Colon Rectal Surg 2021;34:144–50. doi:10.1055/s-0040-1718686
      OpenUrl
    11. ↵
      1. Ntakakis G,
      2. Plomariti C,
      3. Frantzidis C, et al
      . Exploring the use of virtual reality in surgical education. World J Transplant 2023;13:36–43. doi:10.5500/wjt.v13.i2.36
      OpenUrl
    12. ↵
      1. McLean KA,
      2. Sgrò A,
      3. Brown LR, et al
      . Evaluation of remote digital postoperative wound monitoring in routine surgical practice. NPJ Digit Med 2023;6:85. doi:10.1038/s41746-023-00824-9
    13. ↵
      1. Singh TP,
      2. Zaman J,
      3. Cutler J
      . Robotic surgery: at the crossroads of a data explosion. World J Surg 2021;45:3484–92. doi:10.1007/s00268-021-06321-y
      OpenUrl
    14. ↵
      1. Pitruzzella B,
      2. Leahy P
      . The future of surgery. The Royal College of Surgeons, 2018. Available: https://futureofsurgery.rcseng.ac.uk/report-2018/
    15. ↵
      1. Lam K,
      2. Abràmoff MD,
      3. Balibrea JM, et al
      . A Delphi consensus statement for digital surgery. NPJ Digit Med 2022;5:100. doi:10.1038/s41746-022-00641-6
    16. ↵
      1. San Miguel L,
      2. Obyn C,
      3. Vinck I, et al
      . Evaluation of digital medical technologies. Health Technology Assessment (HTA) brussels: Belgian health care knowledge centre (KCE). KCE Reports 362s. D/2022/10.273/66. 2022. doi:10.57598/R362C
    17. ↵
      1. Federal Institute for Drugs and Medical Devices
      . The fast-track process for digital health applications (diga) according to section 139e sgb v: a guide for manufacturers, service providers and users. 2020. Available: https://www.bfarm.de/EN/Medical-devices/Tasks/DiGA-and-DiPA/Digital-Health-Applications/_node.html
    18. ↵
      1. Haverinen J,
      2. Keränen N,
      3. Falkenbach P, et al
      . Digi-HTA: Health technology assessment framework for digital healthcare services. FinJeHeW 2019;11. doi:10.23996/fjhw.82538
    19. ↵
      1. FINCCHTA
      . Digi-HTA Assessment Library. Oulu University Hospital, Available: https://oys.fi/fincchta/en/digi-hta-eng/arvioinnit-eng/
    20. ↵
      1. APACMED
      . Overview of Country Health Technology Assessment (HTA) Guidelines for Digital Health Technologies (DHTs), 2020. Available: https://apacmed.org/our-work/digital-health/health-technology-assessment-guidelines-for-digital-health-technologies
    21. ↵
      1. Wilkinson T,
      2. Wang M,
      3. Friedman J, et al
      . A framework for the economic evaluation of digital health interventions. Pol Res Work Pap 2023. doi:10.1596/1813-9450-10407
    22. ↵
      1. Erskine J,
      2. Abrishami P,
      3. Charter R, et al
      . Best practice considerations on the assessment of robotic assisted surgical systems: results from an international consensus expert panel. Int J Technol Assess Health Care 2023;39:e39. doi:10.1017/S0266462323000314
    23. ↵
      1. Murphy MK,
      2. Black NA,
      3. Lamping DL, et al
      . Consensus development methods, and their use in clinical guideline development. Health Technol Assess 1998;2:i–iv, .
      OpenUrlCrossRefPubMed
    24. ↵
      1. FOS:TEST Commission
      . Future of surgery: technology enhanced surgical training: report of the FOS:TEST commission. 2022. doi:10.1308/FOS2.2022
    25. ↵
      1. Michiels C,
      2. Khene Z-E,
      3. Prudhomme T, et al
      . MP49-04 3D-image guided robotic assisted partial nephrectomy: a multi-institutional propensity score matched analysis (UROCCR study 51). J Urol 2021;206. doi:10.1097/JU.0000000000002075.04
    26. ↵
      1. Shirk JD,
      2. Thiel DD,
      3. Wallen EM, et al
      . Effect of 3-dimensional virtual reality models for surgical planning of robotic-assisted partial nephrectomy on surgical outcomes: a randomized clinical trial. JAMA Netw Open 2019;2:e1911598. doi:10.1001/jamanetworkopen.2019.11598
    27. ↵
      1. Janetschek G,
      2. Bartsch G,
      3. Kavoussi LR, et al
      . Transcontinental interactive laparoscopic telesurgery between the United States and Europe. J Urol 1998;160:1413. doi:10.1097/00005392-199810000-00054
    28. ↵
      1. Bailo P,
      2. Gibelli F,
      3. Blandino A, et al
      . Telemedicine applications in the Era of COVID-19: telesurgery issues. Int J Environ Res Public Health 2021;19:323. doi:10.3390/ijerph19010323
    29. ↵
      1. Shahzad N,
      2. Chawla T,
      3. Gala T
      . Telesurgery prospects in delivering healthcare in remote areas. J Pak Med Assoc 2019;69(Suppl 1):S69–71.
      OpenUrl
    30. ↵
      1. Royal College of Surgeons of England
      . Sustainability in the operating theatre: a guide to good practice, 2022. Available: https://www.rcseng.ac.uk/standards-and-research/standards-and-guidance/good-practice-guides/sustainability-in-operating-theatre/
    31. ↵
    32. ↵
      1. Shah S,
      2. Aydin A,
      3. Fisher R, et al
      . Current status of simulation-based training tools in general surgery: A systematic review. Int J Surg Open 2022;38:100427. doi:10.1016/j.ijso.2021.100427
      OpenUrl
    33. ↵
      1. Hung AJ,
      2. Ma R,
      3. Cen S, et al
      . Surgeon automated performance metrics as predictors of early urinary continence recovery after robotic radical prostatectomy-a prospective bi-institutional study. Eur Urol Open Sci 2021;27:65–72. doi:10.1016/j.euros.2021.03.005
      OpenUrl
    34. ↵
      1. Din N,
      2. Chan CC,
      3. Cohen E, et al
      . Remote surgeon virtual presence: a novel telementoring method for live surgical training. Cornea 2022;41:385–9. doi:10.1097/ICO.0000000000002921
      OpenUrl
    35. ↵
      1. Fleming CA,
      2. Ali O,
      3. Clements JM, et al
      . Surgical trainee experience and opinion of robotic surgery in surgical training and vision for the future: a snapshot study of pan-specialty surgical trainees. J Robot Surg 2022;16:1073–82. doi:10.1007/s11701-021-01344-y
      OpenUrl
    36. ↵
      1. Lazar JF,
      2. Brown K,
      3. Yousaf S, et al
      . Objective performance indicators of cardiothoracic residents are associated with vascular injury during robotic-assisted lobectomy on porcine models. In Review [Preprint] 2022. doi:10.21203/rs.3.rs-1737899/v1
    37. ↵
      1. Hung AJ,
      2. Oh PJ,
      3. Chen J, et al
      . Experts vs super-experts: differences in automated performance metrics and clinical outcomes for robot-assisted radical prostatectomy. BJU Int 2019;123:861–8. doi:10.1111/bju.14599
      OpenUrl
    38. ↵
      1. Lazar JF,
      2. Jarc A,
      3. Oh D
      . Task-based objective performance indicators in robotic lobectomy offer a novel avenue for case assessments. JAMA Surg 2023;158:1103–4. doi:10.1001/jamasurg.2023.0363
      OpenUrl
    39. ↵
      1. NICE
      . NICE real-world evidence framework (ECD9), 2022. Available: https://www.nice.org.uk/corporate/ecd9/chapter/overview
    40. ↵
      1. Rivera SC,
      2. Liu X,
      3. Chan A-W, et al
      . Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI Extension. BMJ 2020;m3210. doi:10.1136/bmj.m3210
    41. ↵
      1. McCullough MC,
      2. Kulber L,
      3. Sammons P, et al
      . Google glass for remote surgical tele-proctoring in low- and middle-income countries: a feasibility study from Mozambique. Plast Reconstr Surg Glob Open 2018;6:e1999. doi:10.1097/GOX.0000000000001999

    Footnotes

    • Contributors JE is responsible for the overall content as the guarantor. JE supported the design of the research, supported the background research for the panel discussions, supported the facilitation of the panel meetings and drafted the first version of the manuscript. RCharter supported the facilitation of the panel meetings and reviewed the manuscript. ML supported the background research for the panel discussions and reviewed the manuscript. AC provided oversight for the design of the research, chaired the expert panel and reviewed the manuscript. PA, J-CB, RCulbertson, JCH, AI, GJ, GM, JSYN, AP, KHR, PS, ST and GT were the members of the panel, attended the panel meetings and reviewed the manuscript.

    • Funding Alira Health received consulting fees from Intuitive Surgical for coordinating the consensus panel meetings. The panel members were reimbursed only for their travel and accommodation costs related to the panel meetings.

    • Competing interests ST is an employee of Intuitive Surgical and as such he receives compensation and benefits as part of his employment agreement, as well as stock from Intuitive Surgical. GJ is employee of Intuitive Surgical and as such she receives compensation and benefits as part of her employment agreement, as well as stock from Intuitive Surgical. She also owns stock or stock options from IBM and Kyndryl and is President and Chair, of the Board of Directors for the American Medical Informatics Association. ML is an employee of Intuitive Surgical and as such he receives compensation and benefits as part of his employment agreement. JE is an employee of Alira Health. Alira Health received consulting fees from Intuitive Surgical for the coordination of the panel. RCulbertson and AC are the co-chairs of the HTAi Medical Device Interest Group. RCulbertson has received payment from the American College of Healthcare Executives for past publications. KHR is an execute of NAVER. JSYN is a board member of the Asian Society for Gyn Robotic Surgery and the Robotic Surgery Society of Singapore. J-CB has received consulting fees and honoraria from Intuitive Surgical, and his institution has received equipment, materials, drugs, medical writing, gifts or other services. PS has received consulting fees from Cambridge Medical Robotics. He has also received an Intuitive Surgical Clinical Research Grant and a Urology Foundation Research & Innovation Grant. Authors who travelled to panel meetings in person received reimbursement for their travel and accommodation from Alira Health. AC did not receive any reimbursement for contribution, travelling or accommodation to the panel meetings.

    • Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

    • Provenance and peer review Not commissioned; externally peer reviewed.