Kenkre et al. designed a multicenter investigation in order to prospectively evaluating preoperative parameters that can predict a stable diabetes remission in obese patients with type 2 diabetes (T2DM) undergoing bariatric/metabolic surgery (1). Candidate predictors , ave been detected from previous retrospective studies and their predicting efficacy will be tested in a large and disseminated cohort of operated patients. Multivariate logistic regression model will be then used to assess the value in terms of prediction of diabetes remission of these preoperative parameters. In severely obese patients with T2DM biliopancreatic diversion (BPD) causes a steadily long term serum glucose level normalization in the majority of the cases. In patients with a preoperative T2DM duration of approximately one year, the diabetes remission was observed in nearly all cases, while in those with a T2DM duration of more than five years the remission rate was lower than 75% (2). This suggest a major role of T2DM duration as a predictor of insulin secretion reserve. In severely obese operated subjects, most likely for a still adequate beta-cell reserve (3), a rapid recovery of insulin secretion after BPD subjects develops (4), thus determining a stable improvement in metabolic pattern. A long T2DM duration entails a severe reduction of functioning pancreatic B cells and a reduced beta bell mass , insulin secretion and a lower postoperative T2DM remission rate. At a lower obesity degree, a progressively lower insulin production and beta cell mass is present and restoration of insulin secretion is lacking (5). By consequence, the postoperative metabolic improvement is essentially sustained by the reduction insulin resistance due to the surgically-obtained weight loss. Since as a rule the extent of weight loss is strictly correlated to the initial body weight, and in most moderate/mild obese subjects a defective beta cell mass can be hypothesized, at low BMI level the T2DM duration loses its predicting efficacy, and the postoperative T2DM remission becomes strictly related only to the initial BMI (5). In our opinion these facts and considerations have to be taken into account when the prediction role of preoperative parameters on the T2DM remission after metabolic/bariatric surgery is investigated, and the hypothesis that any parameter might have not the same predicting efficacy in different cohorts of diabetic patients cannot be ruled out.

REFERENCES

1) Kenkre JS, Ahmed AR, Purkayastha S, Malallah K, Bloom S, Blakemore AI,
Prevost AT, Tan T. Who will benefit from bariatric surgery for diabetes? A
protocol for an observational cohort study. BMJ Open. 2021 Feb 10;11(2):e042355.
doi: 10.1136/bmjopen-2020-042355. PMID: 33568372.

2) Camerini GB, Papadia FS, Carlini F, Catalano M, Adami GF, Scopinaro N. The
long-term impact of biliopancreatic diversion on glycemic control in the
severely obese with type 2 diabetes mellitus in relation to preoperative
duration of diabetes. Surg Obes Relat Dis. 2016 Feb;12(2):345-9. doi:
10.1016/j.soard.2015.05.012. Epub 2015 May 28. PMID: 26381876.

3) Inaishi J, Saisho Y Beta-Cell Mass in Obesity and Type 2 Diabetes, and Its Relation to Pancreas Fat: A Mini-Review. Nutrients. 2020 Dec 16;12(12):3846. doi: 10.3390/nu12123846.

4) Briatore L, Salani B, Andraghetti G, Maggi D, Adami GF, Scopinaro N, Cordera
R. Beta-cell function improvement after biliopancreatic diversion in subjects
with type 2 diabetes and morbid obesity. Obesity (Silver Spring). 2010
May;18(5):932-6. doi: 10.1038/oby.2010.28. Epub 2010 Feb 25. PMID: 20186136.

5) Scopinaro N, Adami GF, Papadia FS, Camerini G, Carlini F, Briatore L,
D'Alessandro G, Parodi C, Weiss A, Andraghetti G, Catalano M, Cordera R. The
effects of biliopancreatic diversion on type 2 diabetes mellitus in patients
with mild obesity (BMI 30-35 kg/m2) and simple overweight (BMI 25-30 kg/m2): a
prospective controlled study. Obes Surg. 2011 Jul;21(7):880-8. doi:
10.1007/s11695-011-0407-0. PMID: 21541815.

6) Scopinaro N, Adami GF, Bruzzi P, Cordera R. Prediction of Diabetes Remission
at Long Term Following Biliopancreatic Diversion. Obes Surg. 2017
Jul;27(7):1705-1708. doi: 10.1007/s11695-017-2555-3. PMID: 28101844.PMID: 33339276

*Kenkre et al. designed a multicenter investigation in order to prospectively evaluating preoperative parameters that can predict a stable diabetes remission in obese patients with type 2 diabetes (T2DM) undergoing bariatric/metabolic surgery (1). Candidate predictors , ave been detected from previous retrospective studies and their predicting efficacy will be tested in a large and disseminated cohort of operated patients. Multivariate logistic regression model will be then used to assess the value in terms of prediction of diabetes remission of these preoperative parameters. In severely obese patients with T2DM biliopancreatic diversion (BPD) causes a steadily long term serum glucose level normalization in the majority of the cases. In patients with a preoperative T2DM duration of approximately one year, the diabetes remission was observed in nearly all cases, while in those with a T2DM duration of more than five years the remission rate was lower than 75% (2). This suggest a major role of T2DM duration as a predictor of insulin secretion reserve. In severely obese operated subjects, most likely for a still adequate beta-cell reserve (3), a rapid recovery of insulin secretion after BPD subjects develops (4), thus determining a stable improvement in metabolic pattern. A long T2DM duration entails a severe reduction of functioning pancreatic B cells and a reduced beta bell mass , insulin secretion and a lower postoperative T2DM remission rate. At a lower obesity degree, a progressively lower insulin production and beta cell mass is present and restoration of insulin secretion is lacking (5). By consequence, the postoperative metabolic improvement is essentially sustained by the reduction insulin resistance due to the surgically-obtained weight loss. Since as a rule the extent of weight loss is strictly correlated to the initial body weight, and in most moderate/mild obese subjects a defective beta cell mass can be hypothesized, at low BMI level the T2DM duration loses its predicting efficacy, and the postoperative T2DM remission becomes strictly related only to the initial BMI (5). In our opinion these facts and considerations have to be taken into account when the prediction role of preoperative parameters on the T2DM remission after metabolic/bariatric surgery is investigated, and the hypothesis that any parameter might have not the same predicting efficacy in different cohorts of diabetic patients cannot be ruled out.

REFERENCES

1) Kenkre JS, Ahmed AR, Purkayastha S, Malallah K, Bloom S, Blakemore AI,
Prevost AT, Tan T. Who will benefit from bariatric surgery for diabetes? A
protocol for an observational cohort study. BMJ Open. 2021 Feb 10;11(2):e042355.
doi: 10.1136/bmjopen-2020-042355. PMID: 33568372.

2) Camerini GB, Papadia FS, Carlini F, Catalano M, Adami GF, Scopinaro N. The
long-term impact of biliopancreatic diversion on glycemic control in the
severely obese with type 2 diabetes mellitus in relation to preoperative
duration of diabetes. Surg Obes Relat Dis. 2016 Feb;12(2):345-9. doi:
10.1016/j.soard.2015.05.012. Epub 2015 May 28. PMID: 26381876.

3) Inaishi J, Saisho Y Beta-Cell Mass in Obesity and Type 2 Diabetes, and Its Relation to Pancreas Fat: A Mini-Review. Nutrients. 2020 Dec 16;12(12):3846. doi: 10.3390/nu12123846.

4) Briatore L, Salani B, Andraghetti G, Maggi D, Adami GF, Scopinaro N, Cordera
R. Beta-cell function improvement after biliopancreatic diversion in subjects
with type 2 diabetes and morbid obesity. Obesity (Silver Spring). 2010
May;18(5):932-6. doi: 10.1038/oby.2010.28. Epub 2010 Feb 25. PMID: 20186136.

5) Scopinaro N, Adami GF, Papadia FS, Camerini G, Carlini F, Briatore L,
D'Alessandro G, Parodi C, Weiss A, Andraghetti G, Catalano M, Cordera R. The
effects of biliopancreatic diversion on type 2 diabetes mellitus in patients
with mild obesity (BMI 30-35 kg/m2) and simple overweight (BMI 25-30 kg/m2): a
prospective controlled study. Obes Surg. 2011 Jul;21(7):880-8. doi:
10.1007/s11695-011-0407-0. PMID: 21541815.

6) Scopinaro N, Adami GF, Bruzzi P, Cordera R. Prediction of Diabetes Remission
at Long Term Following Biliopancreatic Diversion. Obes Surg. 2017
Jul;27(7):1705-1708. doi: 10.1007/s11695-017-2555-3. PMID: 28101844.PMID: 33339276

We thank Schmieding and colleagues for commenting on our study. As noted, we took great effort to compare different symptom checkers fairly.

Vignettes studies have advantages, but as discussed at length in the manuscript, vignette studies also have some inherent limitations. For this reason, as described, we are carrying out real patient studies to further investigate results.

We support the efforts of Schmieding and colleagues to work towards transparency and a standardization of methods in the field of technology-supported clinical decision support: the literature is advancing in this direction. We agree the study of (Semigran et al., 2015) is important, but it evaluated a relatively small number of vignettes and used a small number of researcher vignettes enterers, likely not highly reflective of real world symptom assessment app use.

There was discussion of the limitations of our study in the manuscript which we do not further elaborate here, except with respect to two of the points made by Schmieding and colleagues.

Firstly, regarding open access to the repertoire of case vignettes, in our study, although not open access, researchers can request access to the vignettes as described in the Data Sharing Agreement and this is similar to the approach described for other recently reported studies (Richens et al., 2020). Although there are some advantages in releasing all the vignettes, as done by (Semigran et al., 2015) there is a degree to which this transparency is highly misleading. As soon as any vignettes are made available through open access, some app developers will (entirely legitimately) use these as cases for the optimization of their reasoning engine. This renders the vignettes of no use and misleading for any future studies, as they have unknown objectivity, and overestimate the performance of the (unknown) optimized apps, and only accurately assess the (unknown) non-optimized apps. We have had several requests for access to the vignettes through the procedure described in the Data Sharing Agreement in the paper, and when our the data governance board formally considers these requests, we will discuss whether we will release a random sample of the vignettes to all researchers, holding the remainder back as an objective testing set, thereby providing transparency whilst maintaining a means to ensure objectivity.

Secondly, with respect to transparent reporting of the performance results, for instance, using confusion tables with absolute numbers and percentages of the correct and the observed triage recommendations - the absolute numbers and percentages of the correct and the observed triage recommendations can be derived by readers from the confusion matrices provided in Figure 6, by simply dividing by the weighing applied, which is clearly defined in the paper methods.

References

1. Richens, J.G., Lee, C.M., Johri, S., 2020. Improving the accuracy of medical diagnosis with causal machine learning. Nature Communications 11, 3923. https://doi.org/10.1038/s41467-020-17419-7
2. Semigran, H.L., Linder, J.A., Gidengil, C., Mehrotra, A., 2015. Evaluation of symptom checkers for self diagnosis and triage: audit study. BMJ 351. https://doi.org/10.1136/bmj.h3480

Yours faithfully
Dr. Stephen Gilbert (corresponding author, response discussed with other authors)
Clinical Evaluation Director
Ada Health GmbH
Karl-Liebknecht-Str. 1
10178 Berlin, DE

Conflict of Interest:
Employee of Ada Health GmbH.

We would like to thank the reader for taking interest in our work. We used 2018 international arrival data, as these were the latest published figures at the time of the study. Based on yearly trends in international arrivals before 2018, we have no reason to believe that these figures and more importantly, the between-country differences, would differ from early 2020 figures. We therefore assumed that the 2020 figures recorded prior to the awareness of the spread of SARS-CoV-2 would have been the same as previous years, despite subsequent decreases in international flights occurring after January 2020 as observed by Dr Cairns.

We used 2018 international data as a proxy for countries’ global connections and have therefore found that countries which were better connected globally had significantly higher increases in COVID-19 related mortality during the first wave of the pandemic. A plausible mechanistic link for our findings may therefore be that the virus would have spread significantly out of China before February 2020, after which community transmission would have become predominant in each individual country. Therefore, our findings support the hypothesis that very early travel restrictions should be considered to control the spread of SARS-CoV-2 during future waves of the current and future pandemics. This is of particular importance currently, especially as the world is facing the emergence and international spread of new SARS-CoV-2 strains.

Yours Sincerely,
The Authors

Summary

The study by ADA undertook great effort to compare different symptom checkers competitively. Some of the study’s limitations are unavoidable, such as the authors’ conflict of interest or the limits of vignette-based assessments in general. Other limitations, however, could be avoided in the future. To advance the field of technology-supported clinical decision support, we reason that more transparency and a standardization of methods are needed in at least four areas: (1) open access to the repertoire of case vignettes used to benchmark symptom checker performance; (2) reporting of the performance variation of those who entered the vignettes into the symptom checkers and defining clear guidelines on how to handle the ambiguities that occur when data from vignettes are entered in symptom checkers ; (3) transparent reporting of the performance results, for instance, using confusion tables with absolute numbers and percentages of the correct and the observed triage recommendations; and (4) full reporting of results showing each app’s assessment of each case vignette to make the analyses reproducible, and allow for secondary analysis of the data, as was commendably done by Semigran et al. (2015).

In the following we elaborate these arguments in detail. Ultimately, we think that increasing transparency and standardization of assessment methods will not only advance the field, but also help to address the current discrepancy between industry-funded studies, which tend to make a strong case for how well symptom checkers may perform, and independent studies, which often draw less optimistic conclusions from their data.

Detailed analyses

The study by Gilbert and colleagues evaluates the diagnostic and triage accuracy, safety of triage advice and scope of diagnoses of eight commonly used symptom checker apps. Most similar studies conducted thus far assess either the performance of a single symptom checker, symptom checker performance in a narrow clinical domain, or symptom checkers’ performance on an aggregate level to draw conclusion about the “population” of current symptom checkers. Hence, we acknowledge the authors’ efforts to competitively test and compare the performance of existing symptom checkers apps on a broad set of case vignettes.

Ada’s study design is highly elaborated and probably required higher financial and time investments than most academic research group could afford to conduct such a study: as the authors correctly pointed out in their manuscript, their study includes more case vignettes than prior academic studies, and involved more (wo)men power than comparable studies, as evidenced by the number of authors and contributors involved.

As the authors themselves point out, their inherent conflict of interest limits the interpretability of their study’s results. Hence, we consider the study’s contribution in advancing the methods of assessing technology-supported clinical decision support of greater importance than its results. We would like to review the methods of the present study and point out some avenues toward improving the design of future similar studies and thereby contribute to a framework of best practices for researchers, app developers and regulators for evaluating symptom checker apps in particular, and clinical decision support systems in general.

1) GPs entered the vignettes – but how reliably?

The case vignettes were entered by eight General Practitioners (GPs), but no measures are described to ensure that they entered the vignettes in a manner comparable to each other and/or reproducibly. Without an assessment of the interrater reliability or other measure of rater agreement, the differences in symptom checker’s accuracies observed in the present Ada study might just as well be differences in the eight GPs’ ability to unbiasedly enter clinical vignettes into symptom checkers.

The authors write that “it is known that lay-people are less reliable at entering clinical vignettes than healthcare providers” (see section Strengths and limitations of this study, p. 11). To support this claim, Gilbert et al. refer to the study from Jungmann et al. (2019). The evidence the Jungmann et al. (2019) study provides for this conclusion, however, is scarce and, in our view, does not support the implicit assumption that GPs can be considered the gold standard for entering vignettes without further proof.

Specifically, the Jungmann et al. (2019) study is based on only six participants (two psychotherapists, two psychology students, and two laypersons), case vignettes from the psychiatric domain only, and only considers one app (namely Ada). Although the two psychotherapists had a greater interrater reliability at entering the vignettes than the other groups, they were not perfect either with Cohen’s kappa values of .78 for adult cases, .53 for pediatric cases (this compares to Cohen’s kappa values of .60 and .29 of the laypeople entering the respective cases). In contrast, Semigran et al. (2015), report a Cohen’s kappa of .90 for two non-clinically trained persons entering a random sample of 25 vignette evaluations.

Taken together, the Jungmann et al. (2019) and Semigran et al. (2015) study stress the importance of determining the interrater reliability of the test users entering the vignettes (whether they are healthcare provider or not), rather than relying on the assumption that GPs are the gold standard at entering vignettes.

Although studies based on case vignettes do not provide high ecological validity, they can help to estimate the maximum triage and diagnostic accuracy of a symptom checker under the ideal condition of a user who minimizes variation during data entry. To avoid limitations due to unreliable or untransparent data entry, future studies could (1) let all vignettes be entered by the same person, (2) develop clear guidelines on how to enter vignettes and how to handle ambiguities, (3) determine interrater reliability of those users entering vignettes and/or (4) app developers themselves could specify how information is best entered into their app (an approach followed by Berner et al. 1994). However, the latter approach might not work with all types of symptom checkers.

2) "Fair" case vignettes? - Hard to tell.

Unfortunately, we as readers cannot get a clear picture of the vignettes and the methods how they were created because the authors do not provide open access to the case vignettes. The authors describe that they made significant efforts to generate a pool of representative and suitable case vignettes but leave room for interpretation as to how exactly this was done: For instance, they state that vignettes “were created to be fair cases representing real-world situations.” (p. 2). Similarly, the authors call it a strength that “moderately complex and challenging presentations” (p. 10) were included. They do not elaborate, however, what constitutes a fair vignette in their definition, why unfair vignettes should be excluded, which vignettes are deemed complex or challenging, which ones are not, and why. Further elaboration on how case complexity was operationalized would have been of great benefit for other researcher or app developers planning to replicate or to conduct a similar study.

To advance the general understanding of this study and symptom checkers in general, we suggest that authors should specify whether labels such as fairness and complexity are based on an objective criterion (e.g., number of symptoms, incidence of diagnosis or symptom), on the judgment of the authors who created the vignettes, the GPs who set the gold standard, or the GPs who entered the vignettes. Also, we suggest that the authors of studies on symptom checkers should provide open access to the vignettes used in their studies. Without more transparency, it will remain difficult to judge the validity of research findings, and it will be impossible to replicate this study and other studies independently to advance our understanding of what symptom checkers can and cannot do, which is a goal the authors themselves call for in their paper.

3) Transparent presentation of results

The authors introduce a penalty score to rate minor triage errors differently than major triage errors. That is, they consider a recommendation to seek emergency care for a vignette a greater error if home care rather than a non-urgent primary care consultation is the appropriate response. This approach is valuable. It is not valuable, however, if the presentation of penalty scores replaces a more transparent presentation of the results, for instance, in a confusion table with absolute numbers and percentages of the correct and the observed triage recommendations. Although confusion tables may be simple contingency tables, they can help readers identify patterns in the data on their own and assess the validity of the conclusions drawn by the authors more easily than with the “weighted” and “normalized” penalty score tables and aggregate inferential statistical analyses as provided by the authors. Additionally, we suggest that future studies should provide a transparent presentation of their results similar to Semigran et al. (2015), showing how each symptom checker responded to each case vignette.

Cited articles:

1) Jungmann et al. (2019): https://formative.jmir.org/2019/4/e13863/

2) Semigran et al. (2015): https://www.bmj.com/content/351/bmj.h3480/

3) Berner et al. (1994): http://www.nejm.org/doi/abs/10.1056/NEJM199406233302506