Dear Editor,
Apea VJ et al. should be congratulated on elucidating the interplay between overlapping factors that influence patient outcomes from COVID-19. From the first reports about this disease, observers have noted that adipose tissue, perhaps through its association with diabetes and heart disease, has been linked to the severity of presentation and to subsequent clinical course. In studying a cosmopolitan patient population in a single large centre, the authors have cleverly eliminated many confounding factors and allowed the influence of ethnicity, per se, to be assessed.
In this context though, the use of body mass index as a measure of obesity may be too blunt a discriminator. While a BMI of 30 kg/m2 is conventionally taken to define obesity, this measure has significant limitations.2,3 The most obvious is that it does not actually measure lean versus fatty tissue.3 More particularly, the BMI or Quetelet index threshold of 30 kg/m2 may be insensitive to the presence of obesity in those of Asian origin.4 Other authors have shown that the cut off in Asians may be optimally set at a lower level (e.g. 27 kg/m2 for men, and as low as 25 kg/m2 for women).5 We wonder, were Apea VJ et al.’s observations re-examined using such thresholds, if the poor outcomes seem in Asian patients would be more fully accounted for?
We would welcome their views on this question. As the world’s most populous countries are in Asia, and the pandemic continues apace, this is perhaps a matter of the very greatest global health importance.
Pradipta Bhakta,
Brian O’Brien.
References:
1. Apea VJ, Wan YI, Dhairyawan R, et al. Ethnicity and outcomes in patients hospitalised with COVID-19 infection in East London: an observational cohort study. BMJ Open. 2021 Jan 17;11(1):e042140. doi: 10.1136/bmjopen-2020-042140. PMID: 33455936; PMCID: PMC7813387.
2. Obesity: preventing and managing the global epidemic. Report of a WHO consultation. World Health Organ Tech Rep Ser. 2000;894:i-xii, 1-253.
3. Frankenfield DC, Rowe WA, Cooney RN, Smith JS, Becker D. Limits of body mass index to detect obesity and predict body composition. Nutrition. 2001;17(1):26-30.
4. WHO Expert Consultation. Appropriate body-mass index for Asian populations and its implications for policy and intervention strategies. Lancet. 2004;363(9403):157-63.
5. Goh VH, Tain CF, Tong TY, Mok HP, Wong MT. Are BMI and other anthropometric measures appropriate as indices for obesity? A study in an Asian population. J Lipid Res. 2004;45(10):1892-8.

This is a response to a comment on the paper dated 20 January 2021.

Dear Dr. Ćirković,

We thank Ćirković for welcoming our study. Before answering the points raised by Ćirković I first mention a currently-under-peer-review scientific communication between myself and the editor of the journal which published the article authored by Ćirković, that he cites in his comment [1]. The full text of this Letter to the Editor has been available as a preprint in the public domain since 18 Dec 2021 [2].

- I address below the points raised by Ćirković on his comment on our BMJ Open paper in turn:

1. TRIPOD and PROBAST checklists - searching the clinical literature using PubMed and Google Scholar identifies no studies that have used these checklists in the evaluation of vignettes studies or studies of symptom assessment applications. We reviewed the literature on symptom assessment app in our paper, and I have looked again at each of these papers, and none of them used the TRIPOD or the PROBAST checklist, nor did a recent paper in this domain from an independent academic group [3], nor did Ćirković himself in [1]. The TRIPOD checklist (transparent reporting of a multivariable prediction model for individual prognosis or diagnosis) is designed for studies into diagnostic and prognostic models for clinicians, and although an interesting suggestion, the use of such a checklist is not compulsory and not having adopted it is not a study limitation in itself. The PROBAST (Prediction model Risk Of Bias ASsessment Tool) tool, also designed for studies into diagnostic and prognostic models for clinicians, was however designed for systematic reviews, not for research studies themselves [4] - as an instrument I find it poorly suited for this purpose. Nonetheless, I have applied the PROBAST method to our study, and find the tool shows a low risk of bias for our study. Previous studies have used much smaller numbers of vignettes and show much higher risk of bias. 200 is the highest number of vignettes used in any multi-symptom assessment app evaluation. We rebut this point and do not find the suggested checklists either compulsory, required, or fairly applied to our study by Ćirković.

2. “choice of the top-3 and top-5 appearing conditions without any rating within according to the respective rank on which a condition is displayed to the user” - this is clearly a misleading statement as we also reported the top-1 and the comprehensiveness metric alongside top-3 and top-5, in every case. Taken together, consideration of top-1, top-3, top-5, comprehensiveness and relevance holistically address the respective rank on which a condition is displayed to the user. This is the approach used in all similar studies, as reviewed in our paper. We searched the literature extensively for a single metric that combines weighting the properties of all five metrics listed, and no such metric has been described except in the highly technical/mathematical artificial intelligence/machine learning literature. If a clinically meaningful and interpretable weighted metric were proposed and validated in the literature we would be very interested in also including it in future studies. We rebut this point raised by Ćirković.

3. Figure 5 colouring - “overconservative advice is in some way “less a problem” than the underconservative one. This may be true from a manufacturer’s standpoint regarding liability issues towards potential users”. The opinions expressed on colour shading of figures by Ćirković are rather subjective. The colour shading chosen is in line with the safety metric used in the paper and we maintain that patient safety is paramount. The Hippocratic ‘first do no harm’ principle dictates that indeed overconservative advice is less of a problem than underconservative advice. This is not for ‘liability issues’ but due to the underlying fundamental importance of patient safety, that is enshrined in the regulatory environment and the principles any responsible manufacturer should and must adopt. Of course too-conservative advice is undesirable, and matching is optimal, but as symptom apps iteratively improve, our position is that this must be through an approach of maintaining sufficient safety whilst making stepwise reduction in too-conservtive advice. Also, note that in physician triage it has been demonstrated that human doctors are often relatively cautious and that a degree of over-cautiousness is also appropriate for safety (but of course a balanced degree): "Pre-hospital triage of emergency patients is necessarily an inexact process and some degree of overtriage must generally be accepted “ [5,6]. We provide the data to allow readers to draw their own conclusions about what constitutes the best triage approach for a specific use, as described in point (4) below.

4. “the choice to merely evaluate “safe advice” as opposed to “correct advice” or similar may not be enough to correctly assess the true value of the apps for patients, healthcare providers or public health in general.” - this point is misleading and false. Safe advice was not evaluated in isolation. Full data on exactly matching advice (i.e. “correct advice”) and for every level of matching/degree-of-not-matching was provided for every app and GP evaluated. This was provided in triplicate in the paper: (i) by means of the bar-chart of each advice grouping in Figure. 5; (ii) by fully detailed complete confusion matrices in Figure 6; and, (iii) the complete data, for all vignettes and for the subgroups of vignettes types in Supplementary Table 11. We believe this addresses the point raised by Ćirković.

5. Although the comment was titled ‘Five missing limitations’, the comment jumps from ‘thirdly’ to ‘lastly’ without listing a fifth point.

References:

1 Ćirković A. Evaluation of Four Artificial Intelligence-Assisted Self-Diagnosis Apps on Three Diagnoses: Two-Year Follow-Up Study. J Med Internet Res 2020;22:e18097. doi:10.2196/18097
2 Letter to the Editor - Evaluation of Four Artificial Intelligence–Assisted Self-Diagnosis Apps on Three Diagnoses: Two-Year Follow-Up Study. JMIR Preprints. https://preprints.jmir.org/preprint/26514 (accessed 22 Jan 2021).
3 Hill MG, Sim M, Mills B. The quality of diagnosis and triage advice provided by free online symptom checkers and apps in Australia. The Medical Journal of Australia 2020;212:514–9. doi:10.5694/mja2.50600
4 Wolff RF, Moons KGM, Riley RD, et al. PROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies. Ann Intern Med 2019;170:51–8. doi:10.7326/M18-1376
5 Rørtveit S, Meland E, Hunskaar S. Changes of triage by GPs during the course of prehospital emergency situations in a Norwegian rural community. Scand J Trauma Resusc Emerg Med 2013;21:89. doi:10.1186/1757-7241-21-89
6 Lossius HM, Rehn M, Tjosevik KE, et al. Calculating trauma triage precision: effects of different definitions of major trauma. J Trauma Manag Outcomes 2012;6:9. doi:10.1186/1752-2897-6-9

Yours faithfully
Dr. Stephen Gilbert Ph.D (corresponding author, response discussed with other authors)
Clinical Evaluation Director
Ada Health GmbH
Karl-Liebknecht-Str. 1
10178 Berlin, DE

This is a response to a comment on the paper dated 7 January 2021.

Dear Oscar Garcia-Esquirol, Cofounder and Chief Medical Director at Mediktor, one of the symptom checkers evaluated in the study,

Thank you for your comment on this paper.

Your letter addressed strong criticism at the paper [1] with respect to bias. I will answer these points in turn below. Our study included a rigorous design process conducted by experienced clinical researchers, data scientists and health policy experts, with the methodology and analysis peer-reviewed by independent and experienced primary care physicians and medical informatics experts at universities in the UK, and in Brown University in the US. To ensure a fair comparison, our team used a large number of clinical vignettes, which were generated from a mix of real patient experiences gleaned from the UK’s NHS 111 telephone triage service and from those generated from the many years’ combined experience of the research team. The gold-standard triage level used in the study was set independently of vignette creation, vignette review and vignette diagnosis gold-standard setting - this was done by a separate panel of three experienced primary care practitioners using a tie-breaker panel method based on the matching process set out by [2].

A strength of this paper is that it not only compares a range of symptom assessment apps to each other, but also compares their performances to that of practicing GPs. While the Ada app performed most strongly of any of the symptom assessment apps in condition-suggestion, it is of note, that it was not the single strongest performing app on the triage safety metric, in which the Symptomate app achieved 97.8%; Ada achieved 97.0%; and Babylon, also performed very strongly achieved 95.1%. The Mediktor app had a safety performance on this study metric
of 87.3% that was outside 2 S.D. of the performance of the evaluated GPs (Mediktor: 87.3% (p=1.3×10-3, statistically significant difference to GP performance). This is relevant to note in this response to your comment, as the performance of Mediktor in the study may be a factor in the nature of your criticism of the paper (the author of the comment is the Chief Medical Officer and co-founder of Mediktor). The analysis was carried out by the independent vignette entry GPs in a manner to be fair to all apps. The study was not designed to discriminate against Mediktor in the urgency advice safety measurement any more than it was designed so that Symptomate or Babylon would do well in the safety measurement.

With respect to criticism of the number of levels of advice. A universal scale of comparison was needed as the urgency advice of apps was to be compared. Six advice levels were used as a universal scale for comparison of the symptom assessment app advice and this was also used as a scale for the tested GPs to provide their advice on. The scale was developed based on independent input and advice. It is not the native urgency advice scale of the Ada app (which provides advice on 8 levels), or any other evaluated app. As you point out, some apps provide advice on five levels. The 6-point scale is a fair choice as most of the apps evaluated provided at least 6 levels of advice. The mapping of 5 levels of advice to a 6-point scale could not have resulted in any disadvantage of any app in the safety metric measurement reported, as this was determined, as stated in the paper, as at “the gold standard level, more conservative, or no more than one level less conservative)”.

The comment states that the ‘observers’ ‘were people who have worked at Ada, or colleagues of those who have worked for this company’ It is not entirely clear how the term ‘observers’ is used in your comment. For the following four groups: (i) the vignettes gold standard condition-suggestion’s review panel; (ii) the vignettes gold standard triage setting panel; (iii) the app-entry-doctors; (iv) the condition matching panel: all were independent, and none ‘were people who have worked at Ada, or colleagues of those who have worked for this company’, some were recruited from wider professional networks - e.g. members of the same professional society or online clinical forum etc, but none of these ‘observers’ had any connection that would result in the ‘Ada variable group becom[ing] more interesting to the researchers than the other variables to be evaluated [, when] measuring the ADA variable, researchers may have more interest in attaining information as they benefit from it in regard to the rest.’ With respect to the condition matching panel, they were blinded to the source of the condition-suggestions, having no information about whether the condition suggestions came from a tested-GP or from an app, or which app they came from. The final group of GPs who participated in the study were the 7 tested-GPs, whose own performance in ‘diagnosing’ the vignettes was assessed. Of these, four had previously worked for the Ada Health telehealth service Dr Chat but were no longer employees at Ada and the other three had never ‘worked at Ada’, and were not ‘colleagues of those who have worked for this company’.

The comment repeatedly makes the point that there were biases related to the researcher, who acted as a patient ‘when GPs are evaluated by phone’, including due to the Hawthorne effect. These points are in their totality incorrect, as the commenter has misinterpreted the study methodology on this point. We designed the study to ensure that all apps and all tested-GPs were compared to the gold standard diagnosis independently. There was no comparison between the tested-GP answers and the tested apps, for Ada or for any other evaluated app. Each tested-GP was compared to the gold-standard, via the matching panel. Likewise, each evaluated app was compared to the gold-standard via the matching panel. The telephone interview was solely for the evaluation of the tested-GPs. There could have been no possibility of biasing the tested-GPs answers towards Ada (or any of the other apps) condition-suggestions, as there was no comparison to the app’s condition-suggestions. There was no possibility of bias towards Ada here. The procedure is clearly explained in the original manuscript, particularly in figure 1, and along with the reiteration here, this point is fully rebutted.

-- Representativeness of the coverage of the vignettes to real population disease prevalence. --
This point of criticism in the comment is already addressed in full within the study and its technical appendix 'Appendix S1: Comparison of the vignettes to UK health statistics'. This appendix shows the degree of match to a fair prevalence data set - the Quality and Outcomes Framework (QOF) for the NHS in England (Note - there is no perfect comparison set which maps real symptom assessment users). The appendix shows there is matching on broad terms across the prevalence of conditions, but also shows that this match is not perfect in all respects. This is also addressed in the study main manuscript.

-- Suggested bias associated with the manner of case entry in apps --
The comments incorrectly states that the study failed to allow for ‘in Mediktor’s case, [that] it is possible to enter more than one symptom, sign or information referring to the patient at the initial stage of consultation’. The study did not stipulate to the app-entry-doctors how they should use each app. They familiarised themselves with each app through the app itself and, through instructions for use for the app (when these were clearly made available within the app interface) In other words, the method of entry of a vignette into an app was not predetermined at all by the study investigators, so as not to bias the entry. Each app-entry-doctor chose their own entry, as guided by the app, and as guided by their role as a user. Each vignette had a listed primary complaint, and the app-entry-doctors were always instructed to enter this first. After this, they were guided by the app interfaces (and app instructions when readily available through the interface) as to whether to enter other symptoms simultaneously or later, to simulate a real patient learning to use the app.

I hope the clarifications provided here aid the understanding as some of the points above related to simple misinterpretation. Whilst it is impossible to remove all biases in a study of this sort, the design sought to use separate groups of doctors at each stage and blind them to the origin of the data where possible.

-- The role of app manufacturers/developers in studies on medical device apps --
Another aspect of Garcia-Esquirol‘s comment is, however, clearly misleading. The comment misrepresents involvement of an app developer/manufacturer in a study as being proof of bias in the study. As the author of the comment is the CMO and co-founder of an medical device app manufacturer, he must surely be fully aware that involvement of manufacturers in studies, both in the planning and in the funding, is required under the EU medical device regulation framework [3]. As is commonplace and fully acceptable, Mediktor (cofounded by the author of the comment) has been affiliated to studies researching their products [4] - see the conflicting interests statement of this paper for the connection between the authors of the study and the Mediktor company. The landscape of clinical studies of individual medical devices is predominated by manufacturer funded studies, and this very often involves some manufacturer personnel or link, as was the case for Mediktor in [4]. Manufacturers have a duty to assess the state-of-the-art and comparison studies, often done through non-inferiority studies, which are generally funded by one manufacturer, and compare their product to the product of a competitor. These are commonplace, standard, and often expected for implanted medical devices and for pharmaceuticals [5]. The comments relating to the involvement of Ada staff in the study and in the funding of the study create a misleading false premise, as neither funding a study nor involvement in a study equate to ‘budgetary biases’ and all involvement in the study was correctly stated in the Contributors, Funding and Competing Interests sections of the paper - as it is done for any manufacturer sponsor study. Not all of the scientific readership of your comment will know the post market clinical evaluation/study requirements on app manufactures [3], and, therefore, these remarks are either wilfully misleading through misrepresenting the standard approach to study funding for medical devices (and pharmaceutical products), or, misrepresenting or being unaware of Mediktor's own stated conflicting interests in their own published studies.

With respect to your point, that there is a need for further studies, we are in full agreement on this and for this reason addressed it in the paper. We stated 'To address the effect of patients entering data directly into an app about their own acute conditions, an observational investigation is currently underway in an acute clinical setting in the USA', and this is being carried out in a completely independently funded study, by investigators who include the non-Ada affiliated coauthor of this study.

In summary, of the critical points made in the comment, there is only one that is of any validity, which relates to whether an optimal universal reference mapping for all apps (and GPs) would be to a 5-level triage scale or a 6-level triage scale. This is open to debate, but in the end, a single universal scale is required in a comparison study, and we defend this 6 point scale as being as fair a choice overall as a 5 point scale. The remaining points are incorrect and the results of misunderstanding, or misrepresentations of stated involvement as proof of bias.

The commenter asks whether the low safety score of Mediktor might relate to bias in the study conduct or design. Upon review, in a number of scenarios Mediktor provided the correct condition-suggestions, but provided the user only a "medium" advice level for vignettes in which call-ambulance was indicated by all/nearly-all other apps, by all/nearly-all of the tested-GPs and by the vignette gold standard. The commenter may contact us directly if he would like additional information regarding these examples (within the terms of the paper's Data Availability Statement).

Overall, this study has been well received in the scientific community (see the link to the wider discussion of the paper from this BMJ Open page).

1 Gilbert S, Mehl A, Baluch A, et al. How accurate are digital symptom assessment apps for suggesting conditions and urgency advice? A clinical vignettes comparison to GPs. BMJ Open 2020;10:e040269. doi:10.1136/bmjopen-2020-040269
2 Semigran HL, Linder JA, Gidengil C, et al. Evaluation of symptom checkers for self diagnosis and triage: audit study. BMJ 2015;351. doi:10.1136/bmj.h3480
3 Regulation (EU) 2017/745 of the European Parliament and of the Council of 5 April 2017 on medical devices, amending Directive 2001/83/EC, Regulation (EC) No 178/2002 and Regulation (EC) No 1223/2009 and repealing Council Directives 90/385/EEC and 93/42/EEC (Text with EEA relevance. ). 2017. http://data.europa.eu/eli/reg/2017/745/oj/eng (accessed 11 Jan 2021).
4 Moreno Barriga E, Pueyo Ferrer I, Sánchez Sánchez M, et al. [A new artificial intelligence tool for assessing symptoms in patients seeking emergency department care: the Mediktor application]. Emergencias 2017;29:391–6.
5 Schumi J, Wittes JT. Through the looking glass: understanding non-inferiority. Trials 2011;12:106. doi:10.1186/1745-6215-12-106

Yours faithfully
Dr. Stephen Gilbert (corresponding author, response discussed with other authors)
Clinical Evaluation Director
Ada Health GmbH
Karl-Liebknecht-Str. 1
10178 Berlin, DE

Conflict of Interest:
Employee of Ada Health GmbH.

I read with great interest the article “How accurate are digital symptom assessment apps for suggesting conditions and urgency advice? A clinical vignettes comparison to GPs”
published by Gilbert S. Mehl A, and Baluch A. et al. It evaluated the effectiveness of eight symptom checkers – including Mediktor, of which I am part.

While I wholeheartedly agree with the statement the study researchers make, citing Chambers D et al. that “rigorous studies are required to show that these apps provide safe and reliable information,” I believe this article is a far cry from being such a rigorous study. I consider it to be little more than an advertising campaign by ADA, since the study falls prey to multiple biases, which I will describe below. These biases invalidate the article as a scientific publication.

If we examine the foundations upon which the study rests, we can classify these biases thus:

1) Biases due to the study variables:

By creating vignettes that simulate clinical cases, a bias is created in the observational conditions of the study variables (SV). The diagnostic accuracy of symptom checkers (SC), created to assess patients in real conditions, are evaluated under artificial conditions. This further violates the nature of the measurement since they are simulated cases created by staff with a direct relationship to ADA, one of the symptom checkers analysed in the study.
When comparing the SCs’ capability to perform triage, the researchers create a bias by modifying the possible number of levels of urgency to six, while some of the SVs are based on five levels (results distributed on five levels according to complex criteria of which the researchers are ignorant).

In addition, they did not publish the Vignettes used for the research. This is a clear lack of transparency and make it impossible to replicate or validate the findings.

2) Biases from the observer’s perspective:

The observers are people who have worked at ADA, or colleagues of those who have worked for this company. This causes an inherent evaluation bias in examiners, who are far from impartial since they are influenced by the “culture” at ADA – one of the symptom checkers evaluated in the study.
The ADA collaborator, who acts as a patient when GPs are evaluated by phone, creates a bias when comparing the results of the symptom checkers with the result from the group formed of GPs. Obviously the “ADA variable” result is closer to the result of the GP group with which it is compared when the person answering the questions is an ADA collaborator and the GPs questioning him are colleagues.

3) Measurement tool bias:
The cases selected by ADA to measure the coverage of illnesses in each SC may be subject to a measurement bias. For example, if the vignettes are meant to study the existence (prevalence) of an illness in the SC, then considering unsuitable items when creating the vignettes would alter the study sensitivity. The same measurement tool, with an excessive number of items that are of little significance in relation to an illness, will alter the specificity of the measured result. For this reason, it is not possible to measure diagnostic coverage of the SCs using cases created according to the criteria of people working at ADA, one of the study variables.

Another means of pinpointing the biases in the study is to assess them via the study stages:

Stage 1 - Selection biases
The selection of cases for the vignettes, taken from NHS cases (30%) and ADA staff (70%) carries a selection procedure bias, given that they have not been chosen randomly from a wide case sample but rather specifically selected for the study. This bias is heavily magnified since the cases have been created “bespoke” by ADA employees so as to improve their diagnostic results.

Stage 2 - Information or measurement bias

The study falls prey to a procedural or Feinstein bias, where the ADA variable group becomes more interesting to the researchers than the other variables to be evaluated. When measuring the ADA variable, researchers may have more interest in attaining information as they benefit from it in regard to the rest.
In contrast, three information or measurement biases are incorporated when a GP directly or indirectly related to ADA conducts the telephone interview with the ADA consultant acting as a patient:
• the attentional bias or Hawthorne effect, when the participating GP may alter their responses if they know they are being evaluated;
• the interviewer bias, when the latter is not impartial and may induce specific responses;
• and lastly the obsequiousness bias, when informants (in this case colleagues of the ADA researcher) answer what they believe the interviewer (an ADA collaborator) wishes to hear.

Stage 3 - Confusion bias

In this case, measurement of the result does not take the “exposure effect” into account. However, rather than exposure here it would be omission of an item modifying the results of one of the study variables. In Mediktor’s case, it is possible to enter more than one symptom, sign or information referring to the patient at the initial stage of consultation. This generates weighted questions that are quite different than those created if the artificial intelligence evaluates the inputs one by one. So Mediktor’s capability to pre-diagnose, calculate urgency and issue triage is thereby underestimated.

To conclude, it is clear that the study is riddled with biases ranging from its design as a study to its execution.

During planning, it fell victim to:
• budgetary biases, since it was financed by ADA, one of the study variables;
• biases incorporated during the sampling process, due to the lack of representativeness of the simulated (as opposed to real) cases;
• biases created while gathering data, since the information could not possibly be collected impartially by people with links to ADA;
• and lastly biases in the analysis and interpretation stage by “standardising” the triage scale according to their own criteria.

I would stress the need for fresh clinical studies that are firmly based on good practice and scientific rigour, and free of authorial conflicts of interest. In my opinion, the best practices for scientific and independent symptom checker comparison would include the following components:

• Be conducted with real patients instead of Vignettes.
• If limited to Vignettes, make the Vignettes public.
• Be lead by true independent researchers without professional or financial ties to the SCs being studied.

Such studies would truly aid health professionals and help patients recognise valid symptom checkers. They could make the best decisions about their own and their patients’ health.

Best regards,
Oscar García-Esquirol
MD, Intensive Care Specialist
Co-founder and Medical Director at Mediktor
Barcelona, Spain