We read with interest the assessment tool given the acronym PAGE: Paediatric Admission Guidance in the Emergency Department.
bmjopen-2020-043864.
This PAGE study is of particular importance in emergency departments
in a hospital ( for instance in Oman ) where pediatrics is not amongst a hospital specialities.
Further development of PAGE may answer a recent call to improve the Emergency Department (ED) response to children injuries in Oman.
doi: 10.1136/bmjpo-2018-000310
A recent retrospective review of ED trauma registers in Oman also found, first, high children injuries incidence of falls, home injuries and burns.
Second, suggests future work for age-targeted interventions to better respond to children injury events .
Dear Editors,
We would like to inform you and the readers of BMJ open that we plan to slightly enlarge the enrollment of our trial "Randomised, non-inferiority, controlled procedural outcomes TrIal comParing reverse T And Protrusion versus double-kissing and crush stenting: protocol of the TIP TAP I randomised trial", which was published in BMJ Open. 2020; 10(6): e034264. Our original intention was to recruit 50 patients. Unfortunately, the OCT data pertaining the primary endpoint of the study could not be assessed in two of the patients. We therefore plan to recruit 2 additional patients following the same procedures. The analysis population of the study will therefore maintain its previously planned size of 50, the safety and IIT populations will however include all 52 patients.
Safety outcomes have been and will be regularly reviewed by three cardiologists who are not involved in any procedure of the study.
"We however found that the main determinant was the total number of international arrivals in the country (2018 figures), signifying transmission of the infection through travel. Although the data were from 2018, there is no reason to believe that international travel figures between countries would be different in early 2020."
The study period from December 2019 to June 2020 covered a period in which international passenger capacity reduced by 8% in February 2020, by 46% in March 2020 and by 90% in April 2020 year-on-year compared to 2019 (ICAO data). It is therefore demonstrably incorrect to assert that there is no reason to believe travel figures would be different.
Dear Dr Bhakta and Dr O'Brien,
We are grateful for the very valid insights. Our intention was not to explore BMI as a COVID risk factor due to reasons such as incomplete data and measurement accuracy, but to account for it as a potentially confounding variable in the relationship between ethnicity and outcome. Evidence regarding ethnic differences in BMI remains sparse and historic reports may not accurately reflect cohorts represented in contemporary publications. This is nevertheless an important subject to address for future research.
Yize I Wan,
John R Prowle
We read with interest the MaPs in ENL study protocol (de Barros et al, BMJ open 2020).[1] The availability of an alternative drug for cases of chronic or recurrent type 2 lepra reaction with steroid toxicity or unresponsiveness is the need of the hour and although drugs like thalidomide, cyclosporine etc. are prescribed in such patients but cost, adverse effect or non-availability are the major deterrents in their use. The researchers must be complimented for their bold decision to choose methotrexate based only on case reports or a short series of patients (15 patients in total were given methotrexate for type 2 reactions).[2] This is a well-designed protocol aimed at possibly adding another drug to our therapeutic armamentarium for the crippling erythema nodosum leprosum (ENL) reactions. If found useful, methotrexate has the advantage of being a cheap, easy to administer and readily available drug.
The study proposes to randomize patients with acute, chronic or recurrent ENL into two groups, receiving 20 weeks of prednisolone with either 48 weeks of methotrexate or placebo. While use of methotrexate is justified in patients with chronic and recurrent ENL, , same might not be applicable to cases of acute ENL. Acute ENL, by definition, is an episode that lasts lesser than 24 weeks,[3] as has been defined in the study protocol as well. Most of these cases with acute episode respond well to corticosteroids, clinical response has been observed as early as in 2 week...
We read with interest the MaPs in ENL study protocol (de Barros et al, BMJ open 2020).[1] The availability of an alternative drug for cases of chronic or recurrent type 2 lepra reaction with steroid toxicity or unresponsiveness is the need of the hour and although drugs like thalidomide, cyclosporine etc. are prescribed in such patients but cost, adverse effect or non-availability are the major deterrents in their use. The researchers must be complimented for their bold decision to choose methotrexate based only on case reports or a short series of patients (15 patients in total were given methotrexate for type 2 reactions).[2] This is a well-designed protocol aimed at possibly adding another drug to our therapeutic armamentarium for the crippling erythema nodosum leprosum (ENL) reactions. If found useful, methotrexate has the advantage of being a cheap, easy to administer and readily available drug.
The study proposes to randomize patients with acute, chronic or recurrent ENL into two groups, receiving 20 weeks of prednisolone with either 48 weeks of methotrexate or placebo. While use of methotrexate is justified in patients with chronic and recurrent ENL, , same might not be applicable to cases of acute ENL. Acute ENL, by definition, is an episode that lasts lesser than 24 weeks,[3] as has been defined in the study protocol as well. Most of these cases with acute episode respond well to corticosteroids, clinical response has been observed as early as in 2 weeks.[4] It has been reported in a large epidemiological study that about one third of all ENL reactions are single episode.[5] Moreover, methotrexate may take up to 3 weeks for noticeable therapeutic effect as cited by the authors, so its required benefit will not be available in acute ENL.[6] Current World Health Organization (WHO) guidelines also recommend corticosteroids for a duration of 12 weeks in severe ENL.[7]
Administering methotrexate to such patients for a prolonged duration of 48 weeks (51 weeks to be precise–as per protocol) does not seem to be scientifically or ethically justifiable. Such prolonged administration of methotrexate (up to 25 mg/ week) in acute ENL cases will add to potential hematological and biochemical toxicities of multidrug therapy (MDT).[8] The researchers are definitely not looking at the data for preventing the occurrence of another episode of acute reaction. The continued administration of methotrexate is bound to create a bias- because of the possibility of delay in the occurrence or reduced severity of the next ENL episode. Few unfortunate patients with long standing ENL of more than 4 years could also have been included.
The protocol recommends, daily supplementation of 5 mg of folic acid to reduce the adverse effects of methotrexate. The dosage seems to be in excess of the usual recommendations by various dermatology and rheumatology guidelines, which range from 0.5-2 mg per day.[9-11] Although there is no consensus on dosage of folic acid supplements, there is some evidence suggesting that higher doses may possibly reduce the therapeutic efficacy of methotrexate.[12-14]
Under the ‘indications for additional steroids’- there is mention of Type 1 reaction – it is not clear if these patients will also be included in the analysis.
A recommendatory comment is in order on the continuation and regularity of the MDT for those on MDT. Also if a patient develops ENL after the completion of MDT- how the fear of activity of disease is to be assuaged- whether an additional anti-leprosy drug (clofazimine was recommended earlier)– would be added to the regimen.
REFERENCES
1. de Barros B, Lambert SM, Shah M, et al. Methotrexate and prednisolone study in erythema nodosum leprosum (MaPs in ENL) protocol: a double-blind randomised clinical trial. BMJ Open 2020;10:e037700.
2. Perez-Molina JA, Arce-Garcia O, Chamorro-Tojeiro S, et al. Use of methotrexate for leprosy reactions. Experience of a referral center and systematic review of the literature. Travel Med Infect Dis 2020;37:101670.
3. Walker SL, Balagon M, Darlong J, et al. ENLIST 1: An International Multi-centre Cross-sectional Study of the Clinical Features of Erythema Nodosum Leprosum. PLoS Negl Trop Dis 2015;9:e0004065-e.
4. Kaur I, Dogra S, Narang T, et al. Comparative efficacy of thalidomide and prednisolone in the treatment of moderate to severe erythema nodosum leprosum: a randomized study. Australas J Dermatol 2009;50:181-5.
5. Kumar B, Dogra S, Kaur I. Epidemiological characteristics of leprosy reactions: 15 years experience from north India. Int J Lepr Other Mycobact Dis 2004;72:125-33.
6. Warren RB, Weatherhead SC, Smith CH, et al. British Association of Dermatologists’ guidelines for the safe and effective prescribing of methotrexate for skin disease 2016. Br J Dermatol 2016;175:23-44.
7. World Health Organization. WHO Guidelines for the management of severe erythema nodosum leprosum (ENL) reactions. Geneva, Switzerland: World Health Organization 2015. Available from: https://www.who.int/lep/research/WHOenlguide.pdf
8. Cruz R, Bührer-Sékula S, Penna GO, et al. Clinical trial for uniform multidrug therapy for leprosy patients in Brazil (U-MDT/CT-BR): adverse effects approach. An Bras Dermatol. 2018;93:377-84
9. Singh JA, Furst DE, Bharat A, et al. 2012 update of the 2008 American College of Rheumatology recommendations for the use of disease-modifying antirheumatic drugs and biologic agents in the treatment of rheumatoid arthritis. Arthritis Care Res (Hoboken) 2012;64:625-39.
10. Chakravarty K, McDonald H, Pullar T, et al. BSR/BHPR guideline for disease-modifying anti-rheumatic drug (DMARD) therapy in consultation with the British Association of Dermatologists. Rheumatology (Oxford) 2008;47:924-5.
11. Kalb RE, Strober B, Weinstein G, et al. Methotrexate and psoriasis: 2009 National Psoriasis Foundation Consensus Conference. J Am Acad Dermatol 2009;60:824-37.
12. Khanna D, Park GS, Paulus HE, et al. Reduction of the efficacy of methotrexate by the use of folic acid: post hoc analysis from two randomized controlled studies. Arthritis Rheum 2005;52:3030-8.
13. Salim A, Tan E, Ilchyshyn A, et al. Folic acid supplementation during treatment of psoriasis with methotrexate: a randomized, double-blind, placebo-controlled trial. Br J Dermatol 2006;154:1169-74.
14. Chládek J, Simková M, Vanecková J, et al. The effect of folic acid supplementation on the pharmacokinetics and pharmacodynamics of oral methotrexate during the remission-induction period of treatment for moderate-to-severe plaque psoriasis. Eur J Clin Pharmacol 2008;64:347-55.
Dear Editor,
Apea VJ et al. should be congratulated on elucidating the interplay between overlapping factors that influence patient outcomes from COVID-19. From the first reports about this disease, observers have noted that adipose tissue, perhaps through its association with diabetes and heart disease, has been linked to the severity of presentation and to subsequent clinical course. In studying a cosmopolitan patient population in a single large centre, the authors have cleverly eliminated many confounding factors and allowed the influence of ethnicity, per se, to be assessed.
In this context though, the use of body mass index as a measure of obesity may be too blunt a discriminator. While a BMI of 30 kg/m2 is conventionally taken to define obesity, this measure has significant limitations.2,3 The most obvious is that it does not actually measure lean versus fatty tissue.3 More particularly, the BMI or Quetelet index threshold of 30 kg/m2 may be insensitive to the presence of obesity in those of Asian origin.4 Other authors have shown that the cut off in Asians may be optimally set at a lower level (e.g. 27 kg/m2 for men, and as low as 25 kg/m2 for women).5 We wonder, were Apea VJ et al.’s observations re-examined using such thresholds, if the poor outcomes seem in Asian patients would be more fully accounted for?
We would welcome their views on this question. As the world’s most populous countries are in Asia, and the...
Dear Editor,
Apea VJ et al. should be congratulated on elucidating the interplay between overlapping factors that influence patient outcomes from COVID-19. From the first reports about this disease, observers have noted that adipose tissue, perhaps through its association with diabetes and heart disease, has been linked to the severity of presentation and to subsequent clinical course. In studying a cosmopolitan patient population in a single large centre, the authors have cleverly eliminated many confounding factors and allowed the influence of ethnicity, per se, to be assessed.
In this context though, the use of body mass index as a measure of obesity may be too blunt a discriminator. While a BMI of 30 kg/m2 is conventionally taken to define obesity, this measure has significant limitations.2,3 The most obvious is that it does not actually measure lean versus fatty tissue.3 More particularly, the BMI or Quetelet index threshold of 30 kg/m2 may be insensitive to the presence of obesity in those of Asian origin.4 Other authors have shown that the cut off in Asians may be optimally set at a lower level (e.g. 27 kg/m2 for men, and as low as 25 kg/m2 for women).5 We wonder, were Apea VJ et al.’s observations re-examined using such thresholds, if the poor outcomes seem in Asian patients would be more fully accounted for?
We would welcome their views on this question. As the world’s most populous countries are in Asia, and the pandemic continues apace, this is perhaps a matter of the very greatest global health importance.
Pradipta Bhakta,
Brian O’Brien.
References:
1. Apea VJ, Wan YI, Dhairyawan R, et al. Ethnicity and outcomes in patients hospitalised with COVID-19 infection in East London: an observational cohort study. BMJ Open. 2021 Jan 17;11(1):e042140. doi: 10.1136/bmjopen-2020-042140. PMID: 33455936; PMCID: PMC7813387.
2. Obesity: preventing and managing the global epidemic. Report of a WHO consultation. World Health Organ Tech Rep Ser. 2000;894:i-xii, 1-253.
3. Frankenfield DC, Rowe WA, Cooney RN, Smith JS, Becker D. Limits of body mass index to detect obesity and predict body composition. Nutrition. 2001;17(1):26-30.
4. WHO Expert Consultation. Appropriate body-mass index for Asian populations and its implications for policy and intervention strategies. Lancet. 2004;363(9403):157-63.
5. Goh VH, Tain CF, Tong TY, Mok HP, Wong MT. Are BMI and other anthropometric measures appropriate as indices for obesity? A study in an Asian population. J Lipid Res. 2004;45(10):1892-8.
Goodness who would have thought it, antibiotics can save admissions and probably lives. After years of vilifying GP's for high antibiotic prescribing we now know the truth. Appropriate prescribers are probably those more accessible with higher consultation rates and higher prescribing rates. Some times the better GP's aren't always those that do as they are told.
This is a response to a comment on the paper dated 20 January 2021.
Dear Dr. Ćirković,
We thank Ćirković for welcoming our study. Before answering the points raised by Ćirković I first mention a currently-under-peer-review scientific communication between myself and the editor of the journal which published the article authored by Ćirković, that he cites in his comment [1]. The full text of this Letter to the Editor has been available as a preprint in the public domain since 18 Dec 2021 [2].
- I address below the points raised by Ćirković on his comment on our BMJ Open paper in turn:
1. TRIPOD and PROBAST checklists - searching the clinical literature using PubMed and Google Scholar identifies no studies that have used these checklists in the evaluation of vignettes studies or studies of symptom assessment applications. We reviewed the literature on symptom assessment app in our paper, and I have looked again at each of these papers, and none of them used the TRIPOD or the PROBAST checklist, nor did a recent paper in this domain from an independent academic group [3], nor did Ćirković himself in [1]. The TRIPOD checklist (transparent reporting of a multivariable prediction model for individual prognosis or diagnosis) is designed for studies into diagnostic and prognostic models for clinicians, and although an interesting suggestion, the use of such a checklist is not compulsory and not having adopted it is not a study limitation in itself. The P...
This is a response to a comment on the paper dated 20 January 2021.
Dear Dr. Ćirković,
We thank Ćirković for welcoming our study. Before answering the points raised by Ćirković I first mention a currently-under-peer-review scientific communication between myself and the editor of the journal which published the article authored by Ćirković, that he cites in his comment [1]. The full text of this Letter to the Editor has been available as a preprint in the public domain since 18 Dec 2021 [2].
- I address below the points raised by Ćirković on his comment on our BMJ Open paper in turn:
1. TRIPOD and PROBAST checklists - searching the clinical literature using PubMed and Google Scholar identifies no studies that have used these checklists in the evaluation of vignettes studies or studies of symptom assessment applications. We reviewed the literature on symptom assessment app in our paper, and I have looked again at each of these papers, and none of them used the TRIPOD or the PROBAST checklist, nor did a recent paper in this domain from an independent academic group [3], nor did Ćirković himself in [1]. The TRIPOD checklist (transparent reporting of a multivariable prediction model for individual prognosis or diagnosis) is designed for studies into diagnostic and prognostic models for clinicians, and although an interesting suggestion, the use of such a checklist is not compulsory and not having adopted it is not a study limitation in itself. The PROBAST (Prediction model Risk Of Bias ASsessment Tool) tool, also designed for studies into diagnostic and prognostic models for clinicians, was however designed for systematic reviews, not for research studies themselves [4] - as an instrument I find it poorly suited for this purpose. Nonetheless, I have applied the PROBAST method to our study, and find the tool shows a low risk of bias for our study. Previous studies have used much smaller numbers of vignettes and show much higher risk of bias. 200 is the highest number of vignettes used in any multi-symptom assessment app evaluation. We rebut this point and do not find the suggested checklists either compulsory, required, or fairly applied to our study by Ćirković.
2. “choice of the top-3 and top-5 appearing conditions without any rating within according to the respective rank on which a condition is displayed to the user” - this is clearly a misleading statement as we also reported the top-1 and the comprehensiveness metric alongside top-3 and top-5, in every case. Taken together, consideration of top-1, top-3, top-5, comprehensiveness and relevance holistically address the respective rank on which a condition is displayed to the user. This is the approach used in all similar studies, as reviewed in our paper. We searched the literature extensively for a single metric that combines weighting the properties of all five metrics listed, and no such metric has been described except in the highly technical/mathematical artificial intelligence/machine learning literature. If a clinically meaningful and interpretable weighted metric were proposed and validated in the literature we would be very interested in also including it in future studies. We rebut this point raised by Ćirković.
3. Figure 5 colouring - “overconservative advice is in some way “less a problem” than the underconservative one. This may be true from a manufacturer’s standpoint regarding liability issues towards potential users”. The opinions expressed on colour shading of figures by Ćirković are rather subjective. The colour shading chosen is in line with the safety metric used in the paper and we maintain that patient safety is paramount. The Hippocratic ‘first do no harm’ principle dictates that indeed overconservative advice is less of a problem than underconservative advice. This is not for ‘liability issues’ but due to the underlying fundamental importance of patient safety, that is enshrined in the regulatory environment and the principles any responsible manufacturer should and must adopt. Of course too-conservative advice is undesirable, and matching is optimal, but as symptom apps iteratively improve, our position is that this must be through an approach of maintaining sufficient safety whilst making stepwise reduction in too-conservtive advice. Also, note that in physician triage it has been demonstrated that human doctors are often relatively cautious and that a degree of over-cautiousness is also appropriate for safety (but of course a balanced degree): "Pre-hospital triage of emergency patients is necessarily an inexact process and some degree of overtriage must generally be accepted “ [5,6]. We provide the data to allow readers to draw their own conclusions about what constitutes the best triage approach for a specific use, as described in point (4) below.
4. “the choice to merely evaluate “safe advice” as opposed to “correct advice” or similar may not be enough to correctly assess the true value of the apps for patients, healthcare providers or public health in general.” - this point is misleading and false. Safe advice was not evaluated in isolation. Full data on exactly matching advice (i.e. “correct advice”) and for every level of matching/degree-of-not-matching was provided for every app and GP evaluated. This was provided in triplicate in the paper: (i) by means of the bar-chart of each advice grouping in Figure. 5; (ii) by fully detailed complete confusion matrices in Figure 6; and, (iii) the complete data, for all vignettes and for the subgroups of vignettes types in Supplementary Table 11. We believe this addresses the point raised by Ćirković.
5. Although the comment was titled ‘Five missing limitations’, the comment jumps from ‘thirdly’ to ‘lastly’ without listing a fifth point.
References:
1 Ćirković A. Evaluation of Four Artificial Intelligence-Assisted Self-Diagnosis Apps on Three Diagnoses: Two-Year Follow-Up Study. J Med Internet Res 2020;22:e18097. doi:10.2196/18097
2 Letter to the Editor - Evaluation of Four Artificial Intelligence–Assisted Self-Diagnosis Apps on Three Diagnoses: Two-Year Follow-Up Study. JMIR Preprints. https://preprints.jmir.org/preprint/26514 (accessed 22 Jan 2021).
3 Hill MG, Sim M, Mills B. The quality of diagnosis and triage advice provided by free online symptom checkers and apps in Australia. The Medical Journal of Australia 2020;212:514–9. doi:10.5694/mja2.50600
4 Wolff RF, Moons KGM, Riley RD, et al. PROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies. Ann Intern Med 2019;170:51–8. doi:10.7326/M18-1376
5 Rørtveit S, Meland E, Hunskaar S. Changes of triage by GPs during the course of prehospital emergency situations in a Norwegian rural community. Scand J Trauma Resusc Emerg Med 2013;21:89. doi:10.1186/1757-7241-21-89
6 Lossius HM, Rehn M, Tjosevik KE, et al. Calculating trauma triage precision: effects of different definitions of major trauma. J Trauma Manag Outcomes 2012;6:9. doi:10.1186/1752-2897-6-9
Yours faithfully
Dr. Stephen Gilbert Ph.D (corresponding author, response discussed with other authors)
Clinical Evaluation Director
Ada Health GmbH
Karl-Liebknecht-Str. 1
10178 Berlin, DE
The study is a welcome addition to the yet sparse database of evaluations of self-diagnosis tools that, unlike expert systems, are to be put into patients’ hands directly. However, there are some limitations missing in the Discussions section that I’d like to add. Firstly, as these self-diagnosis tools can have a significant impact on public health [1], it will be crucial for regulating authorities and lawmakers to create a framework that correctly distributes the liabilities on the participants in the system and ensures a positive ratio of benefit vs risk; this will only be possible if there is enough robust information available on the respective systems/programs/apps to thoroughly assess them. As this novel field is just unfolding, there are plenty of obstacles on the way to a truly objective assessment of AI-driven software, as Nagendran et al recently demonstrated here in the BMJ. [2] They used, among others, the TRIPOD and PROBAST checklists for evaluating the validities of respective studies. A quick check on this study with both would give a result of a TRIPOD adherence of 18 of 24 relevant checklist items and a high risk of bias when using the PROBAST checklist. The release of an updated TRIPOD checklist with a focus on AI-driven software has also been announced and will then be relevant for the evaluation of this study. [3] Thus, incomplete adherence to the present and unknown adherence to the presumably relevant future TRIPOD statements and a high risk of bias a...
The study is a welcome addition to the yet sparse database of evaluations of self-diagnosis tools that, unlike expert systems, are to be put into patients’ hands directly. However, there are some limitations missing in the Discussions section that I’d like to add. Firstly, as these self-diagnosis tools can have a significant impact on public health [1], it will be crucial for regulating authorities and lawmakers to create a framework that correctly distributes the liabilities on the participants in the system and ensures a positive ratio of benefit vs risk; this will only be possible if there is enough robust information available on the respective systems/programs/apps to thoroughly assess them. As this novel field is just unfolding, there are plenty of obstacles on the way to a truly objective assessment of AI-driven software, as Nagendran et al recently demonstrated here in the BMJ. [2] They used, among others, the TRIPOD and PROBAST checklists for evaluating the validities of respective studies. A quick check on this study with both would give a result of a TRIPOD adherence of 18 of 24 relevant checklist items and a high risk of bias when using the PROBAST checklist. The release of an updated TRIPOD checklist with a focus on AI-driven software has also been announced and will then be relevant for the evaluation of this study. [3] Thus, incomplete adherence to the present and unknown adherence to the presumably relevant future TRIPOD statements and a high risk of bias according to PROBAST should be added to its limitations. Secondly, the
choice of the top-3 and top-5 appearing conditions – without any rating within according to the respective rank on which a condition is displayed to the user – for comparison with real-life GPs appears deliberate, insufficiently discerning and questionable from a physician’s point of view: When a real-life patient is presented with a range of possible diagnoses without any ranking order that may even contradict each other, this may be confusing and unsettling and certainly would not speak in favor of the diagnosing physician – it does not meet expectations for medical advice. Also, it is clear that any expansion on the number of different outputs included in the evaluation and comparison will inevitably lead to more correct diagnoses, but worsens the discrimination between the apps and GPs. Thus, using the top-3 and top-5 diagnoses for assessing the apps makes this evaluation highly susceptible to overestimating the apps’ performances. The potential confusion for users of the apps was also a point I noted on a small investigation on various apps, some of which are also featured here, in a recent study. [4] Thirdly, in figure 5, overconservative advice is colored in a lighter hue of the same color as the “exactly-matched” and the “overconservative but suitable” categories, whereas anything below these is shown in complementary colors. This subtly suggests to a reader that the overconservative advice is in some way “less a problem” than the underconservative one. This may be true from a manufacturer’s standpoint regarding liability issues towards potential users, but for any other participant it is detrimental – both patients and the healthcare system participants (doctors, nurses, facilities) would suffer from the additional work created by widely exaggerated advices. For an oversimplified example, a most rudimentary “safe advice” would thereby consist of an automaton recommending to each inquirer to go seek emergency care. Thus, such advice should be treated and evaluated to be as undesired an outcome as the underconservative advices. Lastly, the former point also means that the choice to merely evaluate “safe advice” as opposed to “correct advice” or similar may not be enough to correctly assess the true value of the apps for patients, healthcare providers or public health in general.
1 Ćirković A. AI in Self-Diagnosis - History, Theoretical Foundations, Potentials and Current Status. ResearchGate. doi:10.13140/RG.2.2.33461.83684
2 Nagendran M, Chen Y, Lovejoy CA, et al. Artificial intelligence versus clinicians: systematic review of design, reporting standards, and claims of deep learning studies. BMJ 2020;368:m689. doi:10/ggscz3
3 Collins GS, Moons KGM. Reporting of artificial intelligence prediction models. The Lancet 2019;393:1577–9. doi:10/ggjntf
4 Ćirković A. Evaluation of Four Artificial Intelligence–Assisted Self-Diagnosis Apps on Three Diagnoses: Two-Year Follow-Up Study. Journal of Medical Internet Research 2020;22:e18097. doi:10/ghtknd
This is a response to a comment on the paper dated 7 January 2021.
Dear Oscar Garcia-Esquirol, Cofounder and Chief Medical Director at Mediktor, one of the symptom checkers evaluated in the study,
Thank you for your comment on this paper.
Your letter addressed strong criticism at the paper [1] with respect to bias. I will answer these points in turn below. Our study included a rigorous design process conducted by experienced clinical researchers, data scientists and health policy experts, with the methodology and analysis peer-reviewed by independent and experienced primary care physicians and medical informatics experts at universities in the UK, and in Brown University in the US. To ensure a fair comparison, our team used a large number of clinical vignettes, which were generated from a mix of real patient experiences gleaned from the UK’s NHS 111 telephone triage service and from those generated from the many years’ combined experience of the research team. The gold-standard triage level used in the study was set independently of vignette creation, vignette review and vignette diagnosis gold-standard setting - this was done by a separate panel of three experienced primary care practitioners using a tie-breaker panel method based on the matching process set out by [2].
A strength of this paper is that it not only compares a range of symptom assessment apps to each other, but also compares their performances to that of practicing GPs. While the...
This is a response to a comment on the paper dated 7 January 2021.
Dear Oscar Garcia-Esquirol, Cofounder and Chief Medical Director at Mediktor, one of the symptom checkers evaluated in the study,
Thank you for your comment on this paper.
Your letter addressed strong criticism at the paper [1] with respect to bias. I will answer these points in turn below. Our study included a rigorous design process conducted by experienced clinical researchers, data scientists and health policy experts, with the methodology and analysis peer-reviewed by independent and experienced primary care physicians and medical informatics experts at universities in the UK, and in Brown University in the US. To ensure a fair comparison, our team used a large number of clinical vignettes, which were generated from a mix of real patient experiences gleaned from the UK’s NHS 111 telephone triage service and from those generated from the many years’ combined experience of the research team. The gold-standard triage level used in the study was set independently of vignette creation, vignette review and vignette diagnosis gold-standard setting - this was done by a separate panel of three experienced primary care practitioners using a tie-breaker panel method based on the matching process set out by [2].
A strength of this paper is that it not only compares a range of symptom assessment apps to each other, but also compares their performances to that of practicing GPs. While the Ada app performed most strongly of any of the symptom assessment apps in condition-suggestion, it is of note, that it was not the single strongest performing app on the triage safety metric, in which the Symptomate app achieved 97.8%; Ada achieved 97.0%; and Babylon, also performed very strongly achieved 95.1%. The Mediktor app had a safety performance on this study metric
of 87.3% that was outside 2 S.D. of the performance of the evaluated GPs (Mediktor: 87.3% (p=1.3×10-3, statistically significant difference to GP performance). This is relevant to note in this response to your comment, as the performance of Mediktor in the study may be a factor in the nature of your criticism of the paper (the author of the comment is the Chief Medical Officer and co-founder of Mediktor). The analysis was carried out by the independent vignette entry GPs in a manner to be fair to all apps. The study was not designed to discriminate against Mediktor in the urgency advice safety measurement any more than it was designed so that Symptomate or Babylon would do well in the safety measurement.
With respect to criticism of the number of levels of advice. A universal scale of comparison was needed as the urgency advice of apps was to be compared. Six advice levels were used as a universal scale for comparison of the symptom assessment app advice and this was also used as a scale for the tested GPs to provide their advice on. The scale was developed based on independent input and advice. It is not the native urgency advice scale of the Ada app (which provides advice on 8 levels), or any other evaluated app. As you point out, some apps provide advice on five levels. The 6-point scale is a fair choice as most of the apps evaluated provided at least 6 levels of advice. The mapping of 5 levels of advice to a 6-point scale could not have resulted in any disadvantage of any app in the safety metric measurement reported, as this was determined, as stated in the paper, as at “the gold standard level, more conservative, or no more than one level less conservative)”.
The comment states that the ‘observers’ ‘were people who have worked at Ada, or colleagues of those who have worked for this company’ It is not entirely clear how the term ‘observers’ is used in your comment. For the following four groups: (i) the vignettes gold standard condition-suggestion’s review panel; (ii) the vignettes gold standard triage setting panel; (iii) the app-entry-doctors; (iv) the condition matching panel: all were independent, and none ‘were people who have worked at Ada, or colleagues of those who have worked for this company’, some were recruited from wider professional networks - e.g. members of the same professional society or online clinical forum etc, but none of these ‘observers’ had any connection that would result in the ‘Ada variable group becom[ing] more interesting to the researchers than the other variables to be evaluated [, when] measuring the ADA variable, researchers may have more interest in attaining information as they benefit from it in regard to the rest.’ With respect to the condition matching panel, they were blinded to the source of the condition-suggestions, having no information about whether the condition suggestions came from a tested-GP or from an app, or which app they came from. The final group of GPs who participated in the study were the 7 tested-GPs, whose own performance in ‘diagnosing’ the vignettes was assessed. Of these, four had previously worked for the Ada Health telehealth service Dr Chat but were no longer employees at Ada and the other three had never ‘worked at Ada’, and were not ‘colleagues of those who have worked for this company’.
The comment repeatedly makes the point that there were biases related to the researcher, who acted as a patient ‘when GPs are evaluated by phone’, including due to the Hawthorne effect. These points are in their totality incorrect, as the commenter has misinterpreted the study methodology on this point. We designed the study to ensure that all apps and all tested-GPs were compared to the gold standard diagnosis independently. There was no comparison between the tested-GP answers and the tested apps, for Ada or for any other evaluated app. Each tested-GP was compared to the gold-standard, via the matching panel. Likewise, each evaluated app was compared to the gold-standard via the matching panel. The telephone interview was solely for the evaluation of the tested-GPs. There could have been no possibility of biasing the tested-GPs answers towards Ada (or any of the other apps) condition-suggestions, as there was no comparison to the app’s condition-suggestions. There was no possibility of bias towards Ada here. The procedure is clearly explained in the original manuscript, particularly in figure 1, and along with the reiteration here, this point is fully rebutted.
-- Representativeness of the coverage of the vignettes to real population disease prevalence. --
This point of criticism in the comment is already addressed in full within the study and its technical appendix 'Appendix S1: Comparison of the vignettes to UK health statistics'. This appendix shows the degree of match to a fair prevalence data set - the Quality and Outcomes Framework (QOF) for the NHS in England (Note - there is no perfect comparison set which maps real symptom assessment users). The appendix shows there is matching on broad terms across the prevalence of conditions, but also shows that this match is not perfect in all respects. This is also addressed in the study main manuscript.
-- Suggested bias associated with the manner of case entry in apps --
The comments incorrectly states that the study failed to allow for ‘in Mediktor’s case, [that] it is possible to enter more than one symptom, sign or information referring to the patient at the initial stage of consultation’. The study did not stipulate to the app-entry-doctors how they should use each app. They familiarised themselves with each app through the app itself and, through instructions for use for the app (when these were clearly made available within the app interface) In other words, the method of entry of a vignette into an app was not predetermined at all by the study investigators, so as not to bias the entry. Each app-entry-doctor chose their own entry, as guided by the app, and as guided by their role as a user. Each vignette had a listed primary complaint, and the app-entry-doctors were always instructed to enter this first. After this, they were guided by the app interfaces (and app instructions when readily available through the interface) as to whether to enter other symptoms simultaneously or later, to simulate a real patient learning to use the app.
I hope the clarifications provided here aid the understanding as some of the points above related to simple misinterpretation. Whilst it is impossible to remove all biases in a study of this sort, the design sought to use separate groups of doctors at each stage and blind them to the origin of the data where possible.
-- The role of app manufacturers/developers in studies on medical device apps --
Another aspect of Garcia-Esquirol‘s comment is, however, clearly misleading. The comment misrepresents involvement of an app developer/manufacturer in a study as being proof of bias in the study. As the author of the comment is the CMO and co-founder of an medical device app manufacturer, he must surely be fully aware that involvement of manufacturers in studies, both in the planning and in the funding, is required under the EU medical device regulation framework [3]. As is commonplace and fully acceptable, Mediktor (cofounded by the author of the comment) has been affiliated to studies researching their products [4] - see the conflicting interests statement of this paper for the connection between the authors of the study and the Mediktor company. The landscape of clinical studies of individual medical devices is predominated by manufacturer funded studies, and this very often involves some manufacturer personnel or link, as was the case for Mediktor in [4]. Manufacturers have a duty to assess the state-of-the-art and comparison studies, often done through non-inferiority studies, which are generally funded by one manufacturer, and compare their product to the product of a competitor. These are commonplace, standard, and often expected for implanted medical devices and for pharmaceuticals [5]. The comments relating to the involvement of Ada staff in the study and in the funding of the study create a misleading false premise, as neither funding a study nor involvement in a study equate to ‘budgetary biases’ and all involvement in the study was correctly stated in the Contributors, Funding and Competing Interests sections of the paper - as it is done for any manufacturer sponsor study. Not all of the scientific readership of your comment will know the post market clinical evaluation/study requirements on app manufactures [3], and, therefore, these remarks are either wilfully misleading through misrepresenting the standard approach to study funding for medical devices (and pharmaceutical products), or, misrepresenting or being unaware of Mediktor's own stated conflicting interests in their own published studies.
With respect to your point, that there is a need for further studies, we are in full agreement on this and for this reason addressed it in the paper. We stated 'To address the effect of patients entering data directly into an app about their own acute conditions, an observational investigation is currently underway in an acute clinical setting in the USA', and this is being carried out in a completely independently funded study, by investigators who include the non-Ada affiliated coauthor of this study.
In summary, of the critical points made in the comment, there is only one that is of any validity, which relates to whether an optimal universal reference mapping for all apps (and GPs) would be to a 5-level triage scale or a 6-level triage scale. This is open to debate, but in the end, a single universal scale is required in a comparison study, and we defend this 6 point scale as being as fair a choice overall as a 5 point scale. The remaining points are incorrect and the results of misunderstanding, or misrepresentations of stated involvement as proof of bias.
The commenter asks whether the low safety score of Mediktor might relate to bias in the study conduct or design. Upon review, in a number of scenarios Mediktor provided the correct condition-suggestions, but provided the user only a "medium" advice level for vignettes in which call-ambulance was indicated by all/nearly-all other apps, by all/nearly-all of the tested-GPs and by the vignette gold standard. The commenter may contact us directly if he would like additional information regarding these examples (within the terms of the paper's Data Availability Statement).
Overall, this study has been well received in the scientific community (see the link to the wider discussion of the paper from this BMJ Open page).
1 Gilbert S, Mehl A, Baluch A, et al. How accurate are digital symptom assessment apps for suggesting conditions and urgency advice? A clinical vignettes comparison to GPs. BMJ Open 2020;10:e040269. doi:10.1136/bmjopen-2020-040269
2 Semigran HL, Linder JA, Gidengil C, et al. Evaluation of symptom checkers for self diagnosis and triage: audit study. BMJ 2015;351. doi:10.1136/bmj.h3480
3 Regulation (EU) 2017/745 of the European Parliament and of the Council of 5 April 2017 on medical devices, amending Directive 2001/83/EC, Regulation (EC) No 178/2002 and Regulation (EC) No 1223/2009 and repealing Council Directives 90/385/EEC and 93/42/EEC (Text with EEA relevance. ). 2017. http://data.europa.eu/eli/reg/2017/745/oj/eng (accessed 11 Jan 2021).
4 Moreno Barriga E, Pueyo Ferrer I, Sánchez Sánchez M, et al. [A new artificial intelligence tool for assessing symptoms in patients seeking emergency department care: the Mediktor application]. Emergencias 2017;29:391–6.
5 Schumi J, Wittes JT. Through the looking glass: understanding non-inferiority. Trials 2011;12:106. doi:10.1186/1745-6215-12-106
Yours faithfully
Dr. Stephen Gilbert (corresponding author, response discussed with other authors)
Clinical Evaluation Director
Ada Health GmbH
Karl-Liebknecht-Str. 1
10178 Berlin, DE
Conflict of Interest:
Employee of Ada Health GmbH.
We read with interest the assessment tool given the acronym PAGE: Paediatric Admission Guidance in the Emergency Department.
bmjopen-2020-043864.
This PAGE study is of particular importance in emergency departments
in a hospital ( for instance in Oman ) where pediatrics is not amongst a hospital specialities.
Further development of PAGE may answer a recent call to improve the Emergency Department (ED) response to children injuries in Oman.
doi: 10.1136/bmjpo-2018-000310
A recent retrospective review of ED trauma registers in Oman also found, first, high children injuries incidence of falls, home injuries and burns.
Second, suggests future work for age-targeted interventions to better respond to children injury events .
Dear Editors,
We would like to inform you and the readers of BMJ open that we plan to slightly enlarge the enrollment of our trial "Randomised, non-inferiority, controlled procedural outcomes TrIal comParing reverse T And Protrusion versus double-kissing and crush stenting: protocol of the TIP TAP I randomised trial", which was published in BMJ Open. 2020; 10(6): e034264. Our original intention was to recruit 50 patients. Unfortunately, the OCT data pertaining the primary endpoint of the study could not be assessed in two of the patients. We therefore plan to recruit 2 additional patients following the same procedures. The analysis population of the study will therefore maintain its previously planned size of 50, the safety and IIT populations will however include all 52 patients.
Safety outcomes have been and will be regularly reviewed by three cardiologists who are not involved in any procedure of the study.
Sincerely,
Tommaso Gori
"We however found that the main determinant was the total number of international arrivals in the country (2018 figures), signifying transmission of the infection through travel. Although the data were from 2018, there is no reason to believe that international travel figures between countries would be different in early 2020."
The study period from December 2019 to June 2020 covered a period in which international passenger capacity reduced by 8% in February 2020, by 46% in March 2020 and by 90% in April 2020 year-on-year compared to 2019 (ICAO data). It is therefore demonstrably incorrect to assert that there is no reason to believe travel figures would be different.
Dear Dr Bhakta and Dr O'Brien,
We are grateful for the very valid insights. Our intention was not to explore BMI as a COVID risk factor due to reasons such as incomplete data and measurement accuracy, but to account for it as a potentially confounding variable in the relationship between ethnicity and outcome. Evidence regarding ethnic differences in BMI remains sparse and historic reports may not accurately reflect cohorts represented in contemporary publications. This is nevertheless an important subject to address for future research.
Yize I Wan,
John R Prowle
We read with interest the MaPs in ENL study protocol (de Barros et al, BMJ open 2020).[1] The availability of an alternative drug for cases of chronic or recurrent type 2 lepra reaction with steroid toxicity or unresponsiveness is the need of the hour and although drugs like thalidomide, cyclosporine etc. are prescribed in such patients but cost, adverse effect or non-availability are the major deterrents in their use. The researchers must be complimented for their bold decision to choose methotrexate based only on case reports or a short series of patients (15 patients in total were given methotrexate for type 2 reactions).[2] This is a well-designed protocol aimed at possibly adding another drug to our therapeutic armamentarium for the crippling erythema nodosum leprosum (ENL) reactions. If found useful, methotrexate has the advantage of being a cheap, easy to administer and readily available drug.
The study proposes to randomize patients with acute, chronic or recurrent ENL into two groups, receiving 20 weeks of prednisolone with either 48 weeks of methotrexate or placebo. While use of methotrexate is justified in patients with chronic and recurrent ENL, , same might not be applicable to cases of acute ENL. Acute ENL, by definition, is an episode that lasts lesser than 24 weeks,[3] as has been defined in the study protocol as well. Most of these cases with acute episode respond well to corticosteroids, clinical response has been observed as early as in 2 week...
Show MoreDear Editor,
Show MoreApea VJ et al. should be congratulated on elucidating the interplay between overlapping factors that influence patient outcomes from COVID-19. From the first reports about this disease, observers have noted that adipose tissue, perhaps through its association with diabetes and heart disease, has been linked to the severity of presentation and to subsequent clinical course. In studying a cosmopolitan patient population in a single large centre, the authors have cleverly eliminated many confounding factors and allowed the influence of ethnicity, per se, to be assessed.
In this context though, the use of body mass index as a measure of obesity may be too blunt a discriminator. While a BMI of 30 kg/m2 is conventionally taken to define obesity, this measure has significant limitations.2,3 The most obvious is that it does not actually measure lean versus fatty tissue.3 More particularly, the BMI or Quetelet index threshold of 30 kg/m2 may be insensitive to the presence of obesity in those of Asian origin.4 Other authors have shown that the cut off in Asians may be optimally set at a lower level (e.g. 27 kg/m2 for men, and as low as 25 kg/m2 for women).5 We wonder, were Apea VJ et al.’s observations re-examined using such thresholds, if the poor outcomes seem in Asian patients would be more fully accounted for?
We would welcome their views on this question. As the world’s most populous countries are in Asia, and the...
Goodness who would have thought it, antibiotics can save admissions and probably lives. After years of vilifying GP's for high antibiotic prescribing we now know the truth. Appropriate prescribers are probably those more accessible with higher consultation rates and higher prescribing rates. Some times the better GP's aren't always those that do as they are told.
This is a response to a comment on the paper dated 20 January 2021.
Dear Dr. Ćirković,
We thank Ćirković for welcoming our study. Before answering the points raised by Ćirković I first mention a currently-under-peer-review scientific communication between myself and the editor of the journal which published the article authored by Ćirković, that he cites in his comment [1]. The full text of this Letter to the Editor has been available as a preprint in the public domain since 18 Dec 2021 [2].
- I address below the points raised by Ćirković on his comment on our BMJ Open paper in turn:
1. TRIPOD and PROBAST checklists - searching the clinical literature using PubMed and Google Scholar identifies no studies that have used these checklists in the evaluation of vignettes studies or studies of symptom assessment applications. We reviewed the literature on symptom assessment app in our paper, and I have looked again at each of these papers, and none of them used the TRIPOD or the PROBAST checklist, nor did a recent paper in this domain from an independent academic group [3], nor did Ćirković himself in [1]. The TRIPOD checklist (transparent reporting of a multivariable prediction model for individual prognosis or diagnosis) is designed for studies into diagnostic and prognostic models for clinicians, and although an interesting suggestion, the use of such a checklist is not compulsory and not having adopted it is not a study limitation in itself. The P...
Show MoreThe study is a welcome addition to the yet sparse database of evaluations of self-diagnosis tools that, unlike expert systems, are to be put into patients’ hands directly. However, there are some limitations missing in the Discussions section that I’d like to add. Firstly, as these self-diagnosis tools can have a significant impact on public health [1], it will be crucial for regulating authorities and lawmakers to create a framework that correctly distributes the liabilities on the participants in the system and ensures a positive ratio of benefit vs risk; this will only be possible if there is enough robust information available on the respective systems/programs/apps to thoroughly assess them. As this novel field is just unfolding, there are plenty of obstacles on the way to a truly objective assessment of AI-driven software, as Nagendran et al recently demonstrated here in the BMJ. [2] They used, among others, the TRIPOD and PROBAST checklists for evaluating the validities of respective studies. A quick check on this study with both would give a result of a TRIPOD adherence of 18 of 24 relevant checklist items and a high risk of bias when using the PROBAST checklist. The release of an updated TRIPOD checklist with a focus on AI-driven software has also been announced and will then be relevant for the evaluation of this study. [3] Thus, incomplete adherence to the present and unknown adherence to the presumably relevant future TRIPOD statements and a high risk of bias a...
Show MoreThis is a response to a comment on the paper dated 7 January 2021.
Dear Oscar Garcia-Esquirol, Cofounder and Chief Medical Director at Mediktor, one of the symptom checkers evaluated in the study,
Thank you for your comment on this paper.
Your letter addressed strong criticism at the paper [1] with respect to bias. I will answer these points in turn below. Our study included a rigorous design process conducted by experienced clinical researchers, data scientists and health policy experts, with the methodology and analysis peer-reviewed by independent and experienced primary care physicians and medical informatics experts at universities in the UK, and in Brown University in the US. To ensure a fair comparison, our team used a large number of clinical vignettes, which were generated from a mix of real patient experiences gleaned from the UK’s NHS 111 telephone triage service and from those generated from the many years’ combined experience of the research team. The gold-standard triage level used in the study was set independently of vignette creation, vignette review and vignette diagnosis gold-standard setting - this was done by a separate panel of three experienced primary care practitioners using a tie-breaker panel method based on the matching process set out by [2].
A strength of this paper is that it not only compares a range of symptom assessment apps to each other, but also compares their performances to that of practicing GPs. While the...
Show MorePages