Article Text
PDF
Abstract
Objectives To conduct a systematic review and synthesise qualitative research of electronic risk assessment tools (eRATs) in primary care, examining how they affect the communication and understanding of diagnostic risk and uncertainty. eRATs are computer-based algorithms designed to help clinicians avoid missing important diagnoses, pick up possible symptoms early and facilitate shared decision-making.
Design Systematic search, using predefined criteria of the published literature and synthesis of the qualitative data, using Thematic Synthesis. Database searches on 27 November 2019 were of MEDLINE, Embase, CINAHL and Web of Science, and a secondary search of the references of included articles. Included studies were those involving electronic risk assessment or decision support, pertaining to diagnosis in primary care, where qualitative data were presented. Non-empirical studies and non-English language studies were excluded. 5971 unique studies were identified of which 441 underwent full-text review. 26 studies were included for data extraction. A further two were found from citation searches. Quality appraisal was via the CASP (Critical Appraisal Skills Program) tool. Data extraction was via line by line coding. A thematic synthesis was performed.
Setting Primary care.
Results eRATs included differential diagnosis suggestion tools, tools which produce a future risk of disease development or recurrence or calculate a risk of current undiagnosed disease. Analytical themes were developed to describe separate aspects of the clinical consultation where risk and uncertainty are both central and altered via the use of an eRAT: ‘Novel risk’, ‘Risk refinement’, ‘Autonomy’, ‘Communication’, ‘Fear’ and ‘Mistrust’.
Conclusion eRATs may improve the understanding and communication of risk in the primary care consultation. The themes of ‘Fear’ and ‘Mistrust’ could represent potential challenges with eRATs.
Trial registration number CRD219446.
- primary care
- qualitative research
- risk management
Data availability statement
Data sharing not applicable as no data sets generated and/or analysed for this study.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
This is the first review to address the impact of electronic risk assessment tools in the primary care consultation.
There is inconsistency in nomenclature surrounding these tools.
There is a lack of data from patients in the primary studies.
There are likely biases in the original studies.
Introduction
Diagnosis in primary care is complex and challenging. The main input, a patient’s history and examination, can be variable, imperfect and subjective. Pre-test probability is lower than other healthcare settings.1 2 General Practitioners (GPs) are able to process and communicate risk and accept uncertainty in diagnosis3: doing this in a way which allows a shared understanding of medicine with a patient is one of the seminal challenges of primary care.4
Clinical decision support systems are designed to ‘enhance medical decisions with targeted clinical knowledge, patient information, and other health information’.5 ‘Electronic risk assessment tools’ (hereafter referred to as eRATs) are decision support systems which help with diagnosis. They are software where information about a patient is processed electronically to produce an output pertinent to a diagnosis, either immediately or in the future. The output can be a quantitative risk of a specific diagnosis, or a more qualitative output: prompting the clinician to consider a particular diagnosis or list of possible diagnoses. eRATs can enable extraction of data previously inputted into primary care IT systems, and can prompt users to input missing data. They can incorporate ‘pop ups’ during a consultation to avoid missing important diagnoses.6 7 They have a range of stated aims: making diagnosis more evidence based and objective; avoiding missed diagnoses; or facilitating shared decision-making.
eRATs tend to target diagnoses which are both serious and benefit from early intervention,6 7 an obvious example being cancer. To achieve adequate sensitivity these algorithms tend towards a low positive predictive value.6 7 As eRATs use more complex calculations, invisible to the clinician and insert themselves into the clinician consultation more proactively, they may come to represent a ‘third actor’ in the consultation, alongside the patient and clinician.8 They have the potential to change the consultation dynamic and diagnostic thinking process. Although there have been systematic reviews conducted on the effectiveness of eRATs,9 10 there have been no systematic reviews using qualitative methods examining risk and uncertainty in their use.
Diagnosis in primary care tends to be the outcome of clinical reasoning, which can be considered using the theories of Situativity11: that diagnosis will depend on other actors (as well as the clinician) present during the consultation, and their interaction with multiple environmental inputs; in our case eRATs. Within this clinical situation we are interested in the communication of risk, which is the best proxy we have of the understanding of risk. The Theory of Communicative Rationality12 may help our understanding—communication can be emancipative: an attempt to achieve mutual understanding (as in shared decision-making); or strategic, where the intention of the communication is to achieve a particular aim of an individual, such as calculative manipulation of others.
Aim
This study aims to synthesise qualitative research on eRATs in the primary care consultation, examining how they affect the communication and understanding of the risk of a particular disease, and uncertainty when used making diagnoses.
Methods
Study selection
Searches of MEDLINE, Embase, CINAHL and Web of Science were performed. The search strategy13 targeted studies involving a primary care population, where risk assessment or decision support tools are used, and qualitative data are reported regarding risk or uncertainty. Non-empirical studies and non-English language studies were excluded, as well as studies outside of primary care or where qualitative data were not presented. Using Covidence software,14 three authors (AB, JF-W and BD) independently screened titles and abstracts and then full text for relevance. Disputes were resolved on discussion between authors. The initial 5971 unique studies that were identified reduced to 441 on title/abstract review. These underwent full-text review, which yielded 26 individual studies and 1 systematic review9 (figure 1). These were hand searched for further citations, which yielded two additional studies. Key eRATs embedded into System1, a mainstream UK IT system, were identified and the developers emailed (where qualitative studies had not already been identified) to see if there were unpublished data available. This did not identify further studies. Thus 28 independent studies were included for data extraction (online supplemental table 1).
Supplemental material
Flow diagram for systematic searches of databases and registers.
Quality appraisal data extraction
The 28 studies were assessed for quality (AB, ES and SGD) using the CASP (Critical Appraisal Skills Program) critical appraisal checklist.15 The CASP checklist was designed to be used as educational pedagogical tool and not as a scoring tool for systematic reviews. A point was assigned if the study met 1 of the 10 statements making up the CASP checklist. Discrepancies were resolved on discussion between the authors (online supplemental table 2). None were excluded from analysis due to low quality.16 Data extraction was performed by AB. Quotes and researcher statements regarding risk and uncertainty were extracted electronically and coded into NVivo software.17
Thematic synthesis
Thematic synthesis was chosen as it is based in scientific realism18: the purpose of this review is to examine how clinicians and patients understanding of risk and uncertainty change with the use of eRATs. For the results to be broadly applicable, the underlying assumption is that the understanding of risk, and this communication is consistent, or consistent within a range.
Descriptive codes were developed by AB. Codes were described in the initial studies coded, and subsequent studies coded into these concepts, with new descriptive codes being created, refined or ordered hierarchically when deemed necessary. These descriptive codes were discussed with other authors (SGD, ES, RC and MT). Thematic synthesis involves the ‘translation of concepts from one study to another’.19 The 10 studies with analytical findings pertaining to risk or uncertainty were examined. The developed descriptive codes were applied to each to explore new coding approaches. The descriptive codes were then developed inductively into analytical themes describing separate aspects of the clinical consultation where risk and uncertainty is both central to, and altered via, the use of an eRAT. Selective quotes are used to illustrate the richness of data and to provide support for these themes.
Patient and public involvement
Patients and the public were not involved in the review.
Research ethics approval
This study did not receive or require ethics approval, as it does not involve human or animal participants.
Results
Three conceptually different forms of eRATs were identified: those which produced a risk of an immediate, or ‘clinical’ diagnosis, (eg, the risk of a current bacterial tonsillitis); those which produced a risk of a future or ‘prognostic’ diagnosis, (eg, the risk of cardiovascular disease in the next decade); and those which produced a list of potential alternative diagnoses, or ‘differential diagnosis generators’. This is an important distinction because it gets to the heart of what diagnosis is: separating the affected from the non-affected. This means defining illness, and the concept of current illness is distinct to the threat of future illness.20 It has further implications in that both clinician and patient may be able to change, and perhaps have responsibility for, ‘prognostic’ risk. It was decided that the three conceptual eRATs had aspects which were similar enough to attempt a synthesis: to see if there were findings which applied across the forms of eRATs, and that could help answer the research question.21
Except for a single study,22 those concerning eRATs for clinical diagnosis collected data from clinicians rather than patients. Studies of prognostic eRATs were more orientated towards the communication of risk to patients. The studies examining differential diagnosis generators had data from both clinicians and patients.
Included studies had publication dates ranging from 2003 to 2019.22–49 There was a broad range of targeted diagnoses, as well as data collection and analytical methods. There was also a range of nomenclature used to describe the eRAT being used. Details are available in online supplemental table 1. Only 10 of the studies reported themes or findings pertaining to risk or uncertainty in their analysis.27 33–35 37–40 42 49
Analytical themes
Six analytical themes are presented below with supportive quotes (further supportive quotes are presented in online supplemental table 3). Quotes are from clinicians, unless specified otherwise.
‘Novel risk’: A new risk is introduced into the consultation, either as an unconsidered diagnosis or an unexpected high risk.24 26 28 33 46 47 It brings the risk to the forefront of the consultation and ensures that it is adequately considered.
It’s just electronic highlight, makes you think even if you immediately dismiss it, at least that millisecond you’ve thought about it, and I think that is going to be useful at some point, but for how many people I don’t know.26
I think it’s important to be informed and to get there as early as possible. To go and see the GP, and if there’s a tool perhaps, you know, to help me do this, that may be of interest. Patient Comment22
The eRAT can raise the risk of alternative diagnoses:
Yes, I must admit ovarian didn’t come so high up, I had already fixated on the bowel stuff, which maybe I shouldn’t have. This really said hey, consider ovarian as well. So again that was useful, yes.24
This theme arose mainly from studies regarding clinical diagnosis, and differential diagnosis generators, and arose mainly from data regarding clinicians views.
‘Risk refinement’: In this theme, the risk is processed into a more accurate assessment. It was generated from studies concerning eRATs for prognostic and clinical diagnosis, and mainly from clinicians.
It enables clinicians: ‘to take complex, competing information and bring it together’.36
This refinement of risk serves several purposes. It can give an objective single point estimate of risk.28 36 it can take complex risk factors and combine them into a single risk output,37 and it can orientate the clinician towards guidelines and clinical pathways, triggering them to add missing information to the eRAT,39 suggesting the next diagnostic tests which are appropriate24 to reduce uncertainty and improve diagnostic accuracy.
‘Communication’: eRATs can assist clinicians to communicate risk to patients, and in patient-facing eRATs, to help patients communicate with their clinicians.22 They can provide a focus point for a consultation.39 46 Clinicians can use the eRAT output to justify their next action, be this referral or further testing particularly in eRATs for clinical diagnosis.24 45 Where the risk output is low they can justify inaction.24 31 The eRAT acts as a tool to communicate this:
OK, so our little risk calculator here is recommending that we would swab you for strep throat, and I agree with that.45
I think FRAX [is] useful because [it] can cut down on unnecessary investigations.46
Prognostic eRATs, can emphasise the level of risk, with a preference for colour displays or other qualitative methods of showing risk.39 46 eRAT outputs can be used as a motivator to take treatments or change behaviour.38
…you’re possibly changing what you can change. Patient comment28
‘Autonomy’: The eRAT output is a personalised risk pertaining to a patient’s health, enabling them to make informed decisions.28 37 The eRAT was an enabler of this as it provided a personalised risk and prompted patients to think about it.
Anything that gets people to have a think about their health is a good thing, anything at all.31
The fact that the eRAT has an output which not generic, but personalised to the patient’s data, was perceived to be an important component of this benefit among clinicians and patients across the range of eRATs examined in this review.
‘Fear’: The eRAT could create anxiety or disquiet in both patient and clinician. This can occur at several levels. The first is the fear of a serious diagnosis from the eRAT. Patients have understandable concern about their own health and welfare34:
If I knew I was from a high risk category, and I’m pretty sure I’m not, I reckon I would find that so stressful. I would just worry every time there was a pimple, Patient comment28
Clinicians share this fear, with an overlay of concern about poor understanding of risk among patients:
And I thought, ‘Wow! Patients coming with these tummy pains,’ even if I’m thinking, ‘She’s got a 10% chance,’ to actually think, ‘Wow! Listen, you got a 38% chance of cancer.’ General public are very—I would find that confronting and the general public have a shocking record at understanding risk.24
A second fear from clinicians regarding differential diagnosis generators, and clinical diagnosis eRATs, was of the tool interrupting how they normally think about diagnosis.47 48 The eRAT could distort how they handled diagnostic risk and unwanted risk suggestions made clinicians uncertain. There was a fear that the risk outputs could lead to a loss of control of the consultation, as the uncertainties raised by the eRATs override other agendas.24 46
Finally, there were fears expressed regarding the medicolegal consequences of eRATs. Clinicians expressed concern that any eRAT output, if not acted on, could act as evidence against them26 48 and make them vulnerable to litigation. Here though, there was a negative case in the data, with one clinician reporting the eRAT was reassuring: that it acted as a ‘backstop’, reducing fear of missing a diagnosis.
So although it can cause irritation, the flip side of that is you can relax a little bit more and that you don’t have to remember absolutely everything (…) You know, how to sort the wheat from the chaff. So although people might think it’s irritating, it can be reassuring.26
‘Mistrust’: There was a theme of mistrust in the risk output of the eRAT. Some clinicians felt that the eRAT algorithm was flawed: because the trigger was set at an incorrect level of risk25 48; that it relied on a single parameter, and changes to that parameter could disproportionately change the output39; that the inputs themselves were out of date or flawed in some way; or that the outputs were outside the control of patient and clinician. There was then a suspicion that the eRAT could be either under or over-reporting a risk.
Inevitably it will come up with a diagnosis of cancer. When the patient walks through the door… quite clearly, quickly, we can see this is not a serious diagnosis.48
Although clinicians could use the eRAT output to justify not performing further investigations, this tended to be in high-risk situations where further testing was unnecessary before diagnosis and action. In situations where there was a low-risk output from the eRAT, both clinicians and patients preferred diagnostic tests over the output of an eRAT.31 49 Here, a diagnostic test would be ordered despite the eRAT suggesting a risk low enough that testing was unnecessary. The test was trusted and the eRAT discounted.
There was faith in personal judgement over the eRAT. Patients preferentially trusting their knowledge of their own bodies,28 42 clinicians trusting their clinical acumen or gut instinct.24 31 40 48 Over-ruling an eRAT with clinical acumen was seen as good medicine:
Patients don’t follow flow charts, in my experience.48
Discussion
Summary
This review found that eRATs can influence diagnosis, and the communication of its risks and uncertainties in differing ways. Situativity11 includes the idea of distributed cognition,50 where reasoning is spread between several actors or tools. An eRAT introducing ‘Novel’ risk in a consultation could be seen as an example of this, and could have the potential to overcome the cognitive errors of misdiagnosis (such as premature closure).51 Shared decision-making52 could be seen as a combination of the themes of ‘Communication’, ‘Autonomy’ and ‘Risk Refinement’, helping realise the emancipative communication as described by Habermas.12 In addition, the ‘Risk Refinement’ function of an eRAT has the potential to increase the consistency of diagnostic decision-making, which might improve adherence to guidelines and play a part in cost containment.5 These more positive themes could be undermined by those of ‘Mistrust’ and ‘Fear’, where the risk outputs of the eRAT are disregarded, or seen as something that could challenge or disturb the communication of diagnostic risk.
Strengths and limitations
This review is the first to address how eRATs affect understanding and communication of risk and uncertainty in the primary care consultation. The broad scope of this review means a range of eRATs have been synthesised. Some themes presented derive a greater part of their data from eRATs of a particular conceptual design (‘clinical’, ‘prognostic’ or ‘differential diagnosis’, as described in the results section), which could challenge their transferability across all eRATs. This review was intended to look at the wider clinician and patient interaction (in terms of understanding and communication) with eRATs and the findings are broadly transferable, which supports the analytical approach taken.
Another challenge concerns the nomenclature surrounding these tools: we used search terms to capture all studies where information was processed electronically to produce an output pertinent to a diagnosis, either immediately or in the future: here we have called them eRATs. The language used by the originators is often different and has changed over time. eRATs can also be referred to by other names, such as Diagnostic Decision Support Systems, Electronic Screening Tools and Risk Prediction Tools. The review may have missed interventions which, if examined, might be an eRAT by our definition.
There was limited data from patient’s views regarding eRATs: the majority of the qualitative data regarding risk and uncertainty came from clinicians in the review’s studies, and data from patients was less conceptually rich. If sharing diagnostic decisions with patients is to occur53 more research is needed into the patient’s perspective. There were limited findings regarding uncertainty, compared with those concerning risk. Uncertainty in medicine is an under-researched area3 54–56 thus eRATs are influencing aspects of the consultation which are already poorly understood.
Thematic synthesis includes ‘translation’ of the review findings into the primary studies’ thematic frameworks. This involves ‘the process of taking concepts from one study and recognising the same concepts in another study’.19 However, understanding and communication of risk and uncertainty was included in the analytical findings of only 10 of the primary studies, limiting this aspect of the synthesis. Single author data extraction is also a limitation, as the findings may be more prone to author bias.
Comparison with existing literature
There are studies supporting the use of alerts regarding prescribing errors,5 57 58 and the use of similar alerts to avoid missed diagnoses is a logical step. However, this introduction of novel risk into the consultation deserves a sound evidence base which is currently missing. Mistrust of decision support systems is a recognised issue59 and may be justified: there is limited evidence supporting eRATs in terms of improving survival and patient experience.12,13 The role of clinical judgement in the mistrust of eRATs is a tension which has been described in other areas of evidence-based medicine.60 61 Part of clinical judgement is a clinicians’ gut feelings. The role of gut feelings has been studied in regards to serious diagnoses62 63 although these focused on gut feelings triggering action despite a lack of (what might be considered) objective risk, rather than decisions not to act despite risk information: the latter is likely more common in the low prevalence setting of primary care, and this area warrants further study.
The role of fear in diagnosis, although well recognised,64 is not well researched. A more mature literature exists for fear appeals in public health messaging. Witte and Allen65 have examined the extended parallel process model,66 where fear appeals produce two competing responses which interfere with each other: ‘danger control’, where the response is to avoid the threat producing the fear, and ‘fear control’ where the fear itself is avoided and a health recommendation defensively resisted. Thus, fear generated by eRATs may lead to the avoidance of the eRAT, rather than any intended outcome. Fear is generally a concerning response: the systemic harms of fear on GP retention have been described.67 Although clinical uncertainty is an inevitable part of the diagnostic process, when uncertainty becomes fear, it is likely to lead to worse overall decisions and disengage clinicians from use of eRATs.
Implications for future research and practice
Future research is needed into the understanding of diagnostic risk and uncertainty, from both patient and clinician’s view points, how they differ and how eRATs influence the resulting communication. More evidence is needed examining how eRATs influence a clinician's reasoning, particularly when the eRAT challenges clinical judgement. This review may provide a framework through which a clinician and their patient may consider their own interactions with eRATs. The approach needs to be a nuanced balance: to value a clinician’s judgement; to be aware of the strengths and limitations of eRATs and what they bring to the consultation, while minimising the fear they generate for both patient and clinician.
Data availability statement
Data sharing not applicable as no data sets generated and/or analysed for this study.
Ethics statements
Patient consent for publication
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Twitter @alburns50
Contributors AB: protocol, literature search, citation and full-text screening, quality appraisal, data analysis, article and guarantor. JF-W and BD: citation and full-text screening. ES and SGD: quality appraisal. ES, RC, MT and SGD: supervision at all stages.
Funding AB is a PhD candidate in a post funded by the University of Exeter (PhD Studentship Ref 3521) in conjunction with the ERICA trial (http://www.theericatrial.co.uk), which is funded by the Dennis and Mireille Gillings Foundation, and receives support from the charities Cancer Research UK and Macmillan.
Disclaimer The authors' time (AB, SD and MT) is partly supported by the National Institute for Health and Care Research Applied Research Collaboration South West Peninsula. The views expressed in this publication are those of the author(s) and not necessarily those of the National Institute for Health and Care Research or the Department of Health and Social Care.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.