Article Text

Original research
Clinical diagnosis of seasonal influenza by physicians: a retrospective observational study
  1. Hiroki Maita1,
  2. Tadashi Kobayashi2,
  3. Takashi Akimoto2,
  4. Fumihiko Matsuoka3,
  5. Shigeki Funakoshi3,
  6. Hiroshi Osawa2,
  7. Hiroyuki Kato1,2,4
  1. 1Development of Community Healthcare, Hirosaki University Graduate School of Medicine, Hirosaki, Aomori, Japan
  2. 2Department of General Medicine, Hirosaki University School of Medicine and Hospital, Hirosaki, Aomori, Japan
  3. 3Rokkasho Center for Community and Family Medicine, Rokkasho, Aomori, Japan
  4. 4General Medicine, Hirosaki University Graduate School of Medicine, Hirosaki, Aomori, Japan
  1. Correspondence to Dr Hiroki Maita; maita{at}hirosaki-u.ac.jp

Abstract

Objective To elucidate the diagnostic accuracy of pretest probability of influenza (%) by physicians and the factors affecting the clinical diagnosis.

Design Retrospective, single-centre observational study.

Setting A community primary care clinic in Japan.

Participants The participants were recruited from a database of studies conducted during the influenza season from December 2017 to April 2019.

Primary outcome measure Sensitivity and specificity of the physician’s clinical diagnosis of influenza recorded in the medical record as pretest probability.

Results A total of 335 patients (median age, 31 years; male, 66.6%) were analysed in this study. The area under the curve (AUC) of the physician’s pretest probability was 0.77. At a cut-off value of 30%, the sensitivity and negative likelihood ratio were 92.0% (95% CI 86.7 to 95.7) and 0.19 (95% CI 0.11 to 0.33), respectively. At a cut-off value of 80%, the specificity and positive likelihood ratio were 90.8% (95% CI 85.4 to 94.6) and 4.01 (95% CI 2.41 to 6.66), respectively. The AUCs of patients who had and had not taken any medications before visiting the clinic were 0.77 (95% CI 0.69 to 0.85) and 0.78 (95% CI 0.71 to 0.84), respectively. The AUCs of patients with type A and B influenza were 0.78 (95% CI 0.72 to 0.84) and 0.76 (95% CI 0.70 to 0.82), respectively. The AUCs of vaccinated and unvaccinated patients were 0.80 (95% CI 0.72 to 0.88) and 0.76 (95% CI 0.63 to 0.89), respectively. The AUC for patients less than 12 hours after onset was 0.69 (95% CI 0.51 to 0.88), and that for patients aged younger than 6 years was 0.69 (95% CI 0.49 to 0.88).

Conclusions The physician’s pretest probability of influenza (%) may be useful for both definitive and exclusionary diagnoses within the limits of our study.

  • PRIMARY CARE
  • Infection control
  • GENERAL MEDICINE (see Internal Medicine)

Data availability statement

Data are available upon reasonable request. The data analysed in the current study were mostly included in this article. Additional data are available from the corresponding author on reasonable request.

http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

Statistics from Altmetric.com

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Strengths and limitations of this study

  • Our study adds to reports analysing the diagnostic accuracy of physicians’ quantitative clinical diagnosis of influenza, such as the pretest probability of influenza (%), and factors affecting the clinical diagnosis.

  • We confirmed that the physician’s clinical pretest probability of influenza (%) is useful for both definitive and exclusionary diagnoses.

  • Through this study, physicians were able to increase the accuracy and efficiency of influenza diagnosis in their practice.

  • Our study was undertaken in a single community primary care clinic of Japan. To confirm our findings, it would be necessary to conduct future studies with a larger number of physicians and physicians with various backgrounds.

Introduction

Seasonal influenza is a common disease estimated to affect one billion individuals worldwide annually.1 Generally, the diagnosis of influenza can be established clinically based on the epidemic situation and information from patients.2 3 However, establishing the clinical diagnosis of influenza by physicians has been reported to have low diagnostic accuracy.4–6 In Japan, the use of rapid influenza diagnostic tests (RIDTs) is the standard for seasonal influenza diagnosis.7 Clinical diagnosis can be divided into qualitative diagnosis, which is a binary ‘yes/no’ diagnosis of a disease, and quantitative diagnosis, which uses a continuous variable expressed as a percentage or other measures of the likelihood of a disease. Similar to the clinical diagnosis by physicians, the qualitative diagnostic accuracy of influenza self-diagnosis by patients and influenza diagnosis by guardians for their children was reported to be low.8 However, quantitative self-diagnosis and guardians’ diagnoses have been reported to be useful for influenza diagnosis.9–11 There are no reports analysing the diagnostic accuracy of physicians’ quantitative clinical diagnosis of influenza, such as the pretest probability of influenza (%). Pretest probability is the estimated probability of a disease before the test result is known; it is usually based on the physician’s personal experience, local prevalence data or published reports.12

Various factors may affect the clinical diagnosis of influenza. It has been reported that elderly patients are less likely to develop fever,13 14 and vaccinated patients are less likely to develop severe influenza.15 16 Medication prior to coming to the hospital can also affect the patient’s symptoms. In these cases, the characteristic symptoms of influenza are masked, which may reduce the accuracy of clinical diagnosis by physicians. However, there are no reports that have analysed whether these factors affect the clinical diagnostic accuracy of the physicians.

By analysing the physicians’ clinical diagnosis of influenza and the factors that affect the accuracy of the diagnosis, it is possible to distinguish between cases where the physician’s clinical diagnosis is reliable and cases where further testing is necessary. As a result, we may be able to avoid unnecessary tests for influenza diagnosis, which may cause physical and financial burdens on patients and infection risks to healthcare workers. This study aimed to elucidate the diagnostic accuracy of physicians’ pretest probability of influenza determination (%) as a quantitative clinical diagnosis of seasonal influenza and the factors that affect the clinical diagnosis.

Methods

We conducted a retrospective observational study to analyse the accuracy of the clinical diagnosis of influenza by physicians together with factors affecting the clinical diagnosis using data obtained from previous prospective observational studies conducted at a community clinic (Rokkasho Centre for Community and Family Medicine) and information from the patients’ electronic medical records.

Patients

The inclusion criteria for this study were as follows: (1) patients who had participated in a previous study regarding the accuracy of self-diagnosis of seasonal influenza in primary care medical institutions in Japan (Hirosaki University Graduate School of Medicine Ethics Committee, approval number 2017-1100) and (2) patients whose pretest probability of influenza by physicians could be extracted from their medical records. The exclusion criteria were as follows: (1) patients who had not provided consent and (2) patients with missing data.

The previous studies, including the subject of this study, were prospective observational studies conducted during the influenza season from December 2017 to April 2019. All patients with suspected influenza prior to physician consultation completed a pre-examination checklist, and diagnostic accuracy was investigated for self-diagnosis in patients≥12 years11 and guardian’s diagnosis in patients<11 years, using RIDTs as the reference standard.

Patient and public involvement

There was no public or patient involvement in the design, conduct or presentation of the results of the study.

Physicians

The clinical staff of the research clinic consisted of two staff physicians (family physicians with more than 20 years of experience), two family medicine residents and other part-time physicians. These physicians worked independently. The physician’s clinical influenza diagnosis, estimated as a pretest probability (recommended, not essential), was immediately recorded as part of the medical record information through medical interviews and examinations. RIDT was then ordered, and medication was prescribed if needed after the results were confirmed. The clinic also functioned as an educational centre for family medicine residents, and each physician and resident recorded pretest probabilities as far as possible for clinical discussion and resident training. The review meetings regarding the patients were held at the end of the daily practice; therefore, the recorded pretest probabilities were not influenced by other physicians.

Data collection

We collected data to investigate the diagnostic accuracy of the physicians’ clinical diagnoses of influenza and conduct an exploratory investigation of factors affecting the diagnostic accuracy.

The data extracted from the database of past observational studies were as follows: age, sex, medical history of influenza infection, influenza vaccination status, whether medication had been taken prior to medical visits, clinical signs (axillary temperature at the clinic, axillary temperature at home, pulse rate at the clinic), clinical symptoms (headache, nasal discharge, cough, joint and muscle pain, fatigue, history of fever (acute or sudden, gradual), time of symptom onset, the severity of current symptoms (compared with having a common cold)), time of RIDT, results of RIDT, pretest probability of influenza by the physician and the physician’s final diagnosis.

The data extracted from the medical records were as follows: type of physician (staff physician, resident, others) and the pretest probability of influenza determined by the physician.

Diagnosis of influenza

An RIDT (Prime Check Flu, Alfresa, Tokyo, Japan) was used as the reference standard for influenza, and the results were assessed by a clinical laboratory technician independent of the examining physician and the nurse who performed the pre-examination.

Statistical analyses

Receiver operating characteristic (ROC) curve analysis was performed to estimate the optimal cut-off point and area under the curve (AUC), which was determined to evaluate the discriminatory power of the physician’s diagnosis under various conditions.17 Moreover, 2×2 tables were analysed to calculate the sensitivity (Sn), specificity (Sp) and likelihood ratio at each cut-off point.

Based on previous studies,11 we estimated the AUC to be 0.75 and the influenza-positive rate to be 50%, with an alpha value of 0.05 and a power of 0.9. The sample size required for AUC analysis was estimated to be a minimum of 42 subjects and for subgroup analysis to be at least 100 subjects. All statistical analyses were performed using EZR V.1.40 (Saitama Medical Center, Jichi Medical University, Saitama, Japan).18

Results

Of the 504 patients who had participated in the previous studies,11 we analysed 335 patients (median age (IQR 25% to 75%), 31 (11 to 45); male, 66.6%) for whom the physician’s pretest probability could be extracted from the electronic medical records (online supplemental table S1, figure 1). First, the diagnostic accuracy of the physician’s pretest probability was analysed using the ROC curve. The AUC of the physician’s pretest probability was 0.77 (95% CI 0.72 to 0.82) (figure 2). The optimal cut-off value was 50%, for which the Sn and Sp were 79.0% (95% CI 71.9% to 85.0%) and 65.3% (95% CI 57.7% to 72.4%), respectively. At a cut-off value of 30%, the Sn and negative likelihood ratio were 92.0% (95% CI 86.7% to 95.7%) and 0.19 (95% CI 0.11 to 0.33), respectively. At a cut-off value of 80%, the Sp and positive likelihood ratio were 90.8% (95% CI 85.4% to 94.6%) and 4.01 (95% CI 2.41 to 6.66), respectively. The Sn, Sp and likelihood ratio for each of the other cut-off values are listed in table 1.

Table 1

Diagnostic accuracy of physician’s clinical diagnosis of influenza

Figure 1

Flow diagram of the study population.

Figure 2

Receiver operating characteristic curve of physicians’ clinical diagnosis of influenza.

Among the patients included in the study, 86 (25.7%) had a pretest probability of less than 30%, and 76 (22.7%) had a physician’s pretest probability of 80% or more.

Next, we analysed the diagnostic accuracy of the physicians in each subgroup. The AUCs of patients who had and had not taken some medications before visiting the clinic were 0.77 (95% CI 0.69 to 0.85) and 0.78 (95% CI 0.71 to 0.84), respectively, showing no evident difference. The AUCs of patients with type A and B influenza were 0.78 (95% CI 0.72 to 0.84) and 0.76 (95% CI 0.70 to 0.82), respectively, showing no evident difference. The AUCs of vaccinated and unvaccinated patients were 0.80 (95% CI 0.72 to 0.88) and 0.76 (95% CI 0.63 to 0.89), respectively, showing a slightly higher trend in the vaccinated group. The AUC of family medicine residents was 0.83 (95% CI 0.74 to 0.91), which was higher than that of staff physicians and other physicians. The AUC for patients less than 12 hours after onset was 0.69 (95% CI 0.51 to 0.88), and the AUC for patients aged younger than 6 years was 0.69 (95% CI 0.49 to 0.88), which was the lowest among the age groups (online supplemental table S2).

Discussion

The optimal cut-off value for the physician’s pretest probability (%) was 50%, and the Sn and Sp were 79.0% and 65.3%, respectively. At a cut-off value of 30%, the Sn and negative likelihood ratio were 92.0% and 0.19, respectively, indicating that exclusion diagnosis by clinical diagnosis was useful. At a cut-off value of 80%, the Sp and positive likelihood ratio were 90.8% and 4.01, respectively, indicating that it was useful to establish a definitive diagnosis by clinical diagnosis. Neither medication taken prior to the patient’s visit nor the type of influenza affected the physician’s clinical diagnosis, and vaccination did not make the clinical diagnosis difficult. However, clinical diagnosis is difficult in patients with symptoms less than 12 hours after onset and in paediatric patients aged younger than 6 years.

The usage of the RIDT in Japan and the accuracy of RIDT as a reference standard

RIDT kits are used frequently in Japan. According to Japanese government data, 22.7 million influenza rapid diagnostic kits were supplied to medical institutions during the 2019/2020 influenza season.19 In many countries, it seems normal not to perform RIDT when the physician’s prior probability is high or low, whereas, in Japan, the threshold for performing RIDT is low for the following reasons. (1) Schoolchildren with influenza are prohibited from attending school until they can no longer transmit the infection to others, according to the School Health and Safety Act. (2) Many companies have in-company rules based on the School Health and Safety Act. Schools and companies usually require patients to submit objective laboratory data (sometimes including the type of influenza). Therefore, this study, with the background of Japanese legal and cultural characteristics, was able to investigate the association with RIDT even when physicians estimated a high (or low) prior probability.

In this study, the RIDT was used as the reference standard. In general, the Sn of RIDT has been reported to be low,20 especially within 12 hours of onset.21 However, the Sn and Sp of the RIDT kit used in this study under the best conditions were reported to be 98.4% and 97.3%, respectively. According to the results of this study, the median time from onset to the test was 29.5 hours, suggesting that RIDT was performed under relatively high Sn conditions.

Accuracy of clinical diagnosis of influenza by physicians and perspectives on clinical practice

The accuracy of qualitative clinical diagnosis of influenza by physicians has been reported to be 29%–38% in Sn and 78%–92% in Sp,4–6 with Sn being particularly low. However, quantitative clinical diagnosis can be applied to diagnosis using different Sn and Sp values by changing the cut-off values. The diagnostic accuracy of these qualitative clinical diagnoses of influenza corresponds to a cut-off value of 60%–90% for physicians’ pretest probability of influenza in this study. Based on the results of our study, lowering the cut-off value to 30% resulted in an Sn of 92.0%, which can be used as an exclusion diagnosis; increasing the cut-off value to 80% resulted in an Sp of 90.8%, which can be used as a definitive diagnosis.

In our study, 48.4% of the patients had a pretest probability of less than 30% or greater than 80%. This indicates that the clinical diagnosis is highly reliable in approximately half of the patients in whom doctors strongly suspected (or hardly suspected) influenza. The administration of RIDT to the other half of cases where physicians cannot confirm the diagnosis (pretest probability of 30%–79%) will contribute to a more accurate, efficient and infection risk-averse diagnosis.

Factors affecting clinical diagnosis

When establishing a clinical diagnosis, the milder and less characteristic the symptoms are, the more difficult the clinical diagnosis is likely to be. However, the idea that relieving symptoms makes clinical diagnosis more difficult is not necessarily true, as reports have suggested that the use of opioid analgesics does not increase the risk of diagnostic errors in acute abdomen.22 In this study, there was no obvious difference in the categories of the factor groups, and the accuracy of clinical diagnosis of influenza was not decreased by premedication or vaccination. Annual influenza vaccination was reported to reduce influenza infection and medical visits,23 and the vaccine did not interfere with physicians’ clinical diagnosis of the patients in this study. The accuracy of clinical diagnosis was relatively low in children aged younger than 6 years and in cases less than 12 hours after onset. In children who cannot adequately report symptoms, or in cases shortly after the onset of illness, it is necessary to refer to information from the parents, re-diagnose after an appropriate time, and prioritise laboratory tests and treatment if necessary.

We considered that there were two possible reasons for the higher AUC among family medicine residents. First, the family medicine residents were able to allow sufficient time for medical interviews and examinations and receive supervision from the staff physicians because their practice was also intended as on-the-job training. Second, they had a greater opportunity to examine more typical patients suitable for training. Therefore, it is necessary to distinguish the diagnostic accuracy of residents from that of typical primary care practices.

Expectations for efficient clinical diagnosis

Although reports have indicated that influenza epidemics were significantly suppressed in Asian countries during the COVID-19 pandemic,24–26 the possibility of a simultaneous epidemic of COVID-19 and influenza cannot be excluded when public health control measures are relaxed in the future. In Japan, to prevent the spread of COVID-19 infection through testing for influenza, the Japan Medical Association has recommended that clinical diagnosis should be used instead of RIDT for the diagnosis of influenza, although this was an opinion without sufficient evidence at the time.27 When dealing with multiple diseases of different severities at the same time, it is necessary to carefully consider the priority of testing. In particular, when preventing droplet-transmitted infections, it is reasonable to avoid testing with nasal wipes and pharyngeal wipes as much as possible. Additionally, with limited medical resources such as medical expenses and medical staff, expensive and manpower-intensive tests should be performed only when they are clearly useful. Further validation studies are needed; however, if RIDT could be reduced by half in Japan, approximately 10 million RIDTs annually could be replaced by clinical diagnosis.19

It is reported that physicians overestimate the pretest probability for common diseases. One possible reason for this could be that physicians often do not think in terms of probability.28 Estimates of pretest probabilities generally reflect clinical knowledge and experience, but can also be derived from epidemiological data. When influenza is circulating in the community, patients with both cough and fever within 48 hours of symptom onset are reported to have a 79% pretest probability of influenza.3 We believe that diagnosing diseases with quantitative indexes such as pretest probability will lead to a more accurate and rational clinical diagnosis.

Limitations

This study has four limitations. First, this was a retrospective study in which new medical record data were added to the database of previous studies. The limited number of influenza cases for which we were able to extract the physician pretest probabilities was included in the study, which may have been affected by selection bias. Larger prospective studies that set physicians’ clinical diagnostic accuracy as the primary outcome are needed. Second, RIDT was used as the reference standard for influenza in this study. Although PCR should be used as a reference standard as it has higher Sn and Sp, it is difficult to perform in primary care settings in Japan. It is necessary to validate these results using reference standards with higher diagnostic accuracy in the future. Third, the number of doctors included in the study was small. The study was conducted in a single community medical institution, making it difficult to generalise the results. Since the results are expected to change depending on the background of the physicians (specialty, years of experience and region of work), it is necessary to conduct a study with a larger number of physicians and physicians with various backgrounds. In addition, the accuracy of the physician’s clinical diagnosis needs to be verified in other subjects and settings (nursing home residents, healthcare workers). In particular, this study included only a small number of elderly people, making it difficult to generalise the results to this population. Fourth, the results of the study may not be adaptable in situations such as the COVID-19 pandemic, where infectious diseases other than influenza are simultaneously prevalent and affect the clinical diagnosis of physicians. Influenza and COVID-19 have similar clinical symptoms, making it difficult to distinguish between the two based solely on symptoms.29 In the COVID-19 era, studies using reference tests that can evaluate both viruses simultaneously are desirable to accurately evaluate the physician’s clinical diagnosis of influenza.

Conclusion

The pretest probability of influenza (%) as a physician’s quantitative clinical diagnosis is useful for definitive and exclusionary diagnoses within the limits of our study. Neither the medication taken before the visit, influenza type nor vaccination affected the physician’s clinical diagnosis. However, clinical diagnosis was difficult for patients with influenza less than 12 hours after onset and children aged younger than 6 years than for other age groups.

Data availability statement

Data are available upon reasonable request. The data analysed in the current study were mostly included in this article. Additional data are available from the corresponding author on reasonable request.

Ethics statements

Patient consent for publication

Acknowledgments

The authors would like to thank Shunsuke Soma, MD, and Risa Yamauchi, MD, for their clinical support. We also thank the staff of the Rokkasho Center for Community and Family Medicine and all the patients who participated in our study.

References

Supplementary materials

  • Supplementary Data

    This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

Footnotes

  • Contributors HM conceived the idea and wrote the original draft of the manuscript. HM, TK, TA and FM were responsible for data acquisition and analysis. TK, TA, FM and SF developed the theory, and HO and HK supervised the findings of this study. All authors discussed the data and commented on the manuscript. HM, TK and TA revised and edited the manuscript. All authors have approved the final manuscript before submission. HM is responsible for the overall content as guarantor.

  • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

  • Competing interests None declared.

  • Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting or dissemination plans of this research.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.