Article Text
PDF
Abstract
Objectives To determine the negative predictive value (NPV) of the FebriDx point-of-care host response device in patients presenting with symptoms suggestive of COVID-19 infection in a mostly immunised Australian emergency department (ED) population during the late 2021 phase of the COVID-19 pandemic.
Design Observational diagnostic accuracy study comparing FebriDx point-of-care test to SARS-CoV-2 PCR.
Setting An ED in Melbourne, Australia, with 63 000 annual presentations in 2021.
Participants Patients aged 16 and over who met the Victorian Department of Health case definition for suspected COVID-19 infection PCR testing. Patients meeting any of the following criteria were excluded: <16 years of age; acute respiratory symptom(s) with onset>14 days prior to testing; current immunosuppressive or interferon therapy; live immunisation within the last 30 days; fever lasting>7 days; antibiotic or antiviral use in the preceding 14 days; experience of major trauma, major surgical intervention or severe burns within the last 30 days.
Primary and secondary outcome measures COVID-19 PCR results (detected, not detected) and FebriDx results (bacterial positive, viral negative, viral positive).
Results 94 participants were enrolled (female: 46; male: 48), 34% of participants (tested positive for COVID-19 according to PCR results, with a background incidence among all adult ED attenders of 2.5%. The sensitivity of FebriDx for detection of COVID-19 was 56% (95% CI 40% to 100%) and specificity was 92% (95% CI 84% to 100%). For the population tested, this resulted in an NPV of 80% (95% CI 71% to 100%) and a positive predictive value of 78% (95% CI 60% to 100%).
Conclusions In the context of a population with low COVID-19 infection rates, an evolved variant of COVID-19 and a very high community COVID-19 vaccination rate, FebriDx demonstrated reduced sensitivity and NPV relative to results from earlier international tests. These contextual factors should be considered during any attempt to generalise the current results.
Trial registration number ACTRN12620001029987 (Australian Clinical Trials).
- COVID-19
- ACCIDENT & EMERGENCY MEDICINE
- INFECTIOUS DISEASES
Data availability statement
No data are available. Not applicable.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
Both patients and treating clinicians were blinded to FebriDx results.
The study adopted a pragmatic COVID-19 case definition as defined by local testing practices at the time of patient enrolment.
The relatively small overall sample size resulted in wide CIs.
A very small number of unimmunised participants were enrolled, limiting subanalysis of this group.
Introduction
During the ongoing COVID-19 pandemic in 2021, a large number of patients continued to present to emergency departments (EDs) in Victoria, Australia, with symptoms suggestive of COVID-19 infection. These patients required viral screening on triage to facilitate appropriate disposition and treatment decision-making. This process frequently had a negative impact on patient flow and experience, as COVID-19 suspected patients were required to be held in an isolated zone until their laboratory PCR testing results became available, a process that often took many hours. The emerging availability of rapid antigen testing (RAT) in the final quarter of 2021 provided a potential alternative to PCR testing, but with a significant trade-off in test sensitivity.1–3
FebriDx (Lumos Diagnostics, Sarasota, Florida, USA)4 is a Therapeutic Goods Australia—registered single-use point-of-care testing device that detects two host response proteins, Myxovirus resistance protein A (MxA)5 6 and C-reactive protein (CRP), in finger-prick blood samples. It was designed to distinguish viral from bacterial respiratory infection. Results are available after 10 min and provide three possible outcomes: a negative result (control line only), a viral positive result (control line+MxA line±CRP line) and a bacterial positive result (control line+CRP line).
A UK study performed by Clark et al7 during the early phase of the COVID-19 pandemic found the FebriDx test had high accuracy for the detection of COVID-19 in adults who required hospitalisation and suggested that FebriDx could be deployed as a patient triage tool. In this study of 248 patients, the FebriDx test was shown to have a sensitivity of 93% and specificity of 86% for COVID-19 when compared with PCR testing. This translated to a negative predictive value (NPV) for COVID-19 of 93% for their study population. Similar early studies in Italy8 and the UK9 10 recorded an NPV of 95.3%, 96% and 86.8%, respectively. All studies were conducted during a time of high disease prevalence when vaccines were not yet available.
The aim of this study was to determine the NPV of FebriDx for detecting COVID-19 in patients presenting to an ED with symptoms suggestive of infection in Victoria, Australia. Differences in local testing criteria, lower population COVID-19 infection rates, an evolved strain of COVID-19 and a very high community COVID-19 vaccination rate were considered to be factors that made this population unique compared with those from previously published FebriDx studies.
Methods
A real-world observational diagnostic accuracy study was implemented to compare the sensitivity and specificity of the FebriDx point-of-care test to the reference standard of COVID-19 PCR. PCR tests conducted by the hospital’s pathologists involved one of the following: AusDiagnostics Hi-Plex RT-PCR, BD MaxTM SARS-CoV-2 Assay, Cepheid Xpert Xpress SARS-CoV-2 or the Alinity m SARS-CoV-2 assay. Patients who met the Victorian Department of Health case definition for suspected COVID-19 infection (COVID-S) PCR testing11 at the time of their ED presentation were invited to be tested with FebriDx in addition to their usual care. The definition consisted of the following criteria:
Fever OR chills in the absence of an alternative diagnosis that explains the clinical presentation OR acute respiratory infection (eg, cough, sore throat, shortness of breath, runny nose) OR Loss of smell or loss of taste.
To avoid FebriDx test results being incorporated into medical management decisions, testing was completed by a trained member of the research team. Both the patient and their treating clinician were blinded to FebriDx results. PCR results were available to those carrying out FebriDx tests if they had been performed prior to arrival to ED, while FebriDx results were unavailable to the pathologists generating PCR results.
The study was conducted at Box Hill Hospital ED in Melbourne, Australia, where there were approximately 63 000 presentations in 2021. Patients aged 16 years or over who agreed to participate in the study were required to provide signed consent. A trained member of the research team then obtained approximately 5 µL of blood from a single finger prick and applied this to the FebriDx device. After approximately 10 min, the result was obtained by the research team member and documented in REDCap, a secure web-based research data collection and management tool.12 ED PCR tests performed on the same day as the FebriDx testing were used for comparison. If this was not performed due to a previous positive result performed elsewhere less than 14 days prior, then this result was substituted.
FebriDx Results for all patients were recorded as either ‘negative’, ‘bacterial positive’ or ‘viral positive’ (online supplemental appendix 1). For the purpose of calculating performance characteristics for COVID-19 detection, only ‘viral positive’ results were classified as a positive result and all other results were classified as negative. In addition to the FebriDx test result, clinical data regarding length of illness, symptoms and COVID-19 vaccination status were collected from all patients.
Supplemental material
Sample size and statistical or power issues
An initial sample size of 300 was intended, based on an 80% probability of achieving a lower limit of the 95% CI of at least 0.96 for a calculated NPV. This recruitment target was not achieved due to reduced ongoing availability of recruitment staff and a pending change in local testing practices whereby suspected cases would receive RAT as a substitute for PCR testing. All statistical analyses and CIs were calculated using R.
Eligibility
Adult and paediatric patients (aged 16 and over) were eligible for inclusion. Convenience sampling was used, based on availability of recruitment staff. Participants were required to be capable of reading an English language patient information and consent form and providing in-person informed consent.
Patients undergoing COVID-S screening testing who did not meet the local case definition at the time of testing were excluded (eg, those engaged in asymptomatic preoperative screening and asymptomatic patients awaiting private hospital transfer). Patients who were critically unwell (ie, where the treating clinician felt that testing might interfere with their immediate clinical care) were also excluded.
Finally, patients who met any of the following FebriDx device registration exclusion criteria were excluded:
<16 years of age.
Acute respiratory symptom(s) with onset>14 days prior to testing.
Current immunosuppressive or interferon therapy.
Live immunisation within the last 30 days.
Fever lasting>7 days.
Antibiotic or antiviral use in the preceding 14 days.
Experience of major trauma, major surgical intervention or severe burns within the last 30 days.
Patient and public involvement
Patients or the public were not involved in the design, conduct, reporting or dissemination plans of this research.
Results
A total of 107 patients were enrolled between 8 October 2021 and 2 January 2022. Of these, 13 failed to meet inclusion criteria and were excluded, resulting in a final sample size of 94. Details are shown in figure 1. Note that the original recruitment target was not achieved due to reduced ongoing availability of recruitment staff and a pending change in local testing practices whereby suspected cases would receive RAT as a substitute for PCR testing.
Flow chart of patients and outline of exclusion criteria. ED, emergency department.
Patient and clinical characteristics
A summary of patient clinical characteristics is shown in table 1.
Clinical characteristics of all patients, COVID-19 positive patients and COVID-19 negative patients
COVID-19 incidence in the tested population was 34%, with 32 of the 94 patients tested having a confirmed COVID-19 positive result on PCR. During the study period, a total of 13 294 adult patients presented to the ED, of whom 334 were diagnosed as COVID-19 positive, resulting in a background incidence rate of 2.5%.
Performance characteristics of FebriDx test
Table 2 shows the overall performance characteristics of the FebriDx test for enrolled patients, with an overall NPV of 80% (95% CI 71% to 100%).
Overall performance characteristics of the FebriDx test
For patients with confirmed fever or ongoing respiratory symptoms (one or more of sore throat, runny nose, cough or shortness of breath), the NPV was 78% (95% CI 68% to 100%). This is shown in table 3.
Performance characteristics of the FebriDx test in patients with confirmed fever and/or ongoing respiratory symptoms
Of 14 patients who were unvaccinated or who had received only 1 dose of a COVID-19 vaccine, FebriDx correctly identified 9 out of 11 PCR positive cases. The remaining three yielded negative results with both tests.
Discussion
This real-world study of the FebriDx point-of-care host response device in an Australian ED during the COVID-19 pandemic in late 2021 showed a sensitivity of 56% and an NPV of 80% for the population tested. Subgroup analysis showed little difference for patients with confirmed fever or respiratory tract symptoms, but sensitivity rose to 82% for patients with partial or no prior history of COVID-19 immunisation.
The sensitivity of FebriDx for detecting COVID-19 in this study was lower than that found in international studies, where values ranged from 93%–100%.7 8 13 14 Potential differences in the studied populations exist, including the predominant circulating variant of SARS-CoV-2, population vaccination rates during study recruitment and COVID-19 incidence among suspected cases. Patient enrolment dates for these comparable studies ranged from 2020 until early 2021 when the initial wild type (ancestral) or alpha (B.1.1.7) SARS-CoV-2 virus was dominant in the enrolment countries and vaccination was not widely available.15–17 In contrast, cases detected in Victoria, Australia, during our enrolment period were predominantly of the Delta (B.1.617.2) variant, with a small number of Omicron (B.1.1.529) cases towards late 2021 as the variant emerged, and 85% of our study cohort had completed a primary vaccine course against COVID-19. Vaccination is shown to reduce progression to severe COVID-19 illness and death,18 including the requirement for hospitalisation and oxygen therapy or advanced supportive care. It is possible that vaccination has also abated host production of the two host response proteins,19 MxA and CRP, such that test sensitivity is reduced in the vaccinated host. The potential increase in sensitivity found in the small number of patients recruited who were not fully vaccinated supports this hypothesis.
Point-of-care RAT became widely available in Australia in late 2021, with a sensitivity of 75.5% (95 CI 69.9% to 80.4%) among symptomatic people presenting to the ED.1 While we had hoped to pair FebriDx testing with corresponding RAT results for direct comparison, RAT availability during the trial period meant that few of the patients enrolled into this study had RAT results available and this comparison was not possible. In 2020, Pulia et al20 postulated the pairing of a high sensitivity dual biomarker host response test with a high specificity serology-based test as an effective and rapid initial triage strategy. The reduced sensitivity found in the vaccinated participants in our study suggests that this approach may not have merit throughout all stages of a pandemic, and that the most useful role for biomarker testing might be during the early stages of a disease outbreak, when more specific testing is not yet available.
Limitations
There was a risk of selection bias; sampling was non-random, with non-consecutive participants enrolled due to variability in research team availability. The small size of the sample (n=94) may also have increased the risk of a type II error, especially when performing subgroup analysis. However, the initial powering calculations were designed to estimate NPV, which depends on prevalence. At the time of the original power analysis (August 2020), observed prevalence was low (less than 5% in the target population). This compared with a prevalence of 34% during patient recruitment, making the original powering obsolete.
An initial sample size of 300 was intended, based on an 80% probability of achieving a lower limit of the 95% CI of at least 0.96 for a calculated NPV. This recruitment target was not achieved due to reduced ongoing availability of recruitment staff and a pending change in local testing practices whereby suspected cases would receive RAT as a substitute for PCR testing.
Patients were enrolled between October 2021 and January 2022, when there were two predominant SAR-CoV-2 variants, Delta (B.1.617.2) and Omicron (B.1.1.529). This may affect the comparability of our results to those from earlier variant studies in the UK and Italy.7 8 13
Since 32% of PCR comparator tests were performed prior to presenting to our ED, it is possible comparator results were not standardised throughout the study. COVID-19 PCR assay determination of positive or negative results are impacted by the test’s limit of detection (LoD) as well as number of cycles (cycle thresholds (Ct)) performed to determine if the result is positive/negative. Both the Ct values and LoDs can vary widely between COVID-19 assays. Furthermore, PCR tests can remain positive for days to weeks following an active infection. Therefore, it is possible that PCR characterised patients as positive when they may have been presenting to the hospital with a resolved infection with lingering symptoms or an unrelated acute respiratory illness. Finally, RAT had yet to become widespread in the ED during the study period, making it impractical to compare the respective sensitivity and specificity of FebriDx and RAT for this sample.
Conclusion
The FebriDx point-of-care host response device was found to have an NPV of 80% and sensitivity of 56% for COVID-19 infection when applied to a mostly immunised Australian ED population during the late 2021 phase of the COVID-19 pandemic. This sensitivity was reduced compared with earlier international tests, which may reflect differences in population immunisation rates and the prevalent (SARS-CoV-2) COVID-19 strain at the time of testing.
Data availability statement
No data are available. Not applicable.
Ethics statements
Patient consent for publication
Ethics approval
This study involves human participants. Ethical approval was obtained from the St Vincent’s Hospital (Melbourne) Human Research Ethics Committee (project number HREC-A 136/20) and local governance approval was obtained for Box Hill Hospital (local reference S20-084-65949). Participants gave informed consent to participate in the study before taking part.
Acknowledgments
The authors would like to thank Roy Chean, Grace Buzasi, Catherine Smith, Bronwyn Kirkwood, Matthew Welsh, Suzanne Holm, Christina Katsambanis, Elena Zalac, Brittany Hacket, and Hung Phung.
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Contributors PB was involved in the initial concept and audit design and is a guarantor. PB, JM, AP, FQAW, HM, MA, KHT, MR, FM, SJ and BD contributed to data collection. PB, AP, JM and EA were involved in data analysis. PB, JM, AP, SG, FQAW, HM, MA, KHT, MR, FM, EA, LH, SJ and BD contributed to manuscript writing and revision. All authors have read and approved the final manuscript.
Funding This work was supported by the study sponsor, Planet Innovation, Australia, who were involved in conception, study design, staff training, data and manuscript review. Planet Innovation provided all FebriDx testing kits and iPads for data collection, as well as contributing funding for research support, ethics and governance fees.
Competing interests The lead author and principal investigator is employed by Eastern Health and has no affiliation with Planet Innovation or Lumos Diagnostics. He received no payment or honorarium for involvement in this study.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.