Article Text
PDF
Abstract
Objective Various point-of-care testing (POCT) urine analysers are commercially available for routine urine analysis in general practice. The present study compares analytical performance, agreement and user-friendliness of six different POCT urine analysers for diagnosing urinary tract infection in general practice.
Setting All testing procedures were performed at a diagnostic centre for primary care in the Netherlands. Urine samples were collected at four general practices.
Primary and secondary outcome measures Analytical performance and agreement of the POCT analysers regarding nitrite, leucocytes and erythrocytes, with the laboratory reference standard, was the primary outcome measure, and analysed by calculating sensitivity, specificity, positive and negative predictive value, and Cohen's κ coefficient for agreement. Secondary outcome measures were the user-friendliness of the POCT analysers, in addition to other characteristics of the analysers.
Results The following six POCT analysers were evaluated: Uryxxon Relax (Macherey Nagel), Urisys 1100 (Roche), Clinitek Status (Siemens), Aution 11 (Menarini), Aution Micro (Menarini) and Urilyzer (Analyticon). Analytical performance was good for all analysers. Compared with laboratory reference standards, overall agreement was good, but differed per parameter and per analyser. Concerning the nitrite test, the most important test for clinical practice, all but one showed perfect agreement with the laboratory standard. For leucocytes and erythrocytes specificity was high, but sensitivity was considerably lower. Agreement for leucocytes varied between good to very good, and for the erythrocyte test between fair and good. First-time users indicated that the analysers were easy to use. They expected higher productivity and accuracy when using these analysers in daily practice.
Conclusions The overall performance and user-friendliness of all six commercially available POCT urine analysers was sufficient to justify routine use in suspected urinary tract infections in general practice.
- GENERAL MEDICINE (see Internal Medicine)
- PRIMARY CARE
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Strengths and limitations of this study
This is the first time that different commercially available point of care testing (POCT) urine analysers have been evaluated on analytical performance, agreement and user-friendliness.
All six POCT urine analysers in our study have good analytical performance.
First-time users indicate that the analysers are easy to use. They expect higher productivity and accuracy when using these analysers in daily practice.
This study does not address whether the use of POCT urine analysers in the primary care setting has added value compared with visual reading. A study on this subject is currently being conducted.
Introduction
With urinary tract infection (UTI) being part of the top 10 diagnoses for which a general practitioner (GP) is consulted in the Netherlands, urine test strip analysis to diagnose UTI is a common diagnostic procedure in general practice. Urine analysis takes place in almost 2% of all consultations.1 Commonly, UTI is diagnosed based on the test results for nitrite, leucocytes and erythrocytes on the test strip, performed at the GP's office by the GP nurse or midwife. Previous studies have shown that, in general practice, negative test strip test results for nitrite and leucocytes are useful in excluding UTI.2 Professional guidelines in the Netherlands recommend performance of a test strip evaluation of urine in case of a clinical suspicion of UTI. In case of a nitrite negative test strip result, the presence of leucocytes and erythrocytes is assessed. If one of these is positive, a (semi)quantitative urine culture needs to be performed for conclusive diagnosis.3
For urine analysis, a test strip for various tests is dipped in urine. After contact with the urine, the coloured pads of the test strip for analysis of nitrite, leucocytes, protein, glucose, ketones, urobilinogen, bilirubin and erythrocytes (dependent of the type of strip used), can be assessed visually by comparing them to a colour chart most frequently printed on the side of the bottle containing the test strips. Visual inspection of test strips is subjective and errors may occur due to a variety of reasons, including poor technique and poor, inadequate or untimely reading of the test results.4 ,5 It has been suggested that the use of point of care testing (POCT) urine analysers will increase the quality of urinalysis.6 ,7 With the introduction of commercially available POCT urine analysers, it may be possible to improve the quality of urinalysis in general practice. These analysers, therefore, need to have good analytical performance and agreement with laboratory standards, but data are lacking for the different parameters. We set out to test, as the primary outcome measure, analytical performance and agreement of six POCT urine analysers on nitrite, leucocytes and erythrocytes, compared with laboratory standard reference tests. Furthermore, as secondary outcome measures, user-friendliness, costs and other characteristics of the analysers were compared.
Methods
Setting and samples
All testing procedures were performed at Saltro Diagnostic Centre, an accredited diagnostic centre that provides laboratory and other diagnostic services in routine primary care for approximately 850 GPs, and over one million inhabitants. We collected 77 urine samples of patients who delivered a urine sample for routine investigation at one of the four participating general practices. Samples were stabilised using laboratory tubes containing Stabilur (Greiner BioOne, Kremsmuenster, Austria). Urine samples are transported from various locations (GP offices and blood draw locations) to the central laboratory for analysis. Saltro Diagnostic Centre routinely uses Stabilur to preserve urine samples for particle counting. Stabilur preservative tubes are acceptable for test strip analysis within 8 h on the day of urine collection.8 Before introduction into the Saltro laboratory, validity of these tubes was tested (personal documentation). Urine samples were analysed on six POCT urine analysers and the laboratory reference standards, the Urisys 2400 and the Sedimax urine analyser, by experienced laboratory technicians. All measurements were carried out within 2 h to avoid any incorrect test results due to ageing of the urine samples.
Automatic urine strip POCT analysers
We included six commercially available POCT urine analysers in our study: Uryxxon Relax (Macherey Nagel, Düren, Germany), Urisys 1100 (Roche Diagnostics Ltd, Rotkreuz, Switzerland), Clinitek Status (Siemens Healthcare, Munich, Germany), Aution 11 and Aution Micro (both Menarini Diagnostics, Florence, Italy) and Urilyzer (Analyticon Biotechnologies AG, Lichtenfels, Germany). All analysers qualitatively and semiquantitatively measure blood, urobilinogen, bilirubin, protein, nitrite, ketones, glucose, pH, specific gravity and leucocytes. Additionally, the Clinitek Status, and Aution 11 and Aution Micro, can also measure albumin and creatinine.
The different device characteristics are set out in table 1.
Device characteristics
Laboratory reference standard
We used the Urisys 2400 automated urine analyser (Roche Diagnostics Ltd, Rotkreuz, Switzerland) as the laboratory reference standard for the six different POCT urine analysers. Performance of the reference standard has previously been proven to be good.9–11 This analyser is routinely used for all (primary care) urine samples at Saltro Diagnostic Centre (Utrecht, the Netherlands). The Urisys 2400 test strips are measured 60 s after pipetting of the sample onto the test areas. Green and orange LEDs are used for the evaluation of the reactive test areas. Compensation of the intrinsic urine colour is carried out using the reflectance value of the compensation pad for the green LED. Urine colour result is determined by using the reflectance values of the compensation pad for three wavelengths (orange, green and blue). Measuring an internal reference plate along with each test strip measurement automatically compensates potential influence of varying ambient temperature on the test results. The Urisys 2400 measures pH, leucocytes, nitrite, protein, glucose, ketones, urobilinogen, bilirubin, blood (erythrocytes/haemoglobin), clarity and specific gravity.
We used the Sedimax urine analyser (Menarini, Florence, Italy) as additional reference standard for quantitative evaluation of leucocytes, erythrocytes and sediments, to verify the quantitative results of the different urine test strip POCT analysers. Analytical and diagnostic performance of the Sedimax urine analyser has previously been proven to be good.12 ,13
Analytical performance
To assess within-day and between-day performance, a protocol was used based on the clinical and laboratory standards institute EP5 and EP9 guidelines.14 ,15 For this study, Dipper urine dipstick control (Quantimetrix, California, USA) levels 1 and 2 were used. These are ready-to-use control liquids, intended as a control for urinalysis reagent strips, requiring no reconstitution or dilution. They are prepared from human urine fortified to target levels. Level 1 is used as a negative control and level 2 as a positive control.
The within-day performance of each POCT urine analyser was quantified testing 20 aliquots of level 1 and 20 aliquots of level 2 on all six POCT urine analysers. Results were compared to expected values listed in the manual of the Dipper urine dipstick control levels 1 and 2.
For the between-day performance, 20 aliquots of levels 1 and 2 Dipper urine dipstick control were tested on every device on 10 consecutive days, twice a day.
Agreement
We assessed agreement of the six POCT urine analysers with the laboratory reference standard urine analyser (Urisys 2400). The routinely used cut-off values of 25 leucocytes/μL and 10 erythrocytes/μL of the Urisys 2400 representing a positive test result, were also used for this study. POCT urine test strips were considered positive when the result was at least ‘1+’, or when its quantitative equivalent was present, as reported in the test strip manual. For a full report on qualitative and equivalent quantitative results per test strip, see online supplementary appendix 1.
Additional reference standard tests to verify the test results on the POCT urine analysers in the quantitative evaluation of leucocytes and erythrocytes were performed using the Sedimax urine analyser using the same cut-off values.
Statistical analysis
Analytical performance and agreement of the POCT analysers regarding nitrite, leucocytes and erythrocytes, with the laboratory reference standard, was the primary outcome measure. Analytical performance of the different POCT urine analysers compared with the reference methods was evaluated by calculating sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV). Agreement between the POCT urine analysers and the reference methods was analysed by calculation of the Cohen's κ coefficient and its 95% CI. Correlations were ranked as very good: κ=0.81–1.00; good: κ=0.61–0.81; moderate: κ=0.41–0.60; fair: κ=0.21–0.40; poor κ<0.20.16
User-friendliness
User-friendliness of the six POCT urine analysers was assessed in a separate survey, using a standardised questionnaire.17 The questionnaire contained five questions concerning the user-friendliness of the analyser, test procedure and the susceptibility to flaws. A group of seven GP assistants and two midwives unfamiliar with the devices was asked to perform tests on all six POCT urine analysers in random order. GP assistants and midwives were not instructed on the use of the different POCT urine analysers prior to testing. However, they did receive the short written instruction provided by the manufacturers. Immediately after performing each test, the questionnaire was completed. At the end, first-time users were asked which analyser they found to be most user-friendly.
Subsequently, first-time users were asked if a POCT urine analyser was deemed useful in their daily practice and if they thought that using such an analyser would improve their productivity and efficiency, and whether the analyser would make urine analysis more precise.
Additionally, analyser characteristics collected from manufacturers’ information sheets were evaluated. The combined data were used to evaluate user-friendliness.
Results
Analytical performance
Within-day and between-day performance was assessed using level 1 (negative) and level 2 (positive) Dipper Urine Dipstick Controls (Quantimetrix). All results were in agreement with the predefined expected values, meaning they showed negative results for level 1 controls and positive results for level 2 controls for nitrite, leucocytes and erythrocytes at the 20 within-day analyses and the 20 between-day analyses.
Agreement with laboratory reference standards
Seventy-seven anonymous patient samples were collected and analysed using the laboratory standard (Urisys 2400).
Eleven samples showed a positive result for nitrite, 32 samples tested positive on leucocytes (>25 leucocytes/μL) and 44 of the 77 samples showed the presence of >10 erythrocytes/μL. Samples were then analysed on the six different POCT urine analysers. Table 2 shows the sensitivity, specificity, PPVs and NPVs, as well as the κ coefficient and its 95% CI for nitrite, leucocytes and erythrocytes per POCT urine analyser using the Urisys 2400 as reference standard.
Sensitivity, specificity, PPV, NPV and κ, with 95% CI between all six different urine analysers and the primary laboratory reference test (Urisys 2400)
For nitrite, all test characteristics (sensitivity, specificity, PPV, NPV) were high for all POCT urine analysers. For the leucocyte and erythrocyte test, specificity was higher than sensitivity for all POCT urine analysers. PPV and NPV were high for the leucocyte tests whereas for the erythrocyte test the PPV was high but the NPV was lower.
Agreement of the nitrite test between the various POCT urine analysers compared with the laboratory analyser was very good based on the κ coefficient for all variables. For the leucocytes tests, agreement varied between good to very good, and for the erythrocyte test, between moderate and good (table 2).
Additionally, samples were reanalysed using the Sedimax urine analyser as a secondary reference standard, and control for the primary reference standard (Urisys 2400), to obtain quantitative results concerning the presence of leucocytes and erythrocytes. Of the 77 samples, 27 samples contained more than 25 leucocytes/μL and 12 samples showed the presence of more than 10 erythrocytes/μL. Using the Sedimax analyser as the secondary reference standard, we calculated the sensitivity, specificity, PPVs and NPVs, as well as the κ coefficient and its 95% CI of the different analysers, as shown in table 3.
Sensitivity, specificity, PPV, NPV and κ, with 95% CI for the POCT urine analysers, compared with the secondary laboratory reference test (Sedimax)
For the leucocytes test, characteristics (sensitivity, specificity, PPV and NPV) were high for all POCT urine analysers. For erythrocytes, sensitivity was generally higher than specificity. The PPV for the erythrocyte test was generally low and NPV high for all POCT urine analysers.
Agreement of the leucocyte test between the various POCT urine analysers compared with the laboratory analyser was good to very good based on the κ coefficient for all variables. For the erythrocyte test, agreement varied between fair to good (table 3).
User-friendliness
The secondary outcome measures were the user-friendliness of the POCT analysers, in addition to the analysers’ other characteristics. The GP practice nurses (all first-time users) found POCT urine analysers easy to use and most frequently did not have problems receiving test results. The susceptibility to flaws, either in preparation of the analyser, performing the analysis or reading the test results was considered lowest for the Uryxxon Relax (Macherey Nagel) and Urisys 1100 (Roche). Overall, the Uryxxon Relax (Macherey Nagel) was found to be most user-friendly by six of the nine first-time users. Six first-time users found that the POCT urine analyser would be useful in their daily practice. The majority of first-time users felt that the POCT urine analysers would improve efficacy and productivity. Seven out of nine first-time users felt that the use of POCT urine analysers would lead to a more accurate evaluation of the urine strip.
Device characteristics differ mainly in size and price of the device. All but the Aution Micro (Menarini) have the ability to connect to the GP's electronic medical record and to the laboratory information system. The Clinitek Status (Siemens), Aution 11 and Aution Micro (Menarini) also make it possible to perform additional tests, but this was not part of our study. All test strips used in this study contain pads for the analysis of additional tests such as glucose and protein. Although these are also used in general practice, we chose to focus on the tests most often used by GPs, that is, nitrite, leucocytes and erythrocytes, for the diagnosis of UTI.
Discussion
This is the first time that different commercially available POCT urine analysers have been evaluated on analytical performance, agreement and user-friendliness. All POCT urine analysers showed perfect analytical performance when evaluating within-day and between-day performance using control materials. Compared with laboratory reference standards, agreement was generally good to very good, but differed per parameter and per analyser. Concerning the nitrite test, all but one showed perfect agreement with the laboratory standard. For leucocytes and erythrocytes, specificity was high, but sensitivity was generally lower, when compared with the reference standard. Users (GP practice nurses and midwives) reported that the different POCT urine analysers were easy to use.
When comparing our study to the currently available literature, it is apparent that this is the first time that POCT urine analysers have been compared with a laboratory standard for use in general practice. Various studies report on the validation of new laboratory urine analysers.12 ,18–20 Two studies have shown that automatic reading of urine test strips is more accurate than visual reading of the same test strip.6 ,7 Peele et al already reported on this in 1977 using a semiautomated dipstick reader in a laboratory setting. However, technology has developed since and the described analyser is no longer available.6 Tighe7 performed his study in a laboratory setting, using aqueous spiked samples instead of patient samples, and although they did evaluate a POCT urine analyser, it was not compared with a laboratory reference standard.
Multiple studies, all conducted in a hospital laboratory setting, have shown that automated urinalysis, using different automated urinalysis systems (not for POCT), is acceptable for the screening of urine samples, and thus avoiding unnecessary urine culture.9 ,21 ,22 Mayo et al showed that when comparing two different automated test strip analysers in a laboratory setting, agreement was 99% for nitrite, 82% for leucocytes and 91% for erythrocytes. When comparing the Urisys analyser to manual examination of the urine sediment agreement was 90% for nitrite, 71% for leucocytes and 86% for erythrocytes.22 Chien et al compared three different automated test strip analysers, also in a laboratory setting. They found agreement was 97% for nitrite, 97% for leucocytes and 86% for erythrocytes.9 We have studied and compared all commercially available POCT urine analysers in the Netherlands concerning the most clinically relevant parameters for general practice. We found that agreement with the laboratory standard is similar to agreement found between urine analysers in a laboratory setting, and between urine analysers and manual examination of the urine sediment in a laboratory setting, in the above studies. This confirms that the currently tested POCT urine analysers perform sufficiently for use in primary care practices.
According to the guideline for UTI of the Dutch College of General Practitioners,3 nitrite, leucocyte and erythrocyte tests need to be performed for patients with possible UTI. A positive nitrite test is conclusive for a UTI. A negative nitrite test needs to be combined with the results of the leucocyte and/or erythrocyte test. If either the leucocyte and/or erythrocyte test is positive, the test needs to be followed up with (semi) urine culture. Our test results show that the POCT urine analysers, except for one, have excellent agreement for nitrite compared with the reference method. The presence of leucocytes and/or erythrocytes when tested on the POCT urine analysers rules in UTI in most cases (high PPV when compared with the reference method), whereas the absence of erythrocytes or leucocytes does not rule out UTI in all cases (relatively low NPV). The chance of missing a UTI is, however, reduced by the guideline statement that either leucocytes or erythrocytes needs to be positive to follow-up with (semi) urine culture.
Several limitations of this study need to be addressed. We used the Urisys 2400 automated urine analyser, the routine laboratory analyser at Saltro Diagnostic Centre, as our reference standard. Although the quality of this analyser is generally acknowledged, we cannot exclude the possibility that study results would have been different if another laboratory analyser had been used as a reference standard. Therefore, we have chosen to add additional reference standards for the measurement of leucocytes and erythrocytes. When interpreting the results, the differences in these reference standards have to be considered. When using the Sedimax urine analyser for quantitative measurement of leucocytes and erythrocytes, less leucocyte and erythrocyte positive samples were found than when samples were analysed using the laboratory standard Urisys 2400. An explanation for this is that leucocytes and erythrocytes may have lysed, which can occur when there is a renal infection, and may therefore not have been counted with the Sedimax urine analyser. Dipsticks do not have this limitation, which makes them more sensitive, and therefore possibly accounts for the difference in positive samples.
We compared different dipsticks from different manufacturers. Most dipsticks show their results in qualitative (1+, 2+, etc) and quantitative values. However, not all dipsticks use the same cut-off values for displaying their test results. We chose to use the cut-off value of 1+ as cut-off for a positive test because this is most often used in general practice. When translating back to quantitative results, using the tables provided by the manufacturer, values of leucocytes and erythrocytes can be vastly different. For example, 1+ leucocytes on a dipstick manufactured by Siemens translates to a quantitative value of 70 leucocytes/μL. All other dipsticks give a value of 25 leucocytes/μL as 1+. This problem has been addressed before and users need to be aware of the cut-off values as provided by the manufacturer when interpreting the test results of a urine analysis.21
Although our first-time user panel found that all six POCT urine analysers were relatively easy to use, a larger number of GP assistants and midwives performing the analysis in routine practice would have been desirable to draw firm conclusions on user-friendliness. But the simple test procedure (dipping the urine stick into the urine, entering the strip into the device and display reading of the test result) makes it hard to believe that conclusions would have been different.
This study shows that the POCT urine analysers have good analytical performance and their performance is sufficient to support use in general practice. POCT urine analysers may improve the quality of urinalysis in general practice but a direct comparison of visual interpretation versus automatic reading in routine general practice is necessary to judge the additional value of POCT urine analysis. This study is currently being performed at Saltro Diagnostic Centre.
Conclusions
All six POCT urine analysers in our study have good analytical performance for diagnosing UTI (nitrite, leucocytes and erythrocytes). All but one POCT urine analyser showed perfect agreement with the laboratory standard concerning nitrite test results. Concerning leucocytes and erythrocytes most POCT urine analysers showed high specificity, but sensitivity was lower. User-friendliness was judged as good for all analysers.
Acknowledgments
The authors thank all participating GPs and GP assistants who collected the urine samples for our study. They thank Anouk Vrancken, Ton van Dijk and Henny Hatzmann from Saltro Diagnostic Centre for the analysis of the urine samples with the laboratory reference standards.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
- Data supplement 1 - Online supplement
Footnotes
Contributors MJCS contributed to data collection, data analysis and writing of the manuscript. SvD and RMH contributed to study design, data collection, data analysis and writing of the manuscript. AMJK contributed to data analysis and writing of the manuscript. NJdW contributed to critical revision of the manuscript and provided study support.
Funding This study was equally funded by the five manufacturers of the six automatic urine strip POCT analysers and by Saltro Diagnostic Centre. Each manufacturer funded 1/6 of the total sum; Saltro Diagnostic Centre funded the remaining 1/6 share.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement Additional data can be accessed via the Dryad data repository at http://datadryad.org/ with the doi:10.5061/dryad.kv3q7.