Article Text
PDF
Abstract
Objective There is a lack of evidence addressing several important human factors questions pertaining to the quality of supportive information provided by commercial manufacturers that can affect the adoption and use of lateral flow serology assays in practice. We aimed to: (1) identify and assess the quality of information that commercial manufacturers provided for their point-of-care tests (POCTs) and (2) examine the implications of these findings on real-world settings.
Design We used a content analysis methodology in two stages to systematically, code and analyse textual data from documents of commercial manufacturers. A deductive approach was applied using a coding guide based on the validated Point-of-Care Key Evidence Tool (POCKET) multidimensional checklist. An inductive approach was used to identify new patterns or themes generated from our textual analysis.
Setting Publicly available supportive information documents by commercial manufacturers for lateral flow serology, were identified and gathered from online searches.
Participants Supportive information documents retrieved from online searches over 3 months (March 2020 to June 2020).
Results A total of 79 POCTs were identified that met the study inclusion criteria. Using the POCKET coding guide, we found that the quality of information varied significantly between the manufacturers and was often lacking in detail. Our inductive approach further examined these topics and found that several statements were vague and that significant variations in the level of details existed between manufacturers.
Conclusions This study revealed significant concerns surrounding the supportive information reported by manufacturers for lateral flow serology assays. Information transparency was poor and human factor issues were not properly addressed to mitigate the risk of improper device use, although it should be noted that the results of our study are limited by the data that manufactures were prepared to disclose. Overall, commercial manufacturers should improve the quality and value of information presented in their supporting documentation.
- COVID-19
- public health
- infectious diseases
- health services administration & management
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
The study addresses the gap and need for empirical knowledge concerning the quality of supportive information provided by commercial point-of-care tests (POCTs) manufacturers.
This study explores the implications for efficacy, efficiency and value in context-of-use in real-world environments.
Content analysis was used in two-stages to scrutinise the textual data by examining language for meaning, identify recurring patterns or themes and measuring the frequency of categories reported.
This content analysis included 79 publicly available commercial documents for POCTs, retrieved from three online websites.
Results may be biased by the exclusion of tests for which the manufacturers did not provide relevant documentation.
Introduction
The global pandemic of the COVID-19, caused by the SARS-CoV-2, has illuminated significant challenges that health systems face in providing comprehensive population-level testing to detect and slow the spread of the disease and manage the outbreak. Health systems have struggled to provide wide-scale testing that can match the scale and pace of the virus spreading as a result of the long turnaround times for results associated with conventional laboratory testing,1 shortages of supplies to perform tests,2 3 contractual restrictions4 and regulations that initially limited testing at public health laboratories.5–7 In response, government strategies, such as those in the UK and USA, have focused on scaling-up diagnostic capacities by both expanding testing to take place outside the public health laboratories,8–10 and authorising significant research funding programmes to stimulate the accelerated development and dissemination of diagnostic testing devices.11–13
In response, numerous commercial manufacturers have rushed to develop diagnostic testing devices that are primarily based on laboratory molecular techniques and point-of-care test (POCT) methods.14 Molecular techniques, such as real-time reverse transcriptase-PCR (RT-PCR), are regarded as the current gold standard for identifying the SARS-CoV-2 virus, however, scaling up this approach to satisfactorily match the speed and spread of the virus is challenging because it is a time-consuming process to generate results, labour intensive, and requires specialist training and equipment.15 16 Moreover, the diagnostic sensitivity of molecular tests also risks eliciting inconsistent results (false-negative and false-positive results).17 Consequently, the urgency for faster turnaround test times and population-scale testing has placed a larger focus on POCTs to support rapid testing efforts.18 19 POCTs have the potential to deliver results within 10–20 min, be used outside laboratory settings, reduce resource burdens, support containment efforts, and help policymakers and experts gauge the number of symptomatic and asymptomatic cases in a population.20–24 At the same time, uncertainties have been raised in light of the insufficient evidence supporting their scientific and clinical validity.25–27
Despite the limited evidence-base, several CE-labelled serology-based POCTs have entered the market but been plagued by reports of poor inaccuracy. For example, tests developed by Chinese manufacturers and procured by multiple European governments had unacceptably high error rates.28–30 Outside Europe, the US Food and Drug Administration (FDA) recently tightened the rules for approving serology-based POCTs after more than 100 antibody tests were permitted onto the market without prior review, consequently leading to the dissemination of several dubious quality tests, and companies engaging in fraudulent marketing activities, and fabricating unsubstantiated claims about testing accuracy.31 32
While the discussions surrounding the challenges of POCTs have naturally revolved around testing accuracy, several important human factors questions pertaining to the adoption and use of these devices are equally important to bear in mind but remain unaddressed.33 Particularly, POCTs are affected by several environmental and operator-related factors,34 which have important implications on the efficacy, effectiveness and value of these testing kits in both the clinical and domestic settings.35–39 For instance, tests that have been validated in a controlled laboratory environment may not perform as well in the context of real-world settings if individuals operating the device have little or no specialty training in POCTs or background laboratory training,40 41 and operators struggle to understand and adhere to instructions included with the kits.42 The ISO 62366 highlights the importance of the quality of the information of training materials, including accompanying information such as instructions for use. This standard identifies the need for the manufacturer to develop training materials that consider ‘the wording or pictures to be used to ensure clarity and understandability; the immediate recipients (eg, users, service personnel, installers and patients); and the appropriate media for providing the information, (eg, instructions for use, labels)’.43
This highlights that manufacturer materials (eg, medical device labelling and information package) serve as a primary source of information for users, where the characteristics of the information conveyed can affect how device operators form an understanding of how to operate the device safely and effectively.44 Deficiencies in the body of information provided by manufacturers can lead to negative outcomes and poor operator performance,45 46 as related studies on medical devices and documents found that there is an increased risk for user errors, cognitive overload and adverse events associated with incomplete, obscure or confusing information.47–51 Subsequently, findings from a recent survey found that POCT analysts expressed a desire for more manufacturer assistance, including better training materials as well as understandable and standardised testing protocols.52
Consequently, the combination of questionable test performances coupled together with uncertainties in efficient utilisation can lead to misleading results that risks fuelling the loss of confidence in POCTs and negatively impacting the clinical uptake and utilisation of these testing devices.53 54 Unfortunately, regulatory markings granted to several POCTs, such as the CE mark in Europe, can create a false sense of security for potential users as the label is widely considered a poor indicator for test efficiency and effectiveness because obtaining it does not require a profound demonstration of high‐quality clinical data to receive regulatory approval.55 56
These issues imply that there is a pressing need to find the delicate balance between stringent evaluations, human factors, and the swift deployment of POTCs to efficiently scale up rapid diagnostic testing. Existing guidelines and standards for the evaluation for POCTs are not applicable to current COVID-19 testing kits, as these are ‘specific to a particular evidence domain or stakeholder group’.57 As such, the uncertainties surrounding the quality of POTCs and readiness for integration into real-world settings signals the need to generate stronger levels of evidence that also incorporate a human factors perspective to support the fast-track assessment of these tests during this pandemic. In short, there is a gap and need for for empirical knowledge concerning the quality of supportive information provided of commercial POCTs and their implications on efficacy, efficiency, and value in context-of-use in real-world environments.
Objective
We aimed to: (1) identify and assess high-priority areas that need addressing regarding the quality of information that commercial manufacturers provided for their POCTs and (2) examine the implications of these findings on real-world settings and use from a human-factors perspective.
Materials and methods
Study design
We used a content analysis methodology in two stages to systematically, comprehensively code and analyse textual data from documents of commercial manufacturers.58 This approach allowed us to scrutinise the textual data by examining language for meaning, identify recurring patterns or themes, and measure the frequency of categories reported.59
While not a typical literature synthesis, the Preferred Reporting Items for Systematic Reviews and Meta-Analyses criteria guided the conduct and reporting of the methods and results.60
Search strategy and data sources
Data search and gathering took place between 2 March and 13 June 2020 using the Google search engine to identify online resources and applied inclusion and exclusion criteria, data collection and synthesis methods. The following keywords were applied in various forms during the search strategy terms to locate websites that tracked the development and deployment of testing kits: ‘COVID-19’, ‘SARS-CoV-2’, ‘rapid diagnostic testing’, ‘lateral flow immunoassay’, ‘lateral flow assay’, ‘antibody testing’, ‘serology testing’, ‘point-of-care’, ‘immunoassay’ and ‘tracker’. We identified, selected and reviewed three websites until the end of our search period, which provided a comprehensive list of publicly available information on COVID-19 in vitro diagnostic medical devices (IVDs) for point-of-care testing (see box 1). These websites were the Foundation for Innovative New Diagnostics,61 the European Commissions ‘JRC COVID-19 In Vitro Diagnostic Devices and Test Methods Database’62 and the FDAs ‘Emergency Use Authorizations (Medical Devices)’.63 In addition, we conducted a weekly surveillance of online resources that provided updates regarding relevant diagnostic technology news, device regulatory changes that served as relevant leads to new POCTs developed for COVID-19 testing.
Websites with databases or comprehensive lists of commercial point-of-care tests for COVID-19
The Foundation for Innovative New Diagnostics (FIND) website provides a list of over +200 testing kits with varying regulatory status. Several of the testing kits are part of the self-submissions process by manufacturers as part of an expression of interest launched by FIND, which invited test developers to submit their testing kits for an independent evaluation that they are conducting in collaboration with the WHO, the Hôpitaux Universitaires de Genève and others.
The JRC COVID-19 In Vitro Diagnostic Devices and Test Methods database is hosted by the European Commission and contains publicly available information on CE-marked in vitro diagnostic medical devices, including elements of performance, and a collation of relevant scientific literature for coronavirus SARS-CoV-2.
The Food and Drug Administration website lists a table of test kit manufacturers, commercial and non-commercial laboratories who have received Emergency Use Authorizations. The website hosts publicly available documents for each authorised testing kit including the Letter of Authorisation, Fact Sheet for Healthcare Providers, Fact Sheet for Patients, and Manufacturer Instructions/Package Insert, and other documents.
Study selection
While systematically reviewing the three websites, we compiled a list of commercially available serology-based POCTs kits that were categorised or labelled as ‘point-of-care’ and/or ‘rapid diagnostic tests’. During this process, we cross-referenced each of the POCTs for overlap and removed all exact duplicates. We manually checked each POCTs and reviewed their respective manufacturers website to identify, mine and gather documents and materials for additional information. Where possible, we retrieved and downloaded all publicly available documentations (eg, product inserts, instructions for use and manuals) for the testing kits. If the documentation was not publicly available for download, we contacted the manufactures and/or distributors via email to request copies of the files in PDF form.
All POTCs were screened for inclusion and exclusion and any disagreements were resolved through discussion and consensus between the first two authors of this manuscript. The final list of commercial POTCs selected for this study was based on manufacturers responsiveness to our requests for information, publicly available documents, accessible information (in English) and regulatory status of the POTCs. Of note, all documents for the POCTs included in our analysis were provided free of charge and none of the manufacturers were involved in the assessment and interpretation of the results.
Exclusion criteria
POCTs that had not received CE-IVD, FDA (incl. FDA-Emergency Use Authorization (EUA)) or national body regulatory approval were excluded from this study. POCTs that were listed as ‘for research purposes only’ and ‘in development’ were excluded from the study. Documents that were not available in English were also not considered for further analysis. Finally, POCTs where it was not possible to retrieve any documentation via the website or through the manufactures were excluded from the study.
Data extraction, data analysis, and quality assessment
We used content analysis, which allowed us to apply both deductive and inductive analytic approaches in two stages.64 First, we used deductive approach to measure the frequency of manifest content (the surface-level content of the documents and does not require interpretation on the part of the coder) based on a predetermined categories and subcategories. Second, we used an inductive approach for a more detailed qualitative analysis to gain new insight into the meaning of the textual data within the key themes, as well as identify new themes or patterns.
Our deductive approach was guided by the multidimensional checklist, known as the Point-of-Care Key Evidence Tool (POCKET).65 This study has reported acceptable face and content validity for research of this nature. We opted to use the POCKET because of its customisability as a rapid assessment tool grounded in human factors design to support multidimensional evidence reporting of novel point of care diagnostics. The POCKET checklist is intended for real world application, with the aim to reduce the lead time for new POCTs to reach clinical practice. As a result, it should be periodically refined using rich data sources to prioritise the checklist statements, in order to guide the investment of evidence generation according to the device development stage.65
To guarantee its applicability for serology-based rapid diagnostic testing, we aligned the items of the POCKET with the specifications criteria defined in the Target Product Profile (TPP) issued by the British ‘Medicines and Healthcare products Regulatory Agency’ (MHRA) for serology POCTs and self-tests.66 We opted to align the POCKET with a TPP as they are strategic documents that summarise the desirable and minimally acceptable characteristics for a new test and communicate to key stakeholders and device developers the attributions that a new device is expected to conform to.67–70 Thus, we reviewed the MHRA’s preferred and minimally acceptable characteristics and directly mapped the applicable POCKET checklist items to it order to ensure one-to-one equivalence between both documents (see online supplemental file 1). A final version was developed for this study after piloting five randomly selected instructions for use documents.
Supplemental material
The main categories included: (1) clinical pathway; (2) technical description; (3) performance measurement and (4) usability and training. Box 2 provides an overview of the final version of the POCKET categories and subcategories that we used for our codebook for the first stage of our analysis.
Overview of the categories from the Point-of-Care Key Evidence Tool checklist that were used for coding during content analysis
Clinical pathway
This refers to statements relating to the current clinical pathway and the impact to the user and patient when a new test is introduced. For example, subcategories include: the intended use, indicated population, and setting of the test, the clinical pathway incorporating the new test, as well as the consequences of the correct and incorrect test result to the patient.
Technical description
This category broadly covers the technical requirements of the test, examples of which include; the test turnaround time, associated equipment required to perform the test, equivalence to a laboratory test and evidence of regulatory approval.
Performance measurement
This category relates to a brief understanding of the test performance based on the available data. For example, the subcategories included statements relating to the sensitivity, specificity and accuracy of the test result.
Usability and training
This category encompasses statements in relation to the use and associated training required for the test to be performed safely and effectively. Subcategories include, suggested standard operating procedure for the test, instructions appropriate to the end user, risks associated to the patient and personnel conducting the test, as well as the sample disposal procedure. Examples of the training element include training requirements for sampling, using the test and interpreting the test results.
We created a data matrix using Microsoft Excel 365, where each row corresponded to a commercial manufacturer to prepare and extract the data for analysis. The columns of the matrix formed the key descriptors that we wished to extract textual data and match them to the corresponding columns of the data matrix derived from the pre‐determined list of categories and subcategories from the POCKET checklist.
For the first stage of the analysis, documents and related materials from the manufactures were read by two authors (PK, AM) to gain a general understanding of the content and achieve data immersion. We then extracted, mapped and measured the findings from our readings against the descriptive indicators derived from the POCKET checklist. For instance, textual data such as, ‘The result should be read at 10–15 min. Do not interpret the result after 30 min’, would be extracted and mapped to the subcategory ‘turnaround time for a single test result’. Similarly, textual data such as ‘All specimens might be potentially infectious. Proper handling and disposal methods should be established. Wear protective clothing such as laboratory coats, disposable gloves and eye protection when specimens are being tested’, would be coded to the subcategory ‘List risks of the test procedure to personal performing the test’. Descriptive statistical analysis was used to assess the frequency of each code and synthesised into the categories established by the POCKET checklist.
For the second stage of our analysis, all extracted data were read line by line for comprehension. We selected relevant lines of text to be ‘coded’ or sorted into themes. For example, the following excerpt was coded under the ‘broad descriptors to identify intended users’ category: ‘for professional use only’, while ‘whole blood (from venipuncture or fingerstick), serum or plasma’ was coded as ‘definition of whole blood sample’. A process of iterative assessing, revising and testing were conducted for 10 randomly selected POCTs, and coding guidelines were confirmed. The findings from this analysis are presented using typical quotations to illustrate article content.
Overall, the type of data that we could extract was based on the availability of the information made available by the manufacturers. When we were uncertain or information was not available for certain criteria, we attempted to contact the manufacturers directly to ask for more details. If manufacturers were unable to assist us, we made a note in our database that the information was not available or not specified.
Two researchers (PK and AM) independently coded the data. Both researchers brought different disciplinary perspectives to the data (PK is an experienced qualitative researcher with a health services research and informatics background, AM is a biomedical engineer with experience in qualitative methods). They individually read the documents and undertook independent analysis for each type of POCT. The researchers extracted data and matched themes and concepts from the documents to the coding guide (see table 1).
Code guide for content analysis
To ensure interobserver consistency, differences in interpretation were resolved by consensus of the two reviewers through detailed discussion, or by referencing the developer’s original documents, rather than statistical calculation of levels of agreement. A total of six statements were discussed in detail to decide whether they should be included, excluded or modified for the codebook.
As several of the documents analysed were not available in the public domain but instead provided directly from the manufacturer and distributor, all identifying information concerning manufacturer names and their products have been anonymised and assigned a unique identifier.
Patient and public involvement
Patients or the public were not involved in the design, conduct, reporting or the dissemination plans of this research.
Results
Search results
An initial search and repeat searches until the 13 June 2020 revealed a total of 206 POCTs. After duplicates were removed, 202 testing kits were screened for inclusion with 66 excluded based on the predefined criteria. From the 136 POCTs, a further 57 were excluded after manufactures did not respond to our request for further information, opted to not provide us with relevant documents, or chose to send us non-relevant material (eg, brochures, flyers and product catalogue). See figure 1 for an illustrative overview of this process. In total, a thorough review of 79 POCTs was completed and included in this study.
Search strategy and results for point-of-care test kits considered eligible for inclusion in this study.
The commercial POCTs that were included for examination in this study were manufactured in 13 different countries. More than half of POCTs were developed in China (43/79), followed by the USA (10/79), UK (5/79), Republic of Korea (5/79), Germany (3/79), Singapore (3/79), Canada (2/79), India (2/79), Switzerland (2/79), Austria (1/79), Belgium (1/79), Netherlands (1/79), and Norway (1/79).
POCKET results
In the following section, we provide an overview of the characteristics that we found during our examined of the documents for the POCTs in accordance with the POCKET criteria (see table 2). See online supplemental file 2 for a complete overview of the test scores for the individual test kits evaluated in this study.
Supplemental material
Overview of the overall test scores for the characteristics that met the Point-of-Care Key Evidence Tool criteria
Category 1 (clinical pathway)
All manufacturers (79/79) instructions for use documents included a ‘test indication and function’ section, which typically contained general information concerning the technical purpose of the test. This section often encompassed key information stating that the test sought to measure immunoglobin in response to the SARS-CoV-2 virus. All documents (79/79) included the ‘sample type’ in the description of the intended use and specimen/sample collection procedure. Sample types were always categorised as blood, serum and plasma. Not all tests included a ‘description of the intended user’ of the test we found that most (70/79) of the documents referred to the intended user as a ‘professional’ or ‘medical worker’. Less than half (36/79) of the documents that we examined specified the ‘intended setting for the test’. When mentioned, the documents would state laboratories and medical institutions.
Few manufacturers (28/79) described the ‘indicated population’. In this event, individuals considered suitable for testing with the device would be mentioned as those who were displaying symptomatic conditions or were suspected infected patients. It was extremely rare (2/79) for any manufacturers to provide any information concerning ‘how the current clinical pathway will be changed by incorporating the test device’. Only two tests provided diagrams as to how they anticipated the tests would affect the triaging process.
Most of the documents (62/79) provided information concerning ‘the consequences of the test result to the patient’, and typically cautioned individuals that all results must be interpreted together with other clinical information and other test methods available. We observed that most (64/79) of the documents also included information concerning ‘consequences of an incorrect test result to patient’. These were often stated in the ‘limitations’ section warning users that a false negative result can occur if the antibody concentration of the tested sample is below the lower detection limit of the test.
Category 2 (technical description)
All tests (79/79) evaluated obtained the CE-IVD approval for market use in Europe. In addition, three tests also received FDA-EUA approval for use in the USA and one other test received national approval from India Central Drugs Standard Control Organisation. Several of the documents (53/79) used molecular test methods (eg, PCR) as the comparator device or ‘Gold Standard’, and no details were provided concerning any other equivalent laboratory test. Almost all (77/79) documents contained a description of ‘how tests results were presented to the user’.
Most documents (75/79) provided a ‘list of all associated equipment/consumables required to perform the test’. Composition of kits varied between manufacturers but mostly comprised sealed pouches containing a test cassette, a dropper and a desiccant, and buffer. The majority required additional components (such as timer, disposable gloves, blood collection device, safety lancets, alcohol prep-pad, etc) under the heading ‘material required but not provided’. Only a select few were complete kits that did not require additional components. Many of the documents (60/79) ‘listed the number of test kits per box’. These varied and typically ranged from 5, 10, 25 or 50 tests per kit. All (79) tests provided details concerning the ‘turnaround time for a single test result’. In most cases, users were instructed to wait 10–15 min and then read the result visually within 20 min.
Category 3 (performance measurement)
Most documents (65/79) provided details concerning test ‘sensitivity and specificity’. Presentation of these results varied from displaying matrix tables to a few sentences providing the numeral value. Some tests provided sensitivity and specificity scores separately for individual IgM and IgG, whereas others provided a combined IgM/IgG sensitivity and IgM/IgG specificity measurement. Following this, we also found that many tests (53/79) provided information concerning ‘analytical specificity’. It was often reported in the form of cross-reactivity, where manufactures stated that they tested positive samples with antibodies from other diseases, for example, parainfluenza virus antibody, influenza A virus antibody, influenza B virus antibody, adenovirus antibody, respiratory syncytial virus antibody and so on. Similarly, some (48/79) tests included details concerning testing with potential ‘interferences’ present in the blood. This included commonly used anticoagulants, mediations, some therapeutic drugs and commonly used consumables like coffee and alcohol were tested with negative and positive negative specimens.
Category 4 (usability and training)
Nearly all (78/79) of the tests contained a ‘standard operating procedure for test device and process’. Likewise, nearly all (78/79) were considered to have ‘instructions that were appropriate for the end user’. Most (78/79) documents provided ‘identification of operator dependent steps’, which were provided as numbered illustrative step-by-step processes and accompanying texts. ‘Potential risks of test procedure to patient’ was poorly addressed (4/79). In this case, the document warned of possible discomfort or other complications that can happen during sample collection. We found that most documents (70/79) included information concerning ‘potential risks of test procedure to personnel performing test’. Warnings typically cautioned operators to handle all specimens cautiously as if they contain infectious agents and to wear protective clothing such as laboratory coats, disposable gloves and eye protection. We also found that some (58/79) documents provided warnings concerning proper ‘sample disposal procedure including sharps’.
Some of the instructions (49/79) provided instructions concerning the ‘internal quality control protocol’. These were often listed in a separate section that stated that an internal procedural control is included in the test, with a coloured line appearing in the control line region (C) as an internal valid procedural control. Texts would usually caution the user to follow standard laboratory procedure and biosafety guidelines for the handling and disposal of potentially infectious specimens. Most documents (76/79) included guidance concerning ‘test device maintenance required, and level of expertise required to perform it’, more than often this involved instructions regarding the proper storage of testing kits and avoiding use beyond the expiration date.
Documents provided some guidance (55/79) pertaining to the ‘training requirements required for undertaking the sampling procedure’. Some (53/79) documents also mentioned the ‘training requirements for using the test device’. For example, some explicitly stated that testing should be professionally trained operators or trained clinical professionals working in certified labs or clinics. Similarly, some (53/79) documents mentioned the ‘training requirements for interpreting the test results’. There were occasions where the instructions informed users that laboratory personnel using the product must be appropriately trained in immunochromatographic techniques.
Qualitative findings
We identified seven themes relating to the textual data. The themes to emerge around were: (1) underlying purpose for testing; (2) broad descriptors of intended users; (3) identification of appropriate test setting; (4) definition of whole blood sample; (5) minimal comparator details; (6) information underlying test accuracy scores and (7) variations in cross-reactive substances and interfering substances details (see table 3).
Overview of qualitative findings
Manufacturers generally applied similar descriptions to explain what testing kits technically sought to achieve. Typically, they used phrases akin to ‘the qualitative detection of IgG and IgM antibodies to SARS-CoV-2’. We observed that many manufacturers did not often immediately build on this statement to clarify the underlying purposes of antibody in relation to diagnostic testing. For instance:
The <name of test> is an in vitro diagnostic test used for the detection of novel coronavirus (SARS-CoV-2) IgM and IgG antibodies (Manufacturer Test ID: M042)
<name of test> is used for qualitative detection of the IgM and IgG antibodies of COVID-19 in human serum/plasma or whole blood (Manufacturer Test ID: M026)
However, some manufacturers did provide additional text to supplement these types of statements, which explained the underlying intended use of the test. We also observed subtle differences describing the purpose of the testing kits, where the additional text indicated that the tests provided added value to diagnostics, where they served as ‘an aid’, ‘screening’ or ‘supplementary detection marker’ in the diagnosis of infection. In these examples, two manufacturers wrote:
The <name of test> is a rapid chromatographic immunoassay for the qualitative detection of IgG and IgM antibodies to SARS-CoV-2 in human whole blood, serum, or plasma as an aid in the diagnosis of primary and secondary SARS-COV-2 infections (Manufacturer Test ID: M073)
<name of test> is a supplemental tool to detect antibodies in patients suspected of Covid-19. (Manufacturer Test ID: M043)
We found that manufacturers applied broad descriptors to identify intended users. More than often, intended users were referred to as ‘professionals’. Typically, the statement ‘For professional in vitro diagnostic use only’, was included in the ‘intended use’ section of the instructions of use documents or it was mentioned in the ‘limitations’, ‘warning’ or ‘precautions’ sections. In most cases, no additional information was provided that helped define what qualified a user as a professional.
We did observe some situations where manufacturers did specify the expected occupations or training of the intended users. Here, the intended users were identified as professionals working in clinical laboratories or healthcare workers working in a medical institution. For example:
For professional and in vitro use only. For healthcare professionals and professionals at point of care sites. (Manufacturer Test ID: M069)
Most testing kits did not include information to clearly define the ‘appropriate test setting’ of where POCTs should be used in. However, we observed a tendency of where manufacturers who defined the professions of the intended users often included more specific details concerning the intended setting, which they referred to as ‘medical institutions’ and ‘laboratories’. For example:
Testing has to be done in a laboratory with proper testing conditions. All samples and materials in the testing process shall be handled according to the operation specifications of infectious diseases laboratories (Manufacturer Test ID: M060)
<name of manufacturer and test> is only provided for use by clinical laboratories or to healthcare workers for point-of-care testing, and not for at home testing (Manufacturer Test ID: M027)
Interestingly, POCTs that had received Emergency Use Authorization (EUA) from the FDA were very specific in terms of defining that the use of their kits was limited to CLIA laboratories. For example:
Testing is limited to laboratories certified under the Clinical Laboratory Improvement Amendments of 1988 (CLIA), 42 U.S.C. 263a, to perform moderate and high complexity tests. (Manufacturer Test ID: 025)
Some commercial manufacturers who specified the setting and intended users also mentioned a level of ‘expected professional training’ as part of the requirement to operate their testing kit.
Administration of the test and interpretation of results should be done by a trained health professional (Manufacturer Test ID: M083)
Testing should be applied by professionally trained staff working in certified labs or clinics (Manufacturer Test ID: M035)
We also observed several instances of ambiguity and variability around the ‘definition of whole blood sample’. Most manufacturers would generally define the sample type as ‘human serum, plasma or whole blood’. However, the term whole blood was also presented differently by a few manufacturers where fingerstick (or capillary blood) and venous blood was mentioned separate from whole blood or included them jointly under the terminology. For instance:
A rapid test for the qualitative detection of IgM and IgG antibodies to the SARS-CoV-2 in serum, plasma, venous whole blood, or capillary fingertip blood. (Manufacturer Test ID: M006)
The <name of test> is a colloidal gold enhanced, rapid immunochromatographic assay for qualitative detection of IgG and/or IgM to SARS-CoV-2 in human whole blood (venous and fingerstick), serum or plasma specimens (Manufacturer Test ID: M035)
Although most manufacturers mentioned at the start of manual that the test was human serum, plasma or whole blood, we noted some circumstances where a few manufacturers cautioned that the test were not to be used for fingerstick blood.
The <name of test> has not been evaluated with fingerstick specimens. Use of this test with fingerstick blood is not recommended. (Manufacturer Test ID: M038)
The kit is not intended for finger prick testing. (Manufacturer Test ID: M020)
Most performance measurements of the POCTs sensitivity and specificity were compared with molecular testing as the reference standard. However, ‘minimal comparator details’ of the molecular tests used were often presented within the product inserts as some manufacturers opted to describe the comparator as a commercial PCR without further detail as to how the molecular tests were conducted to confirm samples type and testing method. For instance:
The <name of test> was compared with a leading commercial PCR; the results show that <name of test> has high sensitivity and specificity (Manufacturer Test ID: M042)
The <name of test> was compared with a leading commercial PCR; the results show that <name of test> has a high sensitivity and specificity (Manufacturer Test ID: M062)
Overall, most manufacturers presented the results using an overall score for both the sensitivity and specificity to confirm test accuracy. However, we noticed that the level of information underlying these scores (eg, number of positive/negative samples and date of collection) varied significantly between manufacturers. For example:
All the 20 positive specimens were collected from hospitalized individuals who were clinically confirmed positive for SARS-CoV-2 infection. At the time of sample collection these individuals exhibited severe symptoms or they were in recovery stage. (Manufacturer Test ID: M037)
In order to test the detection sensitivity and specificity of this test, blood samples were collected from clinically diagnosed COVID-19 patients in Wuhan. A total of 272 cases were tested: 127 (positive) clinically confirmed patients and 145 non-infected patients (negative). The 127 positive patients were tested 7 days after being clinically diagnosed by PCR and CT. Among the 127 clinically confirmed samples, 125 were detected by the test reagents, with a positive detection rate (sensitivity) of 98.43%. Of the 145 clinically negative samples, 144 were detected by the test reagent, and the negative coincidence rate (specificity) was 99.31%. (Manufacturer Test ID: M028)
There were also a few manufacturers who provided specificity and sensitivity data based on ‘timeframe of disease onset’ with consideration for samples tested to reflect the infection stage and immune response. For example:
The clinical performance of the <name of test> was evaluated by testing a total of 196 clinical samples: 45 positive and 151 negative serum samples confirmed by RT-PCR. Positive samples consisted of 16 samples were collected 1~6 days after symptom onset, 27 samples were collected 7 days after symptom onset and 2 samples were asymptomatic. Out of symptomatic samples, clinical sensitivity was 93.8% (15/16) at 1~6 days after symptom onset and 96.3% (26/27) at 7 days after symptom onset (Manufacturer Test ID: M036)
The <name of test> is a lateral flow chromatographic immunoassay for the qualitative detection of anti-SARS-CoV-2 IgG and IgM in human whole blood, serum or plasma specimens of symptomatic patients (see section 12 ‘Limitations’). Note that in the early stages of infection (3 to 7 days after the onset of symptoms) anti-SARS-CoV-2 IgG and IgM may be below the detection limit of the test. This test is intended for use as an aid in the diagnosis of SARS-CoV-2 infections. The test procedure is not automated and requires no special training or qualification (Manufacturer Test ID: M054)
We noted that the information’s concerning ‘cross-reactive substances and interfering substances’ varied significantly between manufacturers. For example, some were more extensive with the list of cross-reactive substances when compared with another manufacturers kit:
The cross-reactivity was evaluated using serum or plasma specimen samples known to contain antibodies to HCoV-SARS, HCoV-OC43, HCoV-HKU1, influenza A and B virus, adenovirus, Staphylococcus aureus, or Klebsiella pneumoniae. No cross-reactivity was observed (Manufacturer Test ID: M011)
The <name of test> has no interference with infections that are non-COVID-19 based on validated tests performed on sera that are positive for IgM or IgG of the following pathogens; non-2019-nCoV strains (HKU1, OC43, NL63, and 229E), Influenza (A/H1N1-2009, A/seasonal H1N1, A/H3N2, A/H5N1, A/H7N9, B/Yamagata, B/Victoria), Respiratory Syncytial Virus (RSV), Rhinovirus (Group A, B, C), Adenovirus (Type 1, 2, 3, 4, 5, 7, 55), Enterovirus (Group A, B, C, D), Epstein-Barr virus (EBV), Rubeola Virus, Human Cytomegalovirus (HCMV), Rotavirus, Norovirus, Mumps Virus, Varicella-zoster Virus (VZV) and Mycoplasma Pneumoniae. (Manufacturer Test ID: M075)
Similarly, we observed that the data provided around interfering substances was either extensive or limited when comparing manufacturers.
No Interference for Respiratory Specimens (Mucin: bovine submaxillary gland type I-S, Blood (human), EDTA anticoagulated, Biotin), Nasal sprays (Neo-Synephrine, Afrin Nasal Spray, Saline Nasal Spray), Homeopathic allergy relief medicine (Homeopathic Zicam Allergy Relief Nasal Gel, Sodium Cromoglycate, Olopatadine Hydrochloride), Anti-viral drugs (Zanamivir, Oseltamivir, Artemether-lumefantrine, Doxycycline hyclate, Quinine, Lamivudine, Ribavirin, Daclatasvir), Anti-inflammatory medication (Acetaminophen, Acetylsalicylic acid, Ibuprofen), Antibiotic (Mupirocin, Tobramycin, Erythromycin, Ciprofloxacin), Human anti-mouse antibody, Pregnant woman, Elevated levels of C-reactive protein for IgM and IgG (Manufacturer Test ID: M061)
The test result of <name of test> do not be interfered with the substance at the following concentration: Substance Concentration Hemoglobin ≤ 10 g/L; Triglyceride ≤ 6 mmol/L; Bilirubin ≤ 1000μmol/L. No interference from rheumatoid factors, antinuclear antibodies and antimitochondrial antibodies (Manufacturer Test ID: M048)
Discussion
To our knowledge, this is the first analysis of commercial POCTs developed in response to the COVID-19 pandemic that applies a content analysis approach using a human factors perspective. Overall, we identified and uncovered several concerns regarding the quality of supportive information provided by commercial manufacturers. We found notable discrepancies between POCTs in terms of the types of information manufacturers provided, particularly the absence of details underlining testing kit performance and a general lack of human factors emphasis pertinent to optimal context-of-use.
Ambiguous and unspecific terms
First, the defining and descriptive attributes of POCTs needs to be communicated more clearly. While most manufacturers included the ‘type of test’ as part of their product nominal identity, we noted variabilities when manufacturers described the ‘intended use’ of the tests. Several of the documents did not address the purpose underlying the use, whereas others included accompanying texts describing that the tests can aid or supplement diagnosis. However, these statements are descriptive of the test function and do not explain that the primary purpose of serology-based tests is to identify individuals with previous infections and not to diagnose acute or recent cases of COVID-19.71–73 The absence of this information can lead to potential misunderstandings of when to optimally use serological tests if the underlying intended use is not evident.
There was some ambiguity concerning what constituted ‘whole blood’ in relation to the ‘type of sample’ required for testing. Variations with reference to the ‘definition of whole blood sample’ were frequent, where the distinction between finger-stick capillary blood and venous blood were rarely made clear and both types were lumped together under the umbrella of whole blood or presented a separate sample type with plasma and serum. This raises some concerns as results between finger-stick capillary blood and vein blood can vary depending on stage of clinical illness and method of collection.74 As finger-stick capillary blood and vein blood are not identical, it is important to bear in mind that these two sample types can lead to different results when measured on the same device,40 although two recent studies concluded that finger-stick capillary blood were comparable to venous blood samples.20 75 Given the mixed evidence concerning these samples, it is important that manufacturers disclose what type of whole blood samples were used during the test validation as well as inform decision-makers that the accuracies reported need to be interpreted with caution by virtue of the specific sample types used during the validation process.
We also noted that the ‘intended setting’ and ‘intended population’ were seldom specified. This has implications with respect to ensuring that the type of sample collected is prepared for use in the appropriate test setting and with the relevant population for testing. For instance, POCTs for capillary blood sampling may be more suitable and acceptable in non-clinical settings (such as nursing homes),76 as drawing venous whole bloods requires well-trained medical workers and are considered invasive and challenging in older adult populations.77 We also found that the ‘potential risks of test procedure to patient’ were poorly addressed. Considering that pain, anxiety and discomfort are associated with blood collection methods,78–80 efforts are needed to increase awareness of these risks to help test operators consider non-invasive pain and safety management strategies to mitigate distress and pre-procedural anxiety.81 82
Nearly half of the manufacturers listed that their POCTs were ‘for professional use only’, without specifying the conditions or referencing definitions set by medical and health regulators as to what professional use entails. Because the definition of what constituted ‘intended user groups’ was unclear, the broad interpretation of who should administer and manage the tests could introduce some confusion as there was little information to differentiate between the professional characteristics and anticipated skill-level required to operate the POCTs. Given that POCTs are prone to errors and robustness is dependent on the skill level of the individual operating the test,37 clear guidance concerning the prerequisite skill and training levels of the intended users should be made explicit in the documentation together with the training requirements for collecting samples, performing the test and interpreting the results. Also, there is a potential for users to operate the test incorrectly if these factors are not specified. For example, non-medically trained staff collecting blood samples could compromise the test results as ‘capillary blood can easily become diluted with tissue fluid if the puncture site is subjected to excessive pressure’,40 and tests administered by non-laboratory trained users can result in inaccuracies and potential biosafety issues.83
Fit for purpose and use context
Second, we identified several concerns related to the readiness of integrating these tests into real-world settings. The integration processes of implementing POCTs tests into existing ‘clinical pathway’ was lacking in detail. Determining these factors are imperative to ensure that devices are sustainable in the real-world context as existing workflow patterns and professional boundaries may be affected within the already complex and dynamic sociotechnical nature of daily clinical work.84 85 Workflow integration issues are critical to consider and failing to address it adequately can lead to low uptake and support.86 87 Medical devices that have been validated and evaluated in a controlled laboratory environment may not perform as well in the context of real-world settings.35 40 41 Given this, unanticipated challenges could arise from embedding POCTs into complex environments that can lead to usability problems when used outside highly controlled settings.88 Precautionary measures should be included in the documents that inform users of resultant potential impact relevant to integrating these POCTs into different testing facilities as well as help them anticipate potential workflow disruptions and consider appropriate implementation and adoption strategies.
We found variabilities between manufacturers in terms of providing ‘details of equivalent laboratory test’. Most manufactures offered ‘minimal comparator details’ when benchmarking the accuracy of their tests with RT-PCR methods. This method is fraught with several challenges as serology and RT-PCR-based tests fundamentally different. More so, it is even more difficult to draw comparisons as we observed an overall lack of information to help decision-makers assess the quality of the molecular comparator device. For example, it was generally unclear where the validation samples were sourced from, the type of sample used, when in the course of COVID-19 infection each sample was taken, which RT-PCR assay was used as a gold-standard, where RT-PCR testing was performed, and the nature of antigens used.89–91 Consequently, the accuracy of RT-PCR still raises questions as it is prone to producing false negative and positive results for a variety of reasons such as mismatches between the testing primers and viral genome, different viral load kinetics in different anatomic sites, sampling procedures and timing of disease.17 92
We observed several cases where the accuracy scores did not take in account the ‘timeframe after disease onset’. As such, the ‘interpretation of results’ could be negatively impacted if users are not presented with different levels of performance characteristics based on the window period for antibody tests. For example, higher levels of detectable levels of antibody are available in the second and third week of symptom onset.93–95 Testing too early may lead to a false negative result if performed in the early stages of infection as a result of low antibody concentrations, while testing too late can mean IgM antibodies have decreased.90 96 These factors suggest that test performance data presented by manufacturers need to be stratified by the time of onset of symptoms to properly reflect the relationship between disease stage the detection of viral antibody. A recent Cochrane review asserted that that timing is important, where detection is more accurate in people two or more weeks after their symptoms started.97 In relation, any attempts to compare PCR- and serology testing needs to compare samples taken during the different stages of infection (early vs late) and the immune response (convalescence period).90 Furthermore, comparing results with other serology POCTs may be moot and not reflective of testing efficiency, as studies have found variable performances between kits and high rates of false-positives.98–101
We also noted discrepancies between manufacturer lists of preanalytical issues potentially affecting the sample collected. Testing for ‘cross-reactive and interference substances’ varied considerably between manufacturers. Some listed a few items whereas others were more extensive with their listings. There is an inherent risk for tests to produce false positives results if other antibodies have already been generated against other coronaviruses and influenza viruses,102 103 or interfering substances are present in the specimens.104 A standardised and extensive list of substances during test validation should be followed by manufacturers in accordance to a target product profile. Informing intended users and test recipients of these results can increase transparency as to how robust the test is against cross-reactive and interference substance.
Regulatory concerns
Third, there are significant implications for policies related to medical device regulation. All of the POCTs were already approved for market dissemination in Europe as they had received the European CE-IVD marking, whereas only a handful of lateral flow immunoassay serology tests received EUA from the US FDA. This highlights the differences in regulatory control measures between two major medical devices markets. Considering the number of high-profile cases, the procurement of inaccurate POCTs may reflect these differences,105 as the CE-mark does not presuppose a demonstration of clinical data relating to effectiveness of medical devices.106 This raises questions regarding the reported accuracies provided by manufacturers. Independent evaluations of CE-labelled devices have found performance characteristics to be significantly lower to that reported by commercial manufacturers.89 96 107 Inaccurate tests can lead to incorrect results and a resultant increase in the risk of community transmission.108 These issues amplify concerns surrounding the diagnostic accuracy and lack of confidence in the tests results for POCTs.109–111 The COVID-19 pandemic has further illuminated the existing shortcomings of current European regulations, bringing to light the need for more market vigilance, transparency and validation by accredited laboratories to evaluate the premarket and postmarket evidence, at least until the adoption of the new In Vitro Diagnostic Medical Device Regulation (2017/746) takes effect on 26 May 2022.91
Overall, the application of the POCKET has yielded important lessons for implementing the checklist as a guide to rapidly assess POCTs serology tests for COVID-19. It demonstrated its usefulness in terms of heuristically identifying whether commercial manufacturers have provided necessary supportive information. However, it is important to bear in mind that the POCKET is neither a ‘quick-fix’ nor a tool that can be effectively used to thoroughly screen the quality of POCTs. It is unable to evaluate broad assumptions encoded into the language of supportive information. This implies that the optimal use of the POCKET requires an inductive content analysis to examine assumptions and assess the quality and value of the information reported. The POCKET checklist should be viewed as a practical support tool for assessment during the early premarket and design phases of POCTs prior to evaluate market readiness and could potentially help streamline standards for medical device regulation concerning information transparency and validation.
Limitations and strengths
Study limitations were that the data retrieved was dependant on what the manufactures were prepared to disclose. Also, our analysis is quite specific to the topic of serological point-of-care testing in relation to rapid diagnostic testing, and thus it was possible that we overlooked important conversations around topics of innovative POCTs that are based on molecular and antigen-based techniques. In addition, the landscape surrounding POCTs is fast evolving with a rapidly expanding range of POCTs approved by regulatory agencies. Thus, this study may not represent the most current landscape of testing kits by time of publication. Our exclusion methodology for POCT, which included eliminating testing kits that had not yet received regulatory approval but were available under the classification ‘for research use only’, may have led to an underestimation of the true number of available POCTs. However, we believe that by limiting our study to testing kits that have received regulatory approval, our findings remain useful to gain a general understanding of the POCT landscape for use in clinical and potential non-clinical settings. Additionally, we were unable to perform an economic evidence evaluation as manufacturers did not provide the pricings for their testing kits. Finally, some of the studies we referenced in our discussion section that analysed different testing kit performances were retrieved from a preprint platform (eg, medrxiv.org).
Despite these limitations, there were many strengths of this study. Our study accessed data from 79 commercially approved POCTs and the analysis was carried out over a critical period during the expansion of rapid diagnostic testing landscape for COVID-19. Future studies may be interested in using less restrictive relevance criteria based on molecular and antigen-based devices as well as explore manufacturers reasonings for including and excluding specific types of information in their documents. Our study raises important public health issues related to the expansion of POCTs to non-expert users,112 113 where potential misunderstandings of can be averted with strategies designed to improve the link between health literacy and testing.114 This is a particularly pressing issue in light of existing health inequalities, where groups of lower socioeconomic and specific ethnic minorities are disproportionately affected by the burden of low health literacy and associated with severe disease and mortality and experience worse health outcomes.115–117
Conclusion
Point-of-care testing can significantly help improve the scaling-up of testing capacities for COVID-19. Despite the rapid developments in this field, there are significant concerns surrounding POCTs in terms of both performance and readiness for use. For the former, information transparency was generally not evident in terms of reporting how tests were internally validated, and accuracies assessed against comparator devices. For the latter, the frequently broad use and lack of definitions is a source of confusion that can increase risks of the tests being inappropriately operated. Commercial manufacturers need to improve the quality of information they provide for POCTs. The POCKET checklist can help guide this process. Addressing these issues will positively contribute to the evidence-base of point-of-care testing for COVID-19.
Acknowledgments
Thank you to Drs Massimo Micocci and Melody Ni of the NIHR London In Vitro Diagnostic Co-operative (IVD) group at Imperial College London who helped provide valuable feedback during the writing of this paper. Thank you to the clinicians and laboratory technicians who helped provide us with information in terms of identifying important device specifications criteria for rapid point-of-care tests.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Contributors PK drafted the first iteration of the manuscript. AM contributed to the manuscript preparation and editing. PB provided valuable feedback and contributed to the editing. All authors reviewed the final version of the manuscript for content and contributed to the conclusions of this manuscript.
Funding This research was supported by the National Institute of Health Research (NIHR) In-Vitro Diagnostic Co-operative London at Imperial College Healthcare NHS Trust. The views and opinions expressed by authors in this publication are those of the authors and do not necessarily reflect those of the NHS, the NIHR or the Department of Health. Award/grant number: N/A.
Competing interests None declared.
Patient consent for publication Not required.
Ethics approval The study did not involve active treatment of human participants, ethics review and approval was not necessary because all the material was publicly available or voluntarily provided by the manufacturers or authorised distributors.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement Data are available upon reasonable request. Data are available upon reasonable request.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.