Article Text
PDF
Abstract
Objectives To undertake a systematic review and meta-analysis to evaluate the test performance including sensitivity and specificity of rapid immunochromatographic syphilis (ICS) point-of-care (POC) tests at antenatal clinics compared with reference standard tests (non-treponemal (TP) and TP tests) for active syphilis in pregnant women.
Methods Five electronic databases were searched (PubMed, EMBASE, CRD, Cochrane Library and LILACS) to March 2016 for diagnostic accuracy studies of ICS test and standard reference tests for syphilis in pregnant women. Methodological quality was assessed using QUADAS-2 (Quality Assessment of Diagnostic Accuracy Studies). A bivariate meta-analysis was undertaken to generate pooled estimates of diagnostic parameters. Results were presented using a coupled forest plot of sensitivity and specificity and a scatter plot.
Results The methodological quality of the five included studies with regards to risk of bias and applicability concern judgements was either low or unclear. One study was judged as high risk of bias for patient selection due to exclusion of pregnant women with a previous history of syphilis, and one study was judged at high risk of bias for study flow and timing as not all patients were included in the analysis. Five studies contributed to the meta-analysis, providing a pooled sensitivity and specificity for ICS of 0.85 (95% CrI: 0.73 to 0.92) and 0.98 (95% CrI: 0.95 to 0.99), respectively.
Conclusions This review and meta-analysis observed that rapid ICS POC tests have a high sensitivity and specificity when performed in pregnant women at antenatal clinics. However, the methodological quality of the existing evidence base should be taken into consideration when interpreting these results.
PROSPERO registration number CRD42016036335.
- syphilis screening
- maternal syphilis
- pregnant women
- rapid test
- systematic review
- meta-analysis
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Strengths and limitations of this study
This systematic review addresses the performance of rapid tests to tackle significant public health problems that are maternal and congenital syphilis.
The systematic review was conducted and presented in accordance with recognised standards for undertaking and reporting systematic reviews and
Methodological quality of studies was assessed using a validated method.
Data were synthesised using a bivariate random-effects meta-analysis model, implemented in a Bayesian framework.
There were insufficient data from the included studies to undertake subgroup analyses.
Introduction
Syphilis is a chronic infectious disease, caused by the spirochete bacterium Treponema pallidum, transmitted sexually (acquired syphilis) and vertically (congenital syphilis) through the mother’s placenta to the fetus.1–3 Syphilis is among the high-risk conditions during pregnancy with possible serious birth outcomes if untreated or inadequately treated.1 4
WHO estimates that syphilis infects 1% or more of antenatal care attendees in over 55 countries. Approximately 950 000 pregnant women were infected with syphilis in 2012,5 primarily in the low-income and middle-income countries, sub-Saharan Africa, Southeast Asia and Latin America and the Caribbean (LAC). The prevalence of gestational syphilis in LAC varies from 0.08% to 7.0%6 7 between countries, and accounts for up to 25% of the 2 million annual cases of gestational syphilis in these regions.6 7 Annually, an estimated 100 000 stillbirths in these regions are attributable to congenital syphilis.6 7
Diagnosis of syphilis is complicated due to extremely difficult to cultivate T. pallidum in vitro. Clinical diagnosis, visualisation of treponemes in lesion material using dark-field microscopy or immunofluorescent staining and serological reactions are used for diagnosis. Nevertheless, serological tests are a mainstay of syphilis diagnosis.8 9
Syphilis can be diagnosed serologically by using a non-treponemal (non-TP) test for screening and a TP test for the confirmation of results. TP tests are more sensitive and more specific than non-TP tests, but they cannot distinguish between active and past infection, as antibodies remain circulating during life after treatment. Non-TP tests provide high sensitivity but not for the initial stage of primary syphilis or late period of disease (tertiary syphilis).8–10
In low-income and middle-income countries, many prenatal clinics that provide screening and treatment for syphilis do not have the capability to perform confirmatory diagnostic tests, and so testing is frequently done off-site.11 Furthermore, patients may fail to return for laboratory results and both specimens and results have the potential to be lost in transit, resulting in treatment being delayed or missed.
Currently, new tests using cloned TP antigens and an immunochromatographic technique provide an alternative platform of rapid TP testing to be performed at point-of-care (POC), this is the immunochromatographic syphilis (ICS) test. The ICS test takes approximately 20 min to perform and is generally considered fairly sensitive and specific.8 In addition, the format of the test does not require the use of special equipment such as fluorescence microscopes or microplate readers, thus making it an ideal test for POC settings.12
A rapid syphilis test with immediate results implemented in a field setting has the advantage of allowing women who test positive to be treated on-site at the same visit, avoiding losses to follow-up for return visits and potential adverse outcomes associated with delayed treatment.13 According to the Special Programme for Research and Training in Tropical Diseases (WHO),14 the ideal characteristics of POC tests are that they are affordable, sensitive, specific, user- friendly, rapid and robust.15 On-site screening services can contribute to significant declines in the prevalence of maternal and congenital syphilis in many low-income and middle-income countries,16 where operational research on POC syphilis testing is already underway.17
This systematic review and meta-analysis of diagnostic test accuracy aims to evaluate the performance of rapid ICS POC test at antenatal clinics compared with reference standard tests (non-TP and TP tests) for detecting active syphilis in pregnant women. Furthermore, in comparison to previous reviews by Tucker et al 17 and Jafari et al 18 that included studies among women attending sexually transmitted infection (STI) clinics and antenatal clinics, the present systematic review includes only studies among pregnant women in antenatal care settings, that compare ICS rapid test with the algorithm established as reference standard (non-TP, following by TP test)10 used in many low-income and middle-income countries.
Methods
Protocol and guidelines
The protocol for this review can be found at PROSPERO 2016: CRD42016036335, available from http://www.crd.york.ac.uk/PROSPERO/display_record.asp?ID=CRD42016036335.
The systematic review was conducted and presented in accordance with the general principles recommended in the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement.19
Search strategy
PubMed, EMBASE, CRD, Cochrane Library and LILACS (Literatura Latinoamericana en Ciencias de la Salud) were searched initially from January 1960 to end March 2016 and were updated to end August 2017. The following keywords were used: ‘pregnancy diseases; maternal syphilis; gestational syphilis; congenital syphilis’ and ‘diagnostic; screening; rapid test; serological test(s); point-of-care’ and ‘accuracy; sensitivity, specificity’. The update search also included the term ‘antenatal’. Full search strategies for PubMed and EMBASE databases are presented in online supplementary appendix 1. Filters for diagnostic studies recommended by Information Specialists’ Sub-Group Search Filter Resource (https://sites.google.com/a/york.ac.uk/issg-search-filters-resource/) were used. No language restrictions were applied to the search. Reference lists of all included studies were also searched.
Supplementary file 1
Inclusion criteria for studies
Diagnostic test accuracy studies were included if they evaluated ICS POC tests for syphilis, compared with a standard reference test in pregnant women who receive antenatal care. We only included studies where the standard reference test comprised a non-TP test (Venereal Disease Research Laboratory (VDRL) or rapid plasma reagin (RPR)) followed by a TP test (T. pallidum haemagglutination test (TPHA), or fluorescent treponemal antibody absorbed (FTA-Abs), or T. pallidum particle agglutination (TPPA)). The standard reference test constitutes a non-TP test, followed by TP test where the non-TP test is positive. We also only included studies reporting test performance data (true positive, false positive, false negative, true negative) from which to calculate sensitivity and specificity, or other information from which we were able to calculate performance data. Dual tests such as HIV/syphilis tests and TP/non-TP POC tests were not included in this review.
Study selection process
All citations were downloaded to Mendeley reference management software (https://www.mendeley.com/newsfeed/). Titles and abstracts were sifted by one reviewer (CNPRC) to identify potentially relevant articles. All potentially relevant articles obtained as full text were checked for inclusion by two reviewers (CNPRC, SH).
Data extraction
Data were extracted by CNPRC using a prepiloted data extraction form in Excel and then checked by two independent reviewers (RC and DSM). Any discrepancies were resolved by a third reviewer (MMS-J or SH). Details of study characteristics, population characteristics, test descriptions and performance data outcomes (true positives, false positives, false negatives and true negatives; plus sensitivity and specificity) were extracted from the included studies.
Assessment of methodological quality
The quality of evidence was assessed and described using the Bristol University QUADAS-2 (Quality Assessment of Diagnostic Accuracy Studies) tool to analyse risk of bias.20
QUADAS-2 is structured in four domains of potential sources of bias in primary diagnostic studies: ‘patient selection; index test; reference standard and flow and timing’. There are signalling questions within each of these domains, which drive researchers to overview the potential sources of bias.20
Data analysis
The extracted data and quality assessment variables were presented for each study, both in structured tables and as a narrative description. The combined sensitivity and specificity was estimated using the bivariate random-effects meta-analysis model described by Reitsma et al.21 The data synthesis was conducted under the assumption that different specific implementations of ICS rapid tests (eg, by different manufacturers) can be considered equivalent. Therefore, there is essentially one index test only. A random-effects model was used to allow for heterogeneity between studies that is generally expected in studies of diagnostic test accuracy,22 which in this case will include heterogeneity due to differences in the specific index test used. Further details of the statistical model are provided in online supplementary appendix 2. Results were displayed as forest plots and scatter plots with 95% credible intervals (CrIs) and 95% prediction intervals (PrIs) for sensitivity and specificity.
Analyses were conducted in the R software environment23 using WinBUGS24 and R2WinBUGS software packages.25 Convergence to the target posterior distributions was assessed using the Gelman-Rubin convergence statistic.26 A burn-in of 10 000 iterations of the Markov chain was used with a further 10 000 iterations retained to estimate parameters.
Results
Study selection process
The results of the study selection process are illustrated in figure 1 using a PRISMA flow diagram.19 Following deduplication, the searches identified 484 citations of which 28 were obtained for screening. Of these, 23 full-text publications were excluded. A table of these excluded studies with reasons for exclusion is presented in online supplementary appendix 3. Five diagnostic accuracy studies were included in the review.
Study selection following Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines.19
Characteristics of the included studies
The characteristics of the five included studies, including the study setting, study design, details of included participants and details of the ICS and non-TP/TP tests are presented in online supplementary appendix 4.
All of the included studies were prospective, single-gate (a study design in which only suspected patients for the target condition are recruited) diagnostic accuracy studies. Four studies reported consecutive sampling.27–30 The sampling method was not reported in one study.31 All included studies were undertaken in antenatal clinics27–29 or hospitals.30 31
All the studies were undertaken in low-income and middle-income countries (Brazil, South Africa, Mexico, India, Mozambique and Bolivia) according to the World Bank country classification.32 Two were undertaken in urban health centres30 31 and three at rural sites.27–29
Across the included studies the index tests used were ICS tests. Two studies28 30 used the Determine Syphilis TP test (Abbott Laboratories, Abbott Park, Illinois, USA), two studies29 31 used the SD Bioline V.3.0 test (Standard Diagnostics, Kyonggi-Korea) and one study27 used the VisiTect Syphilis test (Omega Diagnostics, Alloa, Scotland).
These ICS tests detect IgM, IgG and IgA antibodies and involve ICS in which one or multiple T. pallidum recombinant antigens are applied to nitrocellulose strips as a capture reagent. Antibodies in the specimen bound at antigen site on the strip and are revealed with dye bound anti-immunoglobulin or dye bound antigen and a positive reaction appears as a coloured line.9 The individual tests are also very similar in the assay procedure, taking from 20 to 30 min to get the result.
The age of participants was reported by three included studies,27 29 31 in which all women were in the third decade.
Across the five studies in pregnant women, three studies reported that participants were in the second trimester27–29 and one study reported that 16%, 32% and 52% of participants were in the first, second and third trimester, respectively.31 One study in pregnant women did not report the stage of pregnancy.30
Obstetric characteristics were reported by two studies27 29 identifying: previous syphilis (n=4; 2.1%)27 and n=318; 7.1%),29 previous treatment for syphilis, including penicillin (n=51; 7.5%), a previous syphilis diagnosis (n=318; 7.1%)27 and prior partner syphilis history (as reported by the pregnant women) (n=228; 5.3%).29 Three studies did not report obstetric characteristics.28 30 31
For the Bronzan et al 28 study, we analysed on-site ICS results for all the study participants (n=695) instead of on-site ICS results after training on how to properly obtain fingerstick blood (n=341). This study reported sensitivity and specificity both before and after nursing training on how to properly obtain fingerstick blood, (31.3%–85.7% and 94.8%–90.9%, respectively).28 None of the other included studies reported whether nurse training on how to properly obtain fingerstick blood was provided or not. As the inclusion criteria for studies, specified in the review protocol for this review, were not based on whether training was provided or not, it was considered most appropriate to include results over the whole study period. Meta-analysis results including data from the Bronzan et al 28 study after retraining on how to properly obtain fingerstick blood are also displayed for comparison.
In the Kashyap et al 31 study, all patients received both VDRL and TPHA tests, however, TPHA was the confirmatory test, and it was not clear how VDRL-negative/TPHA-positive tests were interpreted. The authors were contacted for clarification, but without response to date.
Methodological quality of included studies
The overall methodological quality (risk of bias and applicability concerns) of the five included studies assessed by QUADAS-2 is summarised in figure 2. Kashyap et al 31 was judged as high risk of bias and high concern for patient selection as the study excluded pregnant women with previous history of syphilis. Montoya et al 29 was judged at high risk of bias for study flow and timing as not all patients were included in the analysis. The remaining risk of bias and applicability concern judgements across studies were either low or unclear.
Risk of bias and applicability concerns summary.
Results reported by included studies
The diagnostic accuracy data reported by the five included studies are presented in table 1.
Diagnostic accuracy data reported by included studies
Results of the meta-analysis
Five studies (14 985 participants) contributed to the meta-analysis. Figure 3 shows the sensitivity and specificity for each study, as estimated by the model. Note that the modelled estimates of sensitivity and specificity for each study are shrunken towards the overall mean, and hence differ from the raw estimates presented in table 1. The pooled sensitivity and specificity of ICS POC test compared with a standard reference test were 0.85 (95% CrI 0.73 to 0.92) and 0.98 (95% CrI: 0.95 to 0.99). The between-study SD for sensitivity and specificity were estimated to be 0.58 (95% CrI: 0.33 to 1.15) and 0.71 (95% CrI: 0.42 to 1.47), respectively. The correlation between logit sensitivity and specificity was not statistically significant 0.42 (95% CrI: −0.46 to 0.88). The 95% PrIs of 0.57 to 0.96 (sensitivity) and 0.88 to 1.00 (specificity) suggest moderate uncertainty in predicting the sensitivity and specificity of a new study. Figure 4 presents the joint distribution for sensitivity and specificity.
Sensitivity and specificity of ICS point-of-care compared with a standard reference test. FN, false negative; FP, false positive; ICS, immunochromatographic syphilis; PrI, prediction interval; TN, true negative; TP, true positive.
Joint sensitivity and specificity plot of immunochromatographic syphilis point-of-care compared with a standard reference test.
Two sensitivity analyses were conducted. Assuming that missing test results in Benzaken et al 27 were negative (rather than positive) reduced the pooled estimate of the sensitivity only slightly to 0.84 (95% CrI: 0.71 to 0.92) (online supplementary figure 1). The impact of retraining was assessed by including the data after training only from Bronzan et al,28 resulting in slightly higher sensitivity of 0.88 (95% CrI: 0.78 to 0.93) (online supplementary figure 2). See online supplementary appendix 5.
Supplementary file 2
![[bmjopen-2017-018132-SP2.jpg]](https://bmjopen.bmj.com/content/bmjopen/8/2/e018132/DC2/embed/inline-supplementary-material-2.jpg?download=true){kind=link}
Supplementary file 3
![[bmjopen-2017-018132-SP3.jpg]](https://bmjopen.bmj.com/content/bmjopen/8/2/e018132/DC3/embed/inline-supplementary-material-3.jpg?download=true){kind=link}
Discussion
This systematic review of the evidence for the diagnostic accuracy of TP antibody-based ICS tests among pregnant women with active syphilis included five studies (14 985 participants). Although the ICS tests employed by the included studies are produced by different manufacturers, they are all similar in their implementation. As such, in order to assess the accuracy of ICS tests, estimates from all tests were pooled together in the meta-analysis.
We used an established algorithm of POC without confirmatory non-TP as a reference standard where a positive result indicates that immediate treatment is required for pregnant women.9 The meta-analysis was conducted under the assumption that the reference standard is 100% sensitive and specific.
The meta-analysis suggested a high average sensitivity 0.85 (95% CrI: 0.73 to 0.92) and specificity 0.98 (95% CrI: 0.95 to 0.99) of the ICS tests, compared with standard tests (non-TP test followed by TP test where the non-TP test is positive). The high sensitivity of the ICS test indicates that it might be a reasonable initial screening test for syphilis, but subsequent clinical management would require a non-TP test. Overall, we observed that rapid and POC tests performed well in both sensitivity and specificity compared with laboratory-based TP-specific tests such as TPPA and TPHA that have sensitivity in the range of 85%–100% and specificity of 98%–100%.33
The risk of bias and concerns over applicability of the five studies was either low or unclear. One study was judged as high risk of bias and high concern for patient selection as the study excluded pregnant women with previous history of syphilis.31 Another study was judged at high risk of bias for study flow and timing as not all patients were included in the analysis.29 The contribution of these studies to the pooled analysis should therefore be taken into consideration. A major limiting factor in assessing methodological quality was unclear or incomplete reporting by authors.
We undertook our review according to PRISMA guidelines, searched a wide range of data sources, undertook double data checking and assessed the methodological quality of the included studies using QUADAS-2. However, the presented review also has limitations. Although HIV status, clinical stage of syphilis infection and syphilis titre might influence ICS test performance,8 there were too few included studies, and insufficient data available from the included studies for us to explore these potential sources of heterogeneity. In one of the included studies,31 it was also not clear how VDRL-negative/TPHA-positive tests were interpreted, which could potentially influence the performance results.
The present review only included studies that used the conventional algorithm of dual reference standard (non-TP and TP tests) as a reference standard. This is in contrast to the review by Tucker et al 17 that included studies of one TP test or an additional non-TP test and the review by Jafari et al 18 that included studies of TP tests, non-TP-specific tests or TP and non-TP-specific tests (the inverse algorithm) but only presented a meta-analysis of studies evaluating a TP-specific reference standard test.
In comparison with the previous review by Tucker et al 17 that included studies in women including sex workers attending STI clinics as well as antenatal clinics, the present review only included studies in pregnant women attending at any care settings (antenatal care settings, primary care settings, maternity hospital). In comparison with the previous review by Jafari et al 18 that included studies in women attending antenatal clinic and sexually transmitted disease (STD) clinics. These differences in included studies are due to the main objective of the present review being to investigate the performance of on-site tests as part of prevention strategy of mother-to-child transmission of syphilis.
The review by Tucker et al 17 reported that although there were variations in ICS test sensitivity across the included studies, that this variation was similar to sensitivity noted in traditional non-TP testing where sensitivity ranged from 78% to 100%.17 From the meta-analysis in the review by Jafari et al,18 the authors observed that rapid and POC TP tests had varied sensitivity (74.26%–86.34%)) and high specificity (95.85%–99.43%) (serum and blood) that were equal or better than laboratory-based non-TP tests. Jafari et al 18 synthesised results from individual studies separately for each index test, using a Bayesian hierarchical summary receiver operating characteristic curve model and adjusted for imperfect reference standards.
Another difference between ours and these previous reviews is that for the Bronzan et al 28 study (included in the reviews by Tucker et al 17 and Jafari et al),18 we used the on-site ICS results for all the study participants (n=695), which was considered to be more consistent with the results of the other included studies, rather than the on-site ICS results after training on how to properly obtain fingerstick blood (n=341).
Our searches identified two studies27 31 that were published after the searches of previous reviews in the area.17 18
The update search also identified the recent systematic review and meta-analysis by Rogozińska et al,34 which included two additional studies evaluating a qualitative RPR test as the index test.35 36
There were several differences in the QUADAS assessment reported by Rogozińska et al 34 compared with the present systematic review. Notable differences were that Kashyap et al 31 was judged as high risk of bias and high concern for patient selection in the present review (unclear and low risk in Rogozińska et al),34 and that Montoya et al 29 was judged at high risk of bias for study flow and timing in the present review (low risk in Rogozińska et al).34 Two reviewers were involved in the QUADAS assessment in Rogozińska et al,34 whereas three reviewers (CNPRC, SH and MM-SJ) were involved in the present review.
The meta-analysis conducted by Rogozińska et al 34 differed to that presented in the current review. Separate meta-analyses were conducted according to the specific ICS test implementation (Determine, VisiTect, SD Bioline). Since a frequentist estimation framework was used, this meant that there were too few studies to fit a bivariate model in the case of two of the ICS implementations (Determine, VisiTect) and so separate meta-analyses were conducted for sensitivity and specificity. For the current review, the synthesis was conducted under the assumption that different specific implementations of ICS rapid tests can be considered equivalent, and therefore combined in single meta-analysis model. The resulting pooled estimates represent the sensitivity/specificity of a general ICS test.
Public health strategies to prevent and treat maternal and fetal syphilis include early identification of infected individuals and high-risk populations, adequate treatment, identification of infected partners and their treatment, modification of high-risk behaviour and promoting the accessibility to and the use of healthcare.1 However, access to testing and implementing screening programmes varies significantly between countries and between rural and urban areas. ‘In primary healthcare settings, there are often technical difficulties associated with performing serological tests, such as retaining trained personnel and assuring that supplies are adequate and tests are of good quality’.8 Additionally, if pregnant women fail to return for laboratory results or they are lost in transit, the treatment is delayed or missed.
Conclusions
This systematic review and meta-analysis observed that rapid ICS POC syphilis tests have a high sensitivity and specificity when performed in pregnant women at antenatal clinics. However, the methodological quality of the existing evidence and clinical heterogeneity of participants across studies should be taken into consideration.
Further diagnostic test accuracy of ICS POC tests at antenatal clinics compared with reference standard test that are reported in line with Standards for Reporting of Diagnostic Accuracy Studies37 would be of benefit, along with studies of duplex TP and non-TP rapid and POC tests. Studies evaluating the diagnostic test accuracy of STD coinfections such as HIV/Syphilis would also be of benefit. Considering the global efforts to eliminate congenital syphilis, the free access to simultaneous screening for these infections during pregnancy, through separate or dual tests, is crucial to tackle mother-to-child transmission of HIV and syphilis. Further studies that estimate the coverage of screening programmes would also be beneficial to help reduce the reservoir of infections in communities to a level at which disease transmission cannot be sustained.
Acknowledgments
Special thanks for Suzy Paisley and Simon Dixon, both at School of Health and Related Research (ScHARR), University of Sheffield, for their helpful support during the scope of this review.
References
Footnotes
Contributors CNPRC, SH and MM-SJ developed the protocol. CNPRC, DSM and RC searched for the evidence and extracted the data. Data were checked by SH and MM-SJ. CNPRC, SH and MM-SJ undertook the quality assessment. JH undertook the statistical analysis. All authors were involved in preparing the final manuscript. CNPRC, DSM, SH, MM-SJ and JH addressed the peer-review comments.
Funding Funding was provided by the National Council for Scientific and Technological Development (CNPq), Ministry of Science, Technology, Innovation and Communication (no. 573642/2008-7; Ministry of Education/CAPES (no. 573642/2008-7); Research Foundation of the State of Rio de Janeiro (FAPERJ) (no. 573642/2008-7) and Oswaldo Cruz Foundation (Fiocruz). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests None declared.
Patient consent Not applicable.
Provenance and peer review Not commissioned; externally peer-reviewed.
Data sharing statement No additional data are available.