Accuracy of rapid lateral flow immunoassays for human leptospirosis A systematic review and meta-analysis

Authors: Teerapat Nualnoi, Luelak Lomlim, Supawadee Naorungroj

Abstract

Background

In the last two decades, several rapid lateral flow immunoassays (LFIs) for the diagnosis of human leptospirosis were developed and commercialized. However, the accuracy and reliability of these LFIs are not well understood. In this study, we aimed to evaluate the accuracy of leptospirosis LFIs as well as the factors affecting the test efficiency using systematic review and meta-analysis.

Methods and results

Original articles reporting the accuracy of human leptospirosis LFIs against microagglutination tests (MAT) or immunofluorescent assays (IFA) were searched from PubMed, Embase, and Scopus, and selected as per pre-set inclusion and exclusion criteria. A total of 49 data entries extracted from 24 eligible records published between 2003 and 2023 were included for meta-analysis. A meta-analysis was performed using STATA. The quality of the included studies was assessed according to the revised QUADAS-2. Only nine studies (32.1%) were considered to have a low risk of bias and no concern for applicability. Pooled sensitivity and specificity were calculated to be 68% (95% confidence interval, CI: 57–78) and 93% (95% CI: 90–95), respectively. However, the ranges of sensitivity (3.6 – 100%) and specificity (53.5 – 100%) of individual entries are dramatically broad, possibly due to the heterogeneity found in both study designs and LFIs themselves. Subgroup analysis demonstrated that IgM detection has better sensitivity than detection of IgG alone. Moreover, the test performance seems to be unaffected by samples from different phases of infection.

Conclusions

The pooled specificity of LFIs observed is somewhat acceptable, but the pooled sensitivity is low. These results, however, must be interpreted with caution because of substantial heterogeneity. Further evaluations of the LFIs with well-standardized design and reference test will be needed for a greater understanding of the test performance. Additionally, IgM detection type should be employed when leptospirosis LFIs are developed in the future.

Introduction

Leptospirosis is the most prevalent zoonotic infections worldwide, with the highest incidence occurring in tropical resource-poor nations [1]. It was estimated to be responsible for 1.03 million cases and 58,900 deaths annually [2]. Humans may acquire the infection from direct contact with infected animals’ fluids or from water or soil contaminated with those through cuts, abrasive skin, or mucosal contact [3]. Leptospirosis cases are usually asymptomatic, but some patients may develop symptoms. The most common symptom observed among leptospirosis patients is an acute febrile illness. The infection can be treated by using common antibiotics such as doxycycline, and azithromycin. However, if the appropriate treatment is not given in a timely manner, the disease may progress to more severe clinical manifestations, which could potentially lead to multiple organ failure and death [1]. Early and accurate diagnosis, therefore, plays a key role in the management of the infection. Unfortunately, clinical presentations of leptospirosis are unspecific, making the disease difficult to differentiate from other infections, e.g., malaria and dengue.

Leptospirosis is caused by spirochete bacteria belonging to the genus Leptospira. More than 300 serovars of Leptospira spp. have been reported to be associated with the infection [4]. The current gold standard diagnosis of leptospirosis is the microagglutination test (MAT) [5]. The test usually requires live leptospires (approximately 20 serovars), expert personnel, and specific equipment, limiting its availability to only central or reference laboratories [6]. To overcome these challenges, several diagnostics such as enzyme-linked immunosorbent assays (ELISA), polymerase chain reactions (PCR), and rapid lateral flow immunoassays (LFIs) have been developed [6,7].

In the last two decades, several LFIs have been created for the diagnosis of human leptospirosis and many of them have been commercialized [7]. The use of these LFIs in clinical settings, however, remains unpopular, and they are only applied as a screening test [7]. This is possibly due to the sensitivity and specificity of the tests which are inconsistent among studies, raising questions about the reliability and applicability of their use [7,8]. Thus, the primary objective of this study is to determine the overall sensitivity and specificity of the currently available human leptospirosis LFIs using systemic review and meta-analysis. Additionally, a secondary objective is set to investigate factors that may affect the assay accuracy or heterogeneity of test results. The investigated factors include brand, detection target, and phase of infection. We expected that our findings would be helpful for not only healthcare providers but also researchers or inventors developing LFIs for the diagnosis of human leptospirosis.

Methods

This systematic review was conducted in accordance with the Preferred Reporting of Systematic Review Meta-analyses (PRISMA) guidelines [9]. The protocol was registered with the International Prospective of Systematic Reviews (PROSPERO registration number CRD42022371788).

Search strategy

We searched the following databases for relevant PubMed, Embase, and Scopus. The search was first conducted on December 1^st^, 2022 and latest updated on January 28^th^, 2024. Restrictions were applied exclusively to English literature, but not for publication date or country of study. A search strategy was constructed with three key concepts to identify studies that report the sensitivity and specificity of different lateral flow assays for leptospirosis diagnosis (S1 Appendix). An additional screening from the reference lists of included studies and published systematic reviews was also performed to identify relevant studies.

Eligibility criteria

Original research articles either prospective or retrospective studies that met the following conditions were included in the meta-analysis: i) included patients intended to be diagnosed with leptospirosis; ii) examined the diagnostic accuracy of LFI; iii) used MAT or immunofluorescent assay (IFA) as a reference test; and iv) reported true positive (TP), false positive (FP), true negative (TN), and false negative (FN) values, allowing for calculation of sensitivity and specificity. We decided to include studies using the IFA as a reference test because the technique is based on the same principle as MAT and has been routinely used in some countries [10–12]. Studies were excluded if one or more of the following criteria were i) published in a language other than English; ii) evaluations of assays other than LFI; iii) evaluations performed in animals; and iv) insufficient data for the calculation of sensitivity and specificity.

Study selection and data extraction

All retrieved articles were imported to Rayyan, and duplicated studies were removed [13]. Two authors (TN and LL) screened and selected the articles independently based on title and abstract, followed by a full-text review. Any disagreements in selection were solved through discussion among all three researchers (TN, LL, and SN).

After piloting and standardization, the developed Microsoft Access form (Microsoft Corp., WA, USA) was used to extract data from the eligible studies. Data extraction was performed in duplicate by two authors (TN and LL) independently, and the collected data were cross-verified. Additionally, the relevant data from each selected article was summarized in a descriptive format (S2 Appendix).

Quality assessment

The quality assessment of the included articles was carried out using the criteria modified from Maze et. al. [7]. The criteria developed are in accordance with the revised Quality Assessment of Diagnostic Accuracy Studies criteria (QUADAS-2) [14]. For each selected study, the risk of bias was assessed in four patient selection, index test, reference test, and flow and timing, using criteria listed in S1 Table. The study applicability was graded in the domains of patient selection, index test, and reference test, using the criteria listed in the S2 Table. The studies were scored as “low risk or no concern, +” if the criteria were met; “high risk or concern, -” if one of the criteria was not met; or “unclear, ?” if the information reported in the studies was not sufficient for assessment.

Statistical analysis

Estimates and plots

The meta-analysis for this study was carried out using STATA version 16.1 (College Station, TX, USA) with all relevant packages (metandi, midas, and mylabels) installed. The extracted or calculated TP, FP, FN, and TN values from each study were included in the dataset. The primary measures are pooled sensitivity and specificity. We also plotted sensitivity and specificity data in forest plots calculated using a random effect bivariate model, and graphed study-specific estimates of sensitivity and specificity with 95% confidence intervals (CIs) in the summary receiver operating characteristic (SROC) curves, and the area under curve (AUC). To identify influential studies, the spike plot with Cook’s distance was used. Outlier detection was performed by interpreting the standardized level 2 residuals for sensitivity and specificity data of the studies included in the quantitative analysis.

Heterogeneity and subgroup analysis

We used a bivariate box plot to evaluate the spread and skewness of the data. We also examined heterogeneity across studies by visually inspecting the forest plots. Subgroup meta-analysis was performed for the selected brand, detection target, and phases of infection.

Publication bias

We assessed publication bias for diagnostic tests using Deeks’ funnel plot asymmetry a scatter plot of the inverse of the square root of the effective sample size [1/root (ESS)] versus the diagnostic log odds ratio (lnDOR). A symmetrical funnel shape with a p-value> 0.05 is considered to have no substantial publication bias.

Results

A total of 459 records were identified from an electronic search. After removing duplicates, 323 records remained for screening titles and abstracts. Of these, 42 reports were considered for full-text evaluation. After the application of inclusion and exclusion criteria, 18 reports were excluded, giving a total of 24 reports eligible for the review and meta-analysis (Fig 1) [11,15–37]. The records that were excluded and reasons for exclusion are listed in S3 Table [38–55].

Characteristics of eligible studies

The articles or reports included in this review are listed in Table 1. These reports were published from 2003 through 2023. There are 28 studies extracted from 24 reports. This is because the reports published by Alia et. al., Lee et. al., and Chang et. al. presented the results from two, three, and two studies, respectively [18,23,27]. Studies were conducted among patients from Argentina, Brazil, Bulgaria, Cambodia, Congo, India, Korea, Laos, Malaysia, Micronesia, Nepal, Netherlands, Philippines, Slovenia, Sri Lanka, Sudan, and Thailand. Among all 28 studies, 12 studies (42.8%) are retrospective studies in which the evaluations were performed using archived samples. The number of patients or samples varied from 50 to approximately 2,800. All the studies used MAT (solely or along with other tests) as a reference test, except one study published by Silpasakorn et. al., in which only IFA was used to define leptospirosis cases [11]. In addition, variety in case definition was observed among the studies (S4 Table). In all 24 reports eligible for this meta-analysis, 16 different commercial and 4 different in-house LFIs were evaluated. However, it should be noted that, in this study, the articles published in non-English languages were excluded. Thus, the total number of 20 LFIs included in this review may not correspond to the number of all developed LFIs.

Study quality

The results of the quality assessment were summarized in Table 2. The assessment was carried out using the criteria presented in the S1 and S2 Tables. The results showed that only nine studies (32.1%) were considered “low risk” of bias and “no concern” for applicability. Thirteen studies (46.4%) were considered “low risk” of bias for all four domains. Eleven studies (39.3%) were classified as “high risk” of bias for patient selection because healthy participants were used as controls; ten of them are retrospective case-control selection studies. Three studies (10.7%) were rated “high risk” of bias for reference tests since MAT was performed only on acute-phase sera. Seven studies (25.0%) were graded “unclear” for flow and timing domain because it is unclear whether samples in each study were subjected to the same reference tests. In terms of applicability, all 28 studies were classified as “no concern” on patient selection and index test domains. However, four studies (14.3%) were rated “unclear” and four studies (14.3%) were rated “concern” in the reference test domain. This is because the serovar panel used in MAT test in those studies was not listed or did not cover serovars circulating in the areas, respectively.

Descriptive analysis

All LFIs that were evaluated in this review are listed in the S5 Table and the evaluation results are summarized in Table 3. Among the 20 LFIs evaluated, 18 were designed to detect Leptospira-specific antibodies in blood or serum samples, whereas the other two assays are antigen detection targeting leptospiral lipopolysaccharide (LPS) in patient urines. These antigen detection assays have a sensitivity of 80–89% and a specificity of 74–87% [30]. Most antibody detection LFIs (11/18) aim to detect specific IgM. All these IgM detection assays were designed to have one reading window with a single control line and a single test line, except the ImmuneMed Leptospira IgM Duo Rapid Test that was developed to contain two reading windows [20]. This design allows samples to be tested at two different cutoff titers (1:50 and 200). The second most common design (5/18) for the antibody detection test strips is IgM/IgG detection. Of these, only the ImmuneMed Leptospira Rapid Test, was designed to contain two separate reading windows for IgM and IgG, while the remaining were constructed with three lines (IgM, IgG, and control lines) on the same strip [23]. The variety of designs of leptospirosis LFIs also includes ImmuneMed AFI rapid Test that was developed in a multiplex format to detect leptospirosis simultaneously with scrub typhus and hemorrhagic fever [11]. Additionally, the Dual Path Platform from Chembio Diagnostic Systems, USA, was designed in such a way that the sample and chase buffer were flown perpendicularly [21,31]. The accuracy of these LFIs were summarized in Tables 3 and S6.

In addition to the variety of LFI configurations described above, we also looked at the leptospiral antigens that were selected to be incorporated into the LFIs. Among the 18 antibody detection LFIs evaluated in this review, only 11 LFIs have disclosed the antigen they used. L. interrogans heat extract was found to be the most common antigens employed in the LFIs, followed by polysaccharide from nonpathogenic L. biflexa serovar Patoc. Of these, recombinant protein has been used in only one LFI [21,31].

Meta-analysis

Of these 28 studies extracted from 24 reports, several studies evaluated LFIs using samples collected from different phases of infection i.e., acute phase sera (mainly collected within 10 days after onset) and convalescent phase sera (mainly collected within two weeks after acute phase). In addition, the study conducted by Nabity et. al. evaluated the accuracy of the Dual Path Platform from Chembio Diagnostic Systems using samples collected by three different means i.e., finger stick blood (FSB), vein whole blood (VWB), and serum [21]. Thus, multiple sensitivity and specificity values were reported from these studies. As a result, a total of 49 data entries were employed for meta-analysis (Table 3).

Pooled sensitivity and specificity

We found that the sensitivity of individual entries varied widely, from 3.6% to 100.0%. The range of specificity for each entry was narrower, ranging from 53.5% to 100.0%. Firstly, the estimation of the pooled sensitivity and specificity was calculated by a random effect bivariate model with all 49 data entries included. The entry-specific results were displayed in the forest plot (Fig 2). The estimated pooled sensitivity and specificity were 68% (95% confidence interval, CI: 57–78) and 93% (95% CI: 90–95), respectively. The I^2^ values for sensitivity and specificity were 95.76% and 96.35%, respectively, indicating that the data entries included might be too heterogeneous to achieve accurate meta-analysis results. The AUC from the SROC curve was 0.92 (Fig 3). The Deeks’ funnel plot asymmetry showed no potential publication bias (Fig 4). The residual-based goodness-of-fit and bivariate normality plots suggested that the random effect model used was applicable. The influence and outlier detection analyses, however, identified 3 common outliers (S1 Fig). Next, we performed sensitivity and specificity analysis by removing the outlier that was graded high risk during the study quality assessment (entry ID 11) and found that the pooled sensitivity and specificity did not change significantly (S2 Fig). Thirdly, we calculated the combined sensitivity and specificity values using only one data entry per study. In this case, sensitivity and specificity obtained from acute phase samples were chosen as representative for the studies reporting multiple data entries. The combined sensitivity and specificity were 66% (95% CI: 51–78) and 92% (95% CI: 88–94), respectively, which were comparable to those obtained from all data entries (S3 Fig).

Subgroup analysis by LFI brands

The sensitivity and specificity of each brand of LFI were calculated using subgroup analysis and summarized in the S6 Table. This analysis, however, was limited to LFIs that have four or more data entries available. The results showed that the Dual Path Platform had the highest sensitivity (90%; 95% CI: 82–94), followed by Leptocheck WB (75%; 95% CI: 66–82), and LeptoTek Lateral Flow (65%; 95% CI: 52–76). The highest specificity was of ImmuneMed AFI rapid (99%; 95% CI: 97–100), followed by Dual Path Platform (89%; 95% CI: 81–94), and LeptoTek Lateral Flow (89%; 95% CI: 76–95), respectively.

Subgroup analysis by detection targets

We also performed subgroup analysis to investigate the influence of the detection targets (S7 Table). The LFIs that provided the greatest sensitivity are the IgM detection type (69%; 95% CI: 58–78), followed by the IgM/IgG detection type (62%; 95% CI: 26–88). IgG detection LFIs demonstrated the lowest sensitivity of 23% (95% CI: 3–73). On the contrary, the greatest specificity is of IgG detection LFIs (100%; 95% CI: 94–100), followed by IgM/IgG (97%; 95% CI: 94–98), and IgM detection (90%; 95% CI: 86–94), respectively.

Subgroup analysis by infection phases

The pooled sensitivity and specificity as grouped by infection phases were also determined (S8 Table). When samples collected from acute phase, convalescent phase, and mixed samples were used to evaluate the LFIs, the pooled sensitivities of the LFIs were 67% (95% CI: 50–80), 64% (95% CI: 39–83), and 69% (95% CI: 56–80), and the pooled specificities were 93% (95% CI: 88–96), 94% (95% CI: 89–97), and 94% (95% CI: 85–97), respectively.

Discussion

In this study, original articles evaluating the human leptospirosis diagnostic LFIs were systematically recruited for review and meta-analysis. The recruited articles were restricted to those that used MAT as a reference test because MAT is considered a gold standard diagnosis for leptospirosis and has been used globally [3,6]. We also included studies with IFA as a reference test because the technique is based on the same antigen-antibody reaction as MAT and has been used routinely in some endemic areas [10–12].

Meta-analysis was initially performed with all 49 data entries (Table 3) extracted from 28 eligible studies (Table 1). We found that the estimated pooled sensitivity and specificity were 68% and 93%, respectively (Fig 2). However, we thought that samples from the acute phase of infection were more relevant to real-world settings, as patients tend to visit hospitals a few days after onset. The pooled sensitivity and specificity were recalculated regarding this matter, and the results are not significantly different compared to the initial analysis. We noted that the pooled specificity of LFIs is somewhat acceptable, but the pooled sensitivity is low [56]. Nevertheless, it is important to emphasize that the estimated pooled sensitivity and specificity should be interpreted with caution due to the substantial heterogeneity observed among the included studies, despite the adoption of a random effect model. Furthermore, the forest plot’s I^2^ values indicate that the included studies were considerably heterogeneous to produce dependable meta-analysis results. Several potential factors contributing to heterogeneity were spotted during data extraction.

First, we identified that the study designs could be a potential cause of the data heterogeneity. Two types of study designs were observed among the eligible a case-control type and a cohort type cross-sectional study [57]. We found that 17 studies (61%) were cohort type cross-sectional studies. These were considered low risk for patient selection bias. The others 11 studies (39%), instead, were case-control type and were graded as high risk of bias in patient selections.

Second, reference tests and case definitions varied dramatically across the studies (S4 Table). For examples, some studies used MAT or IFA solely as a reference (54%) while others (46%) used MAT together with other methods. Also, the Leptospira serovar panel used in MAT might not cover all the predominant serovars in some studies. While several studies defined leptospirosis cases based on a four-fold increase in MAT antibody titers together with single MAT antibody titers, some evaluations used either single titers or paired titers only to define the cases. We noted that the investigators might define the single MAT titer cutoffs differently based on the prevalence of leptospirosis in the areas, and these cutoffs could be distinct from the WHO recommended cutoff of 400 [7,58]. However, various single MAT titer cutoffs within the same settings were detected (S4 Table). We determined that 10 studies (36%) were at risk (high risk and unclear) of bias and applicability for the reference test domain.

Third, we found that the samples themselves could also cause the heterogeneity of the evaluation results. While some investigators used samples collected from the acute phase of infection to evaluate the LFIs, others might use convalescent or mixed samples. Principally, acute phase samples contain a lower level of Leptospira-specific antibodies, and they are mainly IgM. Convalescent phase samples, on the contrary, have a higher titer of Leptospira-specific IgG [6]. Thus, even the same assays could have different sensitivity and specificity if different phases of samples were used for the evaluation.

Fourth, we also found that detection targets varied among the investigated LFIs. These variations include LPS detection LFIs, IgM detection LFIs, IgG detection LFIs, and combined IgM/IgG detection LFIs. Since the presence of IgM and IgG in the samples is determined by the phase of infection, evaluations of IgG detection LFIs with acute phase samples, for example, could give an expected lower sensitivity as compared to IgM detection LFIs [6]. This demonstrates that the detection target of LFI, among other factors, impacts the test performance.

Fifth, in addition to detection target, differences between each brand of the investigated LFIs also i) test designs (single-plex, multiplex, or two reading-windows formats); ii) flow patterns (regular lateral flow or perpendicular flow); iii) types of antigens incorporated (heat extract of the bacteria, purified LPS, recombinant proteins, etc.); and iv) miscellaneous factors such as types of membranes, amounts of antigen or antibody on the test and control lines, particle size of conjugated gold, etc. As a result, different brands or models of LFIs could perform differently due to the variations in these elements.

A question that is often asked in clinical settings (especially when there are several tests available, but test performances are heterogeneous among studies) is which brand of LFIs has the highest accuracy. To answer this question, we investigated further using multivariate meta-regression and subgroup analysis. However, this analysis is limited to the Dual Path Platform, Leptocheck WB, LeptoTek Lateral Flow, and ImmuneMed AFI Rapid only due to the availability of the data. The analysis results illustrated that the sensitivity and specificity are statistically comparable between each brand of LFIs (S6 Table).

Additionally, we conducted another subgroup analysis to understand how the LFIs performed with samples from different phases of infection. The results demonstrated that the sensitivity and specificity are practically unchanged, suggesting that the LFIs could be used with samples collected from either phase (S8 Table). Initially, this finding surprised us because acute phase samples are considerably not suitable for the IgG detection LFIs in principle. This is because the specific IgG has not been raised yet during the acute phase of the infection. However, when we examined the data closely, we found that the majority of the investigated LFIs are IgM and IgM/IgG detection types. Therefore, in this analysis, the results from IgM-targeted LFIs probably masked those from IgG detection types and gave the overall results as reported. We also found it interesting that the sensitivity of IgM detection LFIs remained unaffected when convalescent samples were used. Theoretically, the level of specific IgM declines during the convalescent phase of infections, causing IgM-targeted diagnostics to become less sensitive with convalescent samples. However, it has been reported that the specific IgM may stay in leptospirosis patients for months, which means that IgM-targeted LFIs will react with the convalescent samples [59].

In this study, subgroup analysis on detection targets was also carried out. As expected, IgM detection LFIs demonstrated a higher sensitivity over IgG detection types whereas specificity seems to be unaffected by this covariate (S7 Table). This finding would suggest that the IgM detection type should be chosen for further developments of the leptospirosis diagnostic LFIs.

In summary, during the past two decades, at least 20 LFIs for the diagnosis of human leptospirosis have been developed and marketed. Several studies were conducted to evaluate those LFIs; however, the results were markedly varied. In this study, we attempted to gain a better understanding of the accuracy of the available LFIs using a systematic review and meta-analysis. We found that the available data are heterogeneous; thus, the estimated accuracy of the LFIs derived from this study may not be at most reliable, especially the pooled sensitivity of the assays. This finding is in agreement with the systematic review and meta-analysis studies published previously [7,8]. The sources of heterogeneity in the data potentially came from both LFIs themselves and evaluation procedures. Variations in evaluation procedures seem to contribute to both assay sensitivity and accuracy. On the basis of our findings, it is difficult to ascertain whether the LFIs are clinically useful. Sequentially, we recommend that further evaluations of LFIs are needed, as also suggested by the previous reviews [7,8]. Further evaluations should be conducted as per standard guidelines, such as the Standard for Reporting Diagnostic Accuracy Studies (STARD) [60]. Additionally, they should use reference tests with sensitivity and specificity close to 100% or statistical tools that account for imperfect reference test accuracy, e.g., Bayesian latent class model [10]. Diversity in LFI designs, on the other hand, is likely to primarily affect the sensitivity of the assays. Our finding suggests that IgM or combined IgM/IgG detection types of the LFIs would be more suitable for the diagnosis of leptospirosis as compared to IgG detection types. However, in this current study, the comparison between antigen detection and antibody detection LFIs could not be done because the data from the antigen detection type is not sufficient.

Supporting information