Introduction

The COVID-19 pandemic was announced by the WHO on 11 March 2020, only a few months after its etiological factor, the novel coronavirus SARS-CoV-2, had been identified. There is currently no specific treatment for this potentially fatal disease, but efforts around the world are being made towards combining antiviral and supportive therapy [1]. The accurate and immediate diagnosis is crucial for hampering the spreading of the disease and enabling its early treatment. The gold standard for laboratory diagnosis of COVID-19 is detection of viral RNA in the respiratory tract by means of a nucleic acid amplification test (NAAT), such as real-time RT-PCR [2]. However, the results of this test may be affected by improper sampling and mishandling of the specimen, leading to false negative results. Also, the sensitivity of the molecular methods is the highest in the first 7 days from the symptoms onset and gradually decreases afterwards [3, 4]. Simultaneously, the sensitivity of anti-SARS-CoV-2 antibody detection between 8 and 13 days from the symptoms onset is reported to be 90% [3]. Hence, it has been proposed to employ serological testing as an auxiliary means of COVID-19 diagnosis [5]. Apart from diagnosis, serological testing is used in the epidemiological investigations – determining seroprevalence, previous exposure and contact tracing. In the future, antibody detection and quantification may be useful during convalescent plasma treatment, vaccine evaluation and possibly identifying individuals who are immune to the disease [6, 7].

Currently no harmonized global guidelines or recommendations on the use of serological tests in COVID-19 are available. A thorough validation of the commercial methods on clinical samples is still lacking. Simultaneously, there are multiple assays available on the market and the demand from both clinicians and patients is recognized [8]. Generally, automatized immunoassays seem to be better validated, but in many locations they may not be available and hence the rapid immunochromatography assays (lateral flow immunoassays – LFIAs), suited to point-of-care testing, may also be useful. As shown by several instances, the performance of LFIA tends to be doubtful [9, 10]. However, this may differ between various producers and should be further elucidated.

The aim of this study is therefore to compare the results obtained with different LFIAs with the results of the chemiluminescent immunoassay (CLIA) method (MAGLUMI) and automated enzyme-linked immunosorbent assay (ELISA) (Euroimmun). This may shed some light on the clinical utility of the lateral flow assays and facilitate the selection of serological assays.

Material and methods

Patients and serum samples

This study included 52 samples from patients and healthy volunteers. All 52 samples were tested using CLIA (MAGLUMI, Snibe Diagnostic, Shenzhen, China), automated ELISA (Euroimmun, Lübeck, Germany) and 5 different LFIAs. The control samples (group A) were obtained during the SARS-CoV-2 pandemic from anonymous healthy volunteers (26–69 years; 13 females, 7 males) with no history of SARS-CoV-2 infection (no symptoms or contact with a confirmed COVID-19 case, based on a questionnaire, n = 20). The case cohort (group B) consisted of 32 consecutive patients referred to the Diagnostyka medical laboratory for anti-SARS-CoV-2 antibody investigation with the MAGLUMI CLIA assay (group B).

For the purpose of this study, CLIA was chosen as a comparative method. According to the CLIA results, the patients from the case group were divided as follows:

  • subgroup B1: 19 patients who were both IgG and IgM negative,

  • subgroup B2: 7 patients who were IgG positive but IgM negative,

  • subgroup B3: 6 patients who were both IgG and IgM positive.

Serum samples were used for testing. The samples were aliquoted and, prior to testing, stored refrigerated or frozen (–20°C) – if testing was to be performed after 3 days from the blood draw.

The informed consent forms for participation in the study were obtained from the control group subjects. The study was approved by the local Ethical Committee.

Serological assays

All the assays described below were performed strictly according to the manufacturers’ instructions. The producers’ characteristics of the tested immunoassays are provided in the Tables I and II. The results of the study were analyzed and presented as percent positive agreement, percent negative agreement and percent overall agreement between the methods.

Table I

Characteristics of CLIA and ELISA assays provided by the producers

Kit nameProducerSensitivity (%)Specificity (%)
Anti-SARS-CoV-2 ELISA (IgG)EuroimmunIgG:
33 (< 10d since symptom onset)
80 (> 10d since symptom onset )
IgG + IgA
66.7 (< 10d since symptom onset)
100 (> 10d since symptom onset)
98.5
Anti-SARS-CoV-2 ELISA (IgA)EuroimmunIgA:
50 (< 10d since symptom onset)
100 (> 10d since symptom onset)
IgG + IgA
66.7 (< 10d since symptom onset)
100 (> 10d since symptom onset )
92.5
MAGLUMI 2019-nCoV IgG (CLIA)Snibe DiagnosticsIgG 91.21
IgG + IgM 95.6
IgG 97.33
IgG + IgM 96
MAGLUMI 2019-nCoV IgM (CLIA)Snibe DiagnosticsIgM 78.65
IgG + IgM 89.89
IgM 97.5
IgG + IgM 96.5
Table II

Characteristics of LFIA assays provided by the producers

Kit nameProducerSensitivity (%)Specificity (%)Accuracy (%)
2019-nCov/COVID-19 IgG/IgM Rapid Test DeviceHangzhou Realy Tech Co. Ltd.IgG 96
IgM 92
IgG 100
IgM 100
IgG 98
IgM 96
Covid-19 IgG/IgM Rapid Test KitSafecare Biotech (Hangzhou) Co. Ltd.NANA90.19
2019-nCoV IgG/IgM Rapid Test CassetteHangzhou AllTest Biotech Co. Ltd.IgG 100
IgM 85
IgG 98
IgM 96
IgG 98.6
IgM 92.9
Core Test – Test COVID-19 IgM/IgG AbCore Technology Co. Ltd.IgG 94
IgM 94
IgG + IgM 97.6
IgG 100
IgM 100
IgG + IgM 100
IgG 97.1
IgM 97.1
IgG + IgM 98.8
SARS-CoV-2 IgG/IgM Ab Diagnostics Test Kit (Colloidal Gold)Shenzhen Watmind Medical Co. Ltd.IgG 96.52
IgM 94
IgG 98.52
IgM 97.73
NA

CLIA assay

CLIAs were performed on a MAGLUMI 800 analyzer (Snibe Diagnostic, Shenzhen, China). Kits’ components include positive and negative control samples.

The MAGLUMI 2019-nCoV IgM assay is a capture CLIA. Briefly, the immune complex formed of IgM antibodies in the sample and anti-human IgM antibodies-coated magnetic microbeads precipitate in a magnetic field. Then SARS-CoV-2 recombinant antigen labeled with ABEI (N-(4-aminobutyl)-N-ethylisoluminol) is added to form complexes and the chemiluminescent reaction is started.

The MAGLUMI 2019-nCoV IgG assay is an indirect CLIA. Briefly, the immune complex formed of IgG antibodies in the sample and recombinant SARS-CoV-2 antigen-coated magnetic microbeads precipitate in a magnetic field. Then anti-human IgG antibody labeled with ABEI is added to form complexes and the chemiluminescent reaction is started.

In both assays, the light signal is measured by a photomultiplier as relative light units (RLUs), which is proportional to the concentration of anti-SARS-CoV-2 IgM or IgG antibodies present in the sample. The thresholds of positivity are 1.0 AU/ml for IgM and 1.1 AU/ml for IgG. According to the manufacturer’s declarations, the antibodies used in these assays are directed against both CoV-S (spike) and CoV-N (nucleocapsid) [11].

ELISA assay

Enzyme-linked immunosorbent assays (ELISAs) were performed on the fully automated ELISA system EuroLabWorkstation 45 (Euroimmun, Lübeck, Germany). The ELISA kits include positive and negative control samples.

Anti-SARS-CoV-2 ELISA assays (Euroimmun, Lübeck, Germany) provide a semiquantitative determination of IgA and IgG antibodies against the SARS-CoV-2. The microplate wells are coated with SARS-CoV-2 recombinant S1 protein. The immune complexes are formed between the recombinant antigen and IgA or IgG anti-SARS-CoV-2 antibodies present in the sample. To detect the bound antibodies, a second incubation is carried out using an enzyme (peroxidase)-labelled anti-human IgA or IgG (enzyme conjugate) catalyzing a color reaction. Photometric measurement of the color intensity is performed at a wavelength of 450 nm.

The results are evaluated semi-quantitatively by calculation of the ratio of the extinction of samples over the extinction of the calibrator. The ratio interpretation is as follows: < 0.8 = negative, ≥ 0.8 to < 1.1 = borderline, ≥ 1.1 = positive.

LFIA assays

Lateral flow chromatographic immunoassays, also known as rapid or cassette tests, provide qualitative detection of IgG and IgM antibodies against SARS-CoV-2. This study included 5 different kits: 2019-nCov/COVID-19 IgG/IgM Rapid Test Device (Realy Tech Co. Ltd., Hangzhou, China); Covid-19 IgG/IgM Rapid Test Kit (Safecare Biotech Co. Ltd., Hangzhou, China); 2019-nCoV IgG/IgM Rapid Test Cassette (AllTest Biotech Co. Ltd., Hangzhou, China); Core Test - Test COVID-19 IgM/IgG Ab (Core Technology Co. Ltd., Beijing, China); SARS-CoV-2 IgG/IgM Ab Diagnostics Test Kit (Colloidal Gold) (Watmind Medical Co. Ltd., Shenzhen, China). The producers provide no specific information on the SARS-CoV-2 antigen used in their tests.

Briefly and typically, 1 droplet (ca. 20 μl) of the tested serum and 2 droplets of buffer (ca. 80 μl) are added to the well on the individual test cassette, followed by incubation at room temperature for 10–15 min. As a result, colored bands appear, one in the control area (indicating validity of the test), and – if the antibodies are present in the sample – one or two lines in the test area designated for IgM and IgG separately. The test results are evaluated visually.

Results

Out of the control serum samples none tested positive for anti-SARS-CoV-2 IgM or IgG. Two patients tested positive for anti-SARS-CoV-2 IgA (Euroimmun). Those patients were asked for an additional nasopharyngeal swab which was tested with real-time RT-PCR and came out negative. Further, the repeated testing after 2 weeks showed no seroconversion – the IgA remained positive with a similar antibody ratio and IgGs were still absent. This could imply that IgA results were falsely positive.

The rest of the patients from group A tested negative in all immunoassays performed (Table III).

Table III

The results of anti-SARS-CoV-2 testing obtained with seven different immunoassays

Subgroup:Number of patientsAnti-SARS-CoV-2 antibodies IgG
MAGLUMI (CLIA)Euroimmun (ELISA)RealyTech (LFIA)SafeCare (LFIA)AllTech (LFIA)CoreTest (LFIA)Watmind (LFIA)
A200000000
B1150000000
B120100000
B110000100
B110101100
B211000000
B211101101
B221101111
B211111101
B221111111
B361111111
Subgroup:Number of patientsAnti-SARS-CoV-2 antibodies IgM (or IgA*)
MAGLUMI (CLIA)Euroimmun* (ELISA)RealyTech (LFIA)SafeCare (LFIA)AllTech (LFIA)CoreTest (LFIA)Watmind (LFIA)
A180000000
A20100000
B1130000000
B150100000
B110000100
B210000000
B220100000
B220110000
B210110010
B210110011
B321110000
B311111010
B311111011
B321111111

[i] The table illustrates the results of serological testing in the control and case cohorts, separately for late (IgG) and early (IgM or IgA) immune response. The results are summed up within a given group according to the same constellation of the results. Group A: control cohort, subgroup B1: patients tested negative for both IgG and IgM in CLIA; subgroup B2: patients tested negative for IgM but positive for IgG in CLIA; subgroup B3: patients positive for both IgG and IgM in CLIA. 0 – negative result, 1 – positive result.

Case subgroup B1 consisted of 19 patients who tested negative for both IgM and IgG in CLIA. For these patients a few discordant results, mostly in Euroimmun ELISA, were observed (Table III). High concordance between the comparative method and LFIAs was observed. Three of the tested LFIAs had 100% concordance with the comparative CLIA method in IgG testing and four in IgM testing. In 1 case the sample tested positive only in AllTech LFIA both in IgG and IgM. In this subgroup, 1 case should be considered inconclusive in IgG testing, having positive results in ELISA and two of the tested LFIAs and negative results in CLIA and the other three LFIAs.

Surprisingly, the tested ELISA (Euroimmun) had higher discordance with CLIA than the LFIAs. In discordance with CLIA, ELISA produced positive results in IgG in 3 cases and in 5 cases in IgA testing.

Case subgroup B2 consisted of 7 patients who in the comparative CLIA method tested positive in IgG but negative in IgM. One patient was positive only in IgG MAGLUMI, suggesting a false positive MAGLUMI result. The remaining 6 cases of IgG positive results were confirmed by most of the other tests (Table III).

In this subgroup, the highest concordance with the CLIA method was observed for LFIAs produced by SafeCare and AllTech (6 out of 7 concordant results for IgG and 7 out of 7 for IgM), whereas the RealyTech LFIA produced contrary results in 4 out of 7 cases in both antibody classes. Euroimmun ELISA was concordant with CLIA in the IgG test (in 6 cases) but completely discordant in IgM (in 6 out of 7cases).

Subgroup B3 consisted of 6 patients for whom positive results for both IgM and IgG were obtained. In this subgroup 100% concordance for IgG results was observed between the methods. Full agreement of IgM results was also observed between CLIA, ELISA and one of the tested LFIAs (RealyTech). The other LFIAs tested produced negative results in 2, 3, or even 4 cases (RealyTech), as shown in Table III.

The summarized agreements between the results of the tested methods in the whole group of 52 subjects are shown in Table IV. The highest agreement with the MAGLUMI method was found for Watmind cassettes (98%) in IgG and for SafeCare cassettes in IgM. The lowest agreement was observed for RealyTech and Euroimmun for IgG (92.3%), and for Euroimmun for IgM (75%). The low agreement for Euroimmun ELISA is due to the low concordance with MAGLUMI negative results – suspected false positive Euroimmun IgA results.

Table IV

The concordance of the results obtained with the comparative CLIA method and ELISA or 5 different LFIAs

ParameterConcordance of positive resultsConcordance of negative resultsSummarized concordance
PPA (%)# of samplesPNA (%)# of samplesPOA (%)# of samples
MAGLUMI IgGRealyTech IgG699/1310039/3992.348/52
SafeCare IgG9212/139738/3996.150/52
AllTest IgG9212/139537/3994.249/52
CoreTest IgG7710/1310039/3994.249/52
Watmind IgG9212/1310039/399851/52
Euroimmun IgG9212/139236/3992.348/52
MAGLUMI IgMRealyTech IgM1006/69142/4692.348/52
SafeCare IgM674/610046/4696.150/52
AllTest IgM332/69845/4690.347/52
CoreTest IgM674/69644/4692.348/52
Watmind IgM503/69845/469248/52
Euroimmun IgA1006/67233/467539/52

[i] The table illustrates the summarized concordance for all tested samples. PPA – percent positive agreement, PNA – percent negative agreement, POA – percent overall agreement.

Discussion

There is a scarcity of published data comparing the serological assays for anti-SARS-CoV-2 antibodies. There is one paper analyzing the concordance between MAGLUMI CLIA IgG and IgM to Euroimmun ELISA IgG and IgA [11]. Lippi et al. elegantly compared the results obtained with those methods in patients with suspected COVID-19. Their report pointed to the concordance of MAGLUMI and Euroimmun, calculated to be 88% for IgG and 90% for IgM/IgA. Therefore, the authors attested to the substantial degree of concordance between those methods at their respective cut-offs. Our results confirmed the above with respect to IgG. However, the concordance of MAGLUMI IgM and Euroimmun IgA reported in the current study is much lower (75%).

The weaker correlation of the early phase results between CLIA and ELISA, in comparison to the IgG results, may stem from the difference in the exact analyte tested. It has been reported that the anti-SARS-CoV-2 IgA and IgM seroconversion is similar, with both antibody classes detectable at 3–6 days from the symptom onset [12, 13]. Therefore, IgA antibodies testing instead of IgM in SARS-CoV-2 has been postulated as a means of limiting false positive results caused by the high cross-reactivity of IgM antibodies [14]. However, our results indicate the contrary – there seem to be more false positive results in the IgA assay. The only positive results observed in this study in the control group (healthy volunteers) were in fact IgA in Euroimmun ELISA. These patients were further tested with real-time RT-PCR (nasopharyngeal swab) and the results were negative. Further, the repeated testing after 2 weeks showed no seroconversion – the IgA remained positive with a similar antibody ratio and IgGs were still absent. Also, in the case of subgroup B1, a few patients with IgM and IgG negative in CLIA tested positive solely in IgA ELISA. Hence, isolated positive IgAs in those patients were probably false positive results. It must however be noted that Lassaunière et al. reported high specificity of Euroimmun ELISA – 93% and 96% for Euroimmun IgA and IgG ELISAs, respectively [9]. Nonetheless, those authors also outlined some cross-reactivity of both IgA and IgG Euroimmun ELISAs, mainly in samples containing antibodies to more than one respiratory virus, but also associated with the presence of adenovirus, dengue virus and human coronavirus HKU1 antibodies [9]. Therefore, any isolated anti-SARS-CoV-2 IgA result should be verified by real-time RT-PCR testing, and that may put additional financial strain on the health care providers. A recent paper concerning the validation of the Euroimmun ELISA assays directly states that the IgG assay displays nearly optimal diagnostic accuracy and no obvious gains result from IgA serology [15].

Apart from the suspicion of cross-reactivity of the Euroimmun IgA assay, the high number of IgA positive results in group B2, with positive IgG and negative IgM in MAGLUMI, could also indicate longer persistence of IgA in comparison to IgM. The kinetics of IgM and IgA antibody production in SARS-CoV-2 are not fully elucidated yet and could also be influenced by an individual’s immune response. It would be of interest to check whether the disappearance of IgM or IgA antibodies accompanies the disappearance of the virus and indicates the contagious status of a patient. Our data also revealed a few positive IgG Euroimmun ELISA results that were not confirmed in other methods. The study design does not allow us to draw a definite conclusion whether Euroimmun ELISA is less specific and therefore produces false positive results or it is more sensitive and the results obtained with the other methods tested are false negative. To resolve that, real-time RT-PCR data from these patients and information on antibody seroconversion over time would be needed.

Our study proves that the results obtained with the tested LFIAs are comparable with the results of the CLIA test. However, the agreement is reduced when positive samples are analyzed separately, and the results obtained for IgM positive patients are more discordant than those for IgG positive patients. A more thorough assessment of LFIA IgM performance could be made if the number of IgM positive samples was higher. Also, our cohort lacked patients with isolated IgM positive results, and such cases would shed more light on the utility of the tested methods with respect to IgM assessment.

The performance of different LFIAs varied. Among the tested immunochromatography tests, Watmind seems to be the most accurate for IgG detection, showing 100% concordance for negative CLIA results and 92% concordance for CLIA positive results. The 1 case in which there was discordance was probably false positive in CLIA. However, Watmind LFIA does not prove accurate when it comes to IgM detection. In this case the concordance with CLIA was 98% for negative results but only 50% for positive results.

A similar issue was observed for SelfCare and AllTech LFIAs. Whereas IgG detection was comparable to that of CLIA, the concordance of IgM results was substantially worse, decreasing to 33% for IgM positive by AllTech.

The highest overall concordance between CLIA and LFIA in IgM results was found for the SelfCare test (96.1%), but the other kits tested also showed agreement of more than 90%. Again, the concordance of IgM positive results between CLIA and SelfCare was substantially lower (67%).

The worst performing LFIAs in our study, RealyTech for IgG and AllTest for IgM, still had overall concordance with CLIA of 92.3 and 90.3%, respectively. It must be noted, though, that these kits failed to discern IgG positive patients in 4 out of 13 patients (RealyTech) and IgM positive patients in 4 out of 6 cases (AllTest). False negative results obtained with the tested LFIA devices are of high concern, as they may lead to COVID-19 cases being missed and the possible spread of the disease.

Other reports on the performance of LFIA kits published thus far are equally cautious about the wide use of these tests [9, 10]. The United Kingdom’s National COVID Scientific Advisory Panel has evaluated the performance of in-house ELISA and 9 undisclosed LFIA devices using COVID-19 confirmed cases samples and pre-pandemic control samples. The authors reported high sensitivity of IgG ELISA after 10 days from the symptom onset. The performance of the tested LFIAs was not sufficient to guide their recommendation for individual patient applications. The sensitivity of the tested LFIAs observed in the above study, as compared to ELISA positive, ranged from 65% to 85%, and specificity from 93% to 100%. The authors reported no evidence of differences in sensitivity or specificity between the tested LFIAs [9]. A Danish study assessed three commercial ELISAs and 6 LFIAs. The samples tested were obtained from confirmed COVID-19 cases and pre-pandemic control samples. Although the authors provided some rankings for the accuracy of LFIA devices, they outline that the performance of the different devices varies notably. The observed concordance of the tested LFIAs with Euroimmun ELISA was between 73 and 90%. Thus, the authors warn that results obtained with different methods for a given patient may not be unanimous and suggest that ELISA should be used to confirm LFIA results [9].

Our study is not free from limitations. The small size of the cohort does not allow us to make any final recommendations as to better/worse performing assays. Also, the results reported here are only a comparison of the methods. Since no clinical data or RT-PCR results are available, it is not possible to verify clinical sensitivity and specificity of the tested methods. The suspected false positive results obtained with Euroimmun IgA ELISA also need further investigation on a larger group of patients, preferably with SARS-CoV-2 status verified by real-time RT-PCR.

In conclusion, the results obtained in this study provide evidence for high overall concordance between the comparative CLIA method and different LFIAs (93.8%) not inferior to the other automated method – ELISA. This could justify the use of LFIAs in some settings, where automated assays are not available, provided that some limitation with regards to false negative results is taken into consideration. Also, the agreement between CLIA and different LFIAs varied significantly between the kits, but the small cohort tested does not allow us to unequivocally recommend any of the producers.

The concordance of the results obtained with different methods seems to be dependent on the population tested. In healthy volunteers, where the results are expected to be negative, the concordance was the highest (98.6%). Also, in patients who tested positive in both classes (IgM and IgG) in CLIA, the agreement between the methods was higher (91.6%) than in the subgroup with only IgG positive (71.4%).

The concordance was also higher in IgG testing, in comparison to IgM testing, which might be an outcome of cross-reactivity of antibodies of this class.