COVID-19 antibody tests from Roche and Siemens scored highly on a comparison of emergency authorized tests published by FDA. The comparison, which relies on data from the companies, shows variation in the ability of serology assets to reliably determine if a sample contains antibodies against the pandemic coronavirus.
Roche’s Elecsys Anti-SARS-CoV-2 S, a semi-quantitative test that recently received emergency use authorization, is one of the standout performers. Applied to more than 6,000 samples, 233 of which were positive, the tests correctly determined if anti-coronavirus antibodies were present in almost every case. Applied to a population in which 5% of samples have antibodies, the test would have positive and negative predictive values (PPV/NPV) of 99.7% and 99.8%, respectively.
The older, qualitative version of the Elecsys test fared slightly worse, while still being a top performer, as did a handful of tests from Siemens. A semi-quantitative test from Siemens has a PPV of 98% and NPV of 100% at an antibody prevalence of 5%.
Other tests scored higher than the products from Roche and Siemens, even achieving 100% in both PPV and NPV, but the evidence supporting them is weaker. Hangzhou Biotest Biotech, for example, achieved 100% for PPV and NPV but only tested 110 samples. All of Siemens tests were assessed on more than 1,000 samples. Roche applied each of its two tests to close to 6,000 samples.
It is unclear if the 100% specificity and sensitivity reported by companies including Hangzhou Biotest Biotech would hold up if the tests were used on more samples. The uncertainty is evident in details of the PPV analyses.
The PPV for the Hangzhou Biotest Biotech test could be as low as 51%, based on the 95% confidence interval. In contrast, the lower bounds for the Roche and Siemens tests are, at worst, 93% and 88%, respectively. The bottom of the confidence interval for Roche’s semi-quantitative test is 98%.
Given differences between the studies, it is impossible to use the data to conclusively rank the tests but the results nonetheless provide an impression of how some assays may compare. Abbott, like Roche and Siemens, assessed most of its antibody tests on at least 1,000 samples. The PPVs of the three most comprehensively assessed Abbott tests are approximately 92%, with the lower and upper bounds of the confidence intervals coming in at around 85% and 95%.
That suggests Abbott’s tests perform slightly worse than the best antibody assays but comparably to kits from some other leading diagnostic players such as Beckman Coulter. Thermo Fisher Scientific’s OmniPATH COVID-19 Total Antibody has a PPV far below those of its peers — 67% — but its study only used 110 samples so the confidence interval is wide and direct comparisons may be unreliable.
The range of NPVs and, in particular, PPVs associated with the emergency-authorized serology assays points to the real-world consequences of small changes in sensitivity and specificity. Almost all of the tests are at least 90% sensitive and specific, according to the available data, but small differences in those figures can translate into big differences in PPV.
Biohit Healthcare’s test has a sensitivity of 97% and specificity of 95%, based on a study featuring 110 samples. That translates into a PPV of 50%. The bottom end of the confidence range is 27%. Used in the real world, particularly in areas with low exposure to the coronavirus, tests from Biohit and other manufacturers could deliver more false positives than true positives, causing people to erroneously think they have previously been infected and may have some immunity against the pathogen.
Abbott updated its fact sheet for healthcare providers for its kits at the start of the month to advise the use of a second, different antibody test to counter the risk of false positives. The change followed a request to FDA. Other fact sheets, including those provided by Roche and Siemens, already carry the statement.
The comparison is the latest in a series of wins for the diagnostic team at Siemens. In July, the U.K. reported Siemens’ SARS-CoV-2 Total was the only antibody test to meet its sensitivity and specificity criteria. Tests from Abbott, DiaSorin and Roche fell short of the 98% specificity target in assessments carried out by England’s public health agency. Earlier this month, Siemens’ real-time PCR molecular assay got high marks on FDA’s list of the most sensitive tests for the coronavirus.
Siemens’ antibody tests shone in a market that is so congested FDA has declined to review a filing for emergency authorization of Chembio Diagnostics’ product. Chembio filed for EUA in September. At the start of December, FDA said the volume of EUA requests has led it to prioritize applications based on factors such as public health needs and the availability of alternatives. FDA declined to review the Chembio filing in the belief the kit will have limited impact on access to testing.
Chembio plans to “work with the FDA to seek to establish priority” for its filing. The company has been off the market since FDA revoked its EUA in June in light of concerns about the accuracy of the test.