 |
| © Rebecca Cook A man is tested for COVID-19 for free at the Sheffield Center in Detroit on Tuesday, April 28, 2020. |
By Josh Lederman, NBC News
WASHINGTON — Can you trust the results of a coronavirus antibody test? For the vast majority of tests on the market, the government can't say.
Now Congress is sounding the alarm after early data from researchers in California who "tested the tests" showed troublingly high rates of false results. One in 3 antibody tests they analyzed produced false positives more than 10 percent of the time.
"No one would be happy with those results," Dr. Alex Marson, a physician and microbiologist at the University of California, San Francisco who is conducting the research, told NBC News. "If we give people a false impression, for whatever reason, that they may be in fact protected when they're not, that would be not only irresponsible, it would be dangerous to individuals."
As of this week, 150 antibody tests, also known as serology tests, are on the U.S. market that have faced no government review, according to data from the Food and Drug Administration. Another 24 laboratories are conducting their own antibody testing without having obtained FDA authorization.
In contrast, the FDA has actually verified and authorized just eight tests.
Amid an ongoing investigation, the House Oversight subcommittee on economic policy sent letters Tuesday demanding test data and a list of all purchasers from four companies that have claimed that their tests are more accurate than the California researchers' data shows.
"The discrepancies in these performance characteristics are deeply troubling," the subcommittee's chairman, Rep. Raja Krishnamoorthi, D-Ill., said in the letters, which were obtained by NBC News.
The antibody tests, a pillar of reopening the economy, were rushed to the U.S. market without review by the FDA.
The emergency policy to speed up availability of antibody tests — which tell you whether you once had the virus — was intended to avoid the excruciating delays the U.S. faced in ramping up diagnostic tests — which tell you if you currently have COVID-19, the disease associated with the coronavirus.
Scientists still don't know whether people who test positive for antibodies have immunity to COVID-19 and, if so, how much. But identifying who's already had the virus is critical to assessing when social distancing can be eased and who can safely return to work, with the Trump administration considering issuing "immunity certificates" to those who've already had the virus.
That's why doctors and researchers are particularly concerned about false positives.
"The concern here is that these tests are giving people this false sense of security, that they're going to go out and they're not going to protect themselves or protect other people from them," said NBC News medical correspondent Dr. John Torres. "They could be spreading coronavirus or catching it."
Although the FDA hasn't announced any accuracy benchmark that tests must meet, Dr. Deborah Birx, the White House coronavirus response coordinator, has cited 90 percent "test specificity" as the bar to meet.
Test specificity refers to how many negative samples accurately test negative. So a specificity of 90 percent means the test produces a false positive the other 10 percent of the time. For other common medical tests, a specificity of 95 percent or 98 percent is often the industry standard, medical experts say.
Marson and a team of researchers at UC San Francisco and the University of California, Berkeley have spent months analyzing a dozen of the more than 150 tests on the market. They used the tests to test "known positive" blood samples — from coronavirus patients — as well as "known negative" samples from 2018 or earlier, long before the coronavirus started circulating.
Four of the 12 tests produced false positives for the pre-2019 samples at least 10 percent of the time. One test — from Anhui Deepblue Medical Technology Co. — triggered false positives 16 percent of the time. Only one test — from Sure Biotech — detected samples negative for antibodies 100 percent of the time.
The accuracy results were even worse for samples from patients known to have had COVID-19. With one exception — a high-performing test from Jiangsu Bioperfectus Technologies — none of the other tests exceeded 91 percent accuracy rates for detecting either IgG or IgM antibodies for the coronavirus, even for samples taken 20 or more days after patients noticed symptoms.
Researchers from the two University of California schools cautioned that the results are preliminary and haven't yet undergone peer review. But the early results are alarming enough that the House subcommittee is urging the FDA to revise its policy to require antibody test makers to apply for FDA approval and to allow only effective tests to be sold.
Under an FDA policy announced in March, all that labs and manufacturers must do before starting testing is assert that they've validated the tests themselves. They are encouraged but not required to submit the data for FDA review or emergency approval.
Those selling unapproved tests must inform people in the test results that the tests haven't been FDA-reviewed and "should not be used as the sole basis" to determine whether someone has had COVID-19.
But House investigators have raised concerns about whether the requirements are being followed, and they identified at least one company whose written materials suggested that people who tested positive for IgG antibodies, which the body starts making later in the disease process, are assumed to have "functional immunity" and can return to work or discontinue social distancing.
It's unclear how much policing the FDA is doing when test manufacturers overstate the applicability of their tests. House Oversight Committee staff members found that the FDA has no dedicated staff focused on enforcement regarding antibody tests. The same FDA officials who review applications for emergency authorization are given the double duty of monitoring the market and acting on violations.
FDA Commissioner Stephen Hahn has said "we have and will" take action "such as issuing Warning Letters" telling companies to stop making false claims. But the FDA didn't respond to inquiries about how many letters have been issued or whether the agency has pursued any civil or criminal actions against violators.
Concerns about the tests' accuracy come amid continued uncertainty about whether having coronavirus antibodies proves future immunity and about which of the antibodies produced by the body in response to the virus confers immunity. Experts have said it may take months for answers.
"I would clearly tell people not to rely on tests at this point to see if they are immune to future infections with COVID-19," Marson said.
Even if individual results can't fully be trusted, the antibody tests can still be useful for public health, because they can be used on a wider scale to assess what part of the population more or less has been exposed to the virus. In New York, early results from a state program that has tested thousands of people found exposure rates as high as almost 25 percent in New York City.
But in most parts of the U.S., the percentage of all people who have contracted the coronavirus is assumed to remain in the single digits. That means that if a test is producing false positives for 10 percent of people who take it, the test is actually producing more false positives than true positives.
Faulty antibody tests have already created major problems in some places. Last month, the city of Laredo, Texas, spent half a million dollars on 20,000 coronavirus antibody tests that then failed the city's efforts to validate them. The Department of Homeland Security ultimately seized the tests.
See more at NBC News
WASHINGTON — Can you trust the results of a coronavirus antibody test? For the vast majority of tests on the market, the government can't say.
Now Congress is sounding the alarm after early data from researchers in California who "tested the tests" showed troublingly high rates of false results. One in 3 antibody tests they analyzed produced false positives more than 10 percent of the time.
"No one would be happy with those results," Dr. Alex Marson, a physician and microbiologist at the University of California, San Francisco who is conducting the research, told NBC News. "If we give people a false impression, for whatever reason, that they may be in fact protected when they're not, that would be not only irresponsible, it would be dangerous to individuals."
As of this week, 150 antibody tests, also known as serology tests, are on the U.S. market that have faced no government review, according to data from the Food and Drug Administration. Another 24 laboratories are conducting their own antibody testing without having obtained FDA authorization.
In contrast, the FDA has actually verified and authorized just eight tests.
Amid an ongoing investigation, the House Oversight subcommittee on economic policy sent letters Tuesday demanding test data and a list of all purchasers from four companies that have claimed that their tests are more accurate than the California researchers' data shows.
"The discrepancies in these performance characteristics are deeply troubling," the subcommittee's chairman, Rep. Raja Krishnamoorthi, D-Ill., said in the letters, which were obtained by NBC News.
The antibody tests, a pillar of reopening the economy, were rushed to the U.S. market without review by the FDA.
The emergency policy to speed up availability of antibody tests — which tell you whether you once had the virus — was intended to avoid the excruciating delays the U.S. faced in ramping up diagnostic tests — which tell you if you currently have COVID-19, the disease associated with the coronavirus.
Scientists still don't know whether people who test positive for antibodies have immunity to COVID-19 and, if so, how much. But identifying who's already had the virus is critical to assessing when social distancing can be eased and who can safely return to work, with the Trump administration considering issuing "immunity certificates" to those who've already had the virus.
That's why doctors and researchers are particularly concerned about false positives.
"The concern here is that these tests are giving people this false sense of security, that they're going to go out and they're not going to protect themselves or protect other people from them," said NBC News medical correspondent Dr. John Torres. "They could be spreading coronavirus or catching it."
Although the FDA hasn't announced any accuracy benchmark that tests must meet, Dr. Deborah Birx, the White House coronavirus response coordinator, has cited 90 percent "test specificity" as the bar to meet.
Test specificity refers to how many negative samples accurately test negative. So a specificity of 90 percent means the test produces a false positive the other 10 percent of the time. For other common medical tests, a specificity of 95 percent or 98 percent is often the industry standard, medical experts say.
Marson and a team of researchers at UC San Francisco and the University of California, Berkeley have spent months analyzing a dozen of the more than 150 tests on the market. They used the tests to test "known positive" blood samples — from coronavirus patients — as well as "known negative" samples from 2018 or earlier, long before the coronavirus started circulating.
Four of the 12 tests produced false positives for the pre-2019 samples at least 10 percent of the time. One test — from Anhui Deepblue Medical Technology Co. — triggered false positives 16 percent of the time. Only one test — from Sure Biotech — detected samples negative for antibodies 100 percent of the time.
The accuracy results were even worse for samples from patients known to have had COVID-19. With one exception — a high-performing test from Jiangsu Bioperfectus Technologies — none of the other tests exceeded 91 percent accuracy rates for detecting either IgG or IgM antibodies for the coronavirus, even for samples taken 20 or more days after patients noticed symptoms.
Researchers from the two University of California schools cautioned that the results are preliminary and haven't yet undergone peer review. But the early results are alarming enough that the House subcommittee is urging the FDA to revise its policy to require antibody test makers to apply for FDA approval and to allow only effective tests to be sold.
Under an FDA policy announced in March, all that labs and manufacturers must do before starting testing is assert that they've validated the tests themselves. They are encouraged but not required to submit the data for FDA review or emergency approval.
Those selling unapproved tests must inform people in the test results that the tests haven't been FDA-reviewed and "should not be used as the sole basis" to determine whether someone has had COVID-19.
But House investigators have raised concerns about whether the requirements are being followed, and they identified at least one company whose written materials suggested that people who tested positive for IgG antibodies, which the body starts making later in the disease process, are assumed to have "functional immunity" and can return to work or discontinue social distancing.
It's unclear how much policing the FDA is doing when test manufacturers overstate the applicability of their tests. House Oversight Committee staff members found that the FDA has no dedicated staff focused on enforcement regarding antibody tests. The same FDA officials who review applications for emergency authorization are given the double duty of monitoring the market and acting on violations.
FDA Commissioner Stephen Hahn has said "we have and will" take action "such as issuing Warning Letters" telling companies to stop making false claims. But the FDA didn't respond to inquiries about how many letters have been issued or whether the agency has pursued any civil or criminal actions against violators.
Concerns about the tests' accuracy come amid continued uncertainty about whether having coronavirus antibodies proves future immunity and about which of the antibodies produced by the body in response to the virus confers immunity. Experts have said it may take months for answers.
"I would clearly tell people not to rely on tests at this point to see if they are immune to future infections with COVID-19," Marson said.
Even if individual results can't fully be trusted, the antibody tests can still be useful for public health, because they can be used on a wider scale to assess what part of the population more or less has been exposed to the virus. In New York, early results from a state program that has tested thousands of people found exposure rates as high as almost 25 percent in New York City.
But in most parts of the U.S., the percentage of all people who have contracted the coronavirus is assumed to remain in the single digits. That means that if a test is producing false positives for 10 percent of people who take it, the test is actually producing more false positives than true positives.
Faulty antibody tests have already created major problems in some places. Last month, the city of Laredo, Texas, spent half a million dollars on 20,000 coronavirus antibody tests that then failed the city's efforts to validate them. The Department of Homeland Security ultimately seized the tests.
See more at NBC News
