Testing Snafus – Testing, Testing, 1, 2, Whoa!

Key Takeaways:

• Increasing scrutiny of laboratory-developed tests (LDTs) for COVID-19 casts doubts on accuracy. 
• Critics fear many FDA-approved tests generate an unacceptably high level of false negatives due to imperfect sample collection methods.
• Federal regulators do not require field testing before LDTs enter the market.
• No one is conducting comparison testing that would inform clinicians about which tests work best.
• The push for consumer-friendly tests, like those conducted at home or using short nasal swabs, could compound the false negative problem. 

As the U.S. ramps up its COVID-19 testing capacity, there’s growing concern about the accuracy of the diagnostic tests entering the market. The Food and Drug Administration has already approved over 80 tests under its Emergency Use Authorization powers, with scant evidence for their efficacy in clinical practice.[1]

Testing companies and federal regulators claim the already approved in vitro diagnostic tests, that look for the presence of SARS-CoV-2 in mucous, saliva or blood samples, deliver more than 95% sensitivity. Sensitivity measures the share of active cases that a test accurately identifies. Test producers base those high-sensitivity claims on bench tests of samples already known to contain the virus. 

But hospitals and clinics are reporting that once deployed, some of the FDA-approved tests fail to identify patients with COVID-19 as much as 30% of the time. That’s an extremely high “false negative” rate. Scientists frown on  any test with less than 95% sensitivity. Poor quality tests could wind up devastating communities and workplaces that are pinning their hopes for a near-term reopening on their ability to quickly identify the presence of the disease.

Hospitals have the same problem. They want to resume elective surgeries and procedures cancelled during the first two months of the COVID-10 lockdown, since their cancellation dramatically reduced hospital revenue. New York Presbyterian, for instance, requires COVID-19 testing before any patient can come in for surgery, so will need more testing capability to reschedule treatment.

But Francine Castillo, director of practice operations at Columbia University Medical Center, wrote on the hospital’s website that “it’s still a bit difficult, to be honest, because we don’t have easy testing locations to send people to, not all tests out on the market are adequate.” When she questioned the hospital’s lab director, he told her, “There are no SARS-CoV-2 tests that we are comfortable with the results on the market at this time.” He said one test they tried missed many low-level positive cases, which could lead to asymptomatic patients with COVID-19 infecting the entire hospital.[2]

Some of the experts raising alarms said the main problem may not be the tests themselves, but the sample collection and transportation processes. “The tests aren’t evaluated using symptomatic people,” wrote Dr. Jennifer Nuzzo, associate professor in the Department of Epidemiology at Johns Hopkins Bloomberg School of Public Health, in an email. “Performance of a test in the real world can erode if the specimen is not collected properly, if the test is not performed correctly, and if someone is too early in clinical illness, e.g., they’re tested too early.”

The Trump administration dismissed those concerns in its latest strategic testing plan, released over Memorial Day weekend. The plan delegates responsibility for ramping up testing over the next month to the states. The plan also calls for limiting testing to suspected cases, which it estimates will reach about 9 million people in June. That’s an average around 300,000 tests per day.

“This number is already being achieved through the current testing regimen, and will be far-exceeded by mid-summer,” according to the Health & Human Services report to Congress.[3] The $484 billion coronavirus relief bill, signed into law in late April, included $25 billion for testing.  The law also required states and the federal administration to submit testing plans. State and federal plans also must include strategies for reducing the disparities in poor and minority communities.

The federal government has probably inflated its claim that it has reached its limited goal. Just ahead of Memorial Day weekend, reports surfaced that the Centers for Disease Control and Prevention’s test totals included serology tests, that test for antibodies in people who have recovered from the disease.[4] Health departments in Virginia, Texas and Vermont, among others, admitted that they submitted combined data.[5] Inflated test totals would reduce the reported prevalence and mortality rates of the disease, and thus the perceived need for higher testing levels.   

The federal government bases its claim that only 300,000 tests daily will be necessary on available tests producing a 95% sensitivity rate. The critics call that rate wildly optimistic. The report explicitly rejected assertions by most healthcare experts, who say building an effective tracking and tracing system will require 5 to 20 times more daily tests than 300,000.

Successful tracing requires rapid detection of virus carriers; communication with everyone the carrier saw over the previous few days; testing those individuals; and isolating everyone carrying the virus. For instance, the Harvard University Safra Center for Ethics’ “Roadmap to Pandemic Resilience,” released in late April 2020, estimated the U.S. needs to test 3 million people daily to contain the disease. They based their estimate on an average test sensitivity rate of 80%[6], the rate report writers said reflects actual clinical results.

“The diagnostic tests have good sensitivity but there can be problems with test administration, in particular, with administration of the nasal swabs. It can be hard to do those right.” —Danielle Allen, Director, Edmond J. Safra, Center for Ethics at Harvard University

Ignoring the problem of low sensitivity rates, the FDA gave emergency approval to at-home tests by Everlywell and LabCorp, both of which require deep nasal self-swabbing. Another approved at-home test by RUCDR Infinite Biologics, developed at Rutgers University, uses saliva samples, which experts say are less effective than nasal swabs.[7]

The Cleveland Clinic was among the first hospitals to raise concerns about the adequacy of commercially available tests. Its top lab official questioned Abbott Lab’s claims that its ID NOW test could deliver accurate results in 15 minutes.[8] The Journal of Clinical Virology published research in mid-May that showed the Abbott test had a sensitivity of just 72%.

“ID NOW performs well for strong and moderately positive samples but has reduced sensitivity for weakly positive samples,” the peer-reviewed study by scientists from the University of Pittsburgh School of Medicine and the New York State Public Health Department said. “This sensitivity, among other concerns, should be taken into consideration when using this test for patients with a low suspicion for COVID-19 disease.”[9]

The growing concern about diagnostic test accuracy mirrors the questions raised by the FDA’s decision to allow dozens of serology tests onto the market without even a shred of data showing how well they work. Serology tests measure the presence of antibodies that would indicate a person previously had COVID-19 and recovered.

After critics in Congress began investigating, the FDA reversed course and ordered test manufacturers to submit data from their bench tests within 10 days.[10] At last count, the FDA has removed 29 serology tests from the market, including 18 that originated in China.[11]

The Regulatory Maze

The FDA’s inability to adequately police market entry in laboratory testing is a longstanding problem. Lobbyists, who serve the powerful special interests that make up the industry, convinced Congress to craft a weak regulatory regime that rarely requires evaluating field test results before approval.

The industry’s structure is complex. LabCorp and Quest Diagnostics, both publicly traded companies, dominate the in vitro diagnostic-test market with about $19 billion of the industry’s estimated $25 billion in 2019 sales. They send their kits mostly to hospitals, clinics and physician offices. The clinicians return the test samples to the firms’ centralized labs for analysis. Classified as medical devices and well regulated, the tests used by big firms undergo extensive bench and clinical testing before gaining approval.

Large hospitals, including all the major academic medical centers, maintain their own labs to meet their need for rapid turnaround times on routine tests. They also conduct highly specialized tests. Over the past two decades, a growing number of independent lab companies have sprung up. These companies usually focus on a handful of tests for rare maladies, specific genetic malfunctions and hard-to-treat diseases. The physician-scientists who develop these tests, usually while working on government grants to AMCs, often use venture capital funding to start their companies.  

Tests from hospital labs and these smaller labs – sometimes called laboratory-developed tests or “home-brew” tests – come under rules established by the Clinical Laboratory Improvement Act of 1988 (CLIA). Those laws allow labs to gain pre-qualification to perform tests they’ve developed themselves. Those tests require minimal bench testing before gaining FDA approval for use. LabCorp, Quest Diagnostics and some of their smaller competitors also compete in the CLIA-governed space when genetic or rare disease testing market reaches a critical size.

CLIA rules governed the first diagnostic tests developed for COVID-19, a novel virus. HHS’ emergency declaration in late January threw an extra layer of paperwork onto the process, but essentially left test development to the CLIA-certified labs.

Every one of the 80 tests approved by the FDA for diagnosing COVID-19 came from firms pre-qualified as competent to conduct “highly complex testing” under CLIA. While that’s the highest level of pre-qualification, it doesn’t require any testing beyond bench tests on known samples.

That part makes sense to Dr. Gary Procop, the medical director of Cleveland Clinic’s labs and one of the first to raise qualms about the Abbott ID Now test. “There’s no time to do well-planned studies when a new pathogen appears on your shores,” he said.

But, “you’re going to want to compare your test to some kind of reference standard,” he said, “so then you can begin comparing tests to tests. All tests have slightly different characteristics on performance. You can then begin determining the tests that are most sensitive and the tests that then have not quite as a high sensitivity.”

“What happens if we have this outbreak for a year,” he asked. “We’ll be skating on emergency use authorization data. It shouldn’t be up to us at major medical centers to do this. They (the testing companies) have to be pushed to do this by the FDA.”

So far, it hasn’t happened. LabCorp in early May announced it will have the capacity to perform 150,000 diagnostic tests per day by June. Quest is aiming for a capacity of 150,000 daily tests by mid-June. Those two companies alone would achieve most of the federal administration’s limited goals.

Quest’s latest approved test may accentuate the problem of false negatives due to poorly collected samples. Dr. Jay Wohlgemuth, chief medical officer for Quest, said the company is already using short swabs that can’t reach deep into the nasal cavity at its drive-through sites at Walmart stores. “That’s consumer friendly,” he said.

And on May 28, the FDA approved Quest’s home test – “what we call physician-prescribed self-collection not observed by a provider,” he said. “The workflow becomes Quest shipping a kit (to the person) by FedEx, self-collection, put it back in a pre-bar-coded box, and it goes room temperature to a lab. This will help in minority and other communities not close to the healthcare system.” 

But will home testing give communities and public health officials an accurate picture of COVID-19’s prevalence and spread? The FDA did not require Quest, the nation’s second largest  lab company, to provide the data that would answer that question.

Sources:

1. See https://www.fda.gov/medical-devices/emergency-situations-medical-devices/emergency-use-authorizations#covid19ivd (accessed May 26, 2020).
2. See https://columbiasurgery.org/news/here-s-what-you-need-know-your-surgical-appointment (accessed May 26, 2020).
3. See https://delauro.house.gov/sites/delauro.house.gov/files/HHS_COVID_Testing_Report.pdf (accessed May 26, 2020).
4. See https://www.npr.org/sections/coronavirus-live-updates/2020/05/21/860480756/scientists-warn-cdc-testing-data-could-create-misleading-picture-of-pandemic (accessed May 26, 2020).
5. See https://apnews.com/6dbd9ad370add2ba299c7da46c25004f (accessed May 26, 2020).
6. See https://ethics.harvard.edu/files/center-for-ethics/files/roadmaptopandemicresilience_updated_4.20.20_1.pdf (accessed May 26, 2020).
7. See https://www.nytimes.com/2020/05/16/health/fda-clears-another-coronavirus-testing-kit-for-use-at-home.html (accessed May 26, 2020).
8. See https://www.npr.org/sections/health-shots/2020/04/21/838794281/study-raises-questions-about-false-negatives-from-quick-covid-19-test (accessed May 26, 2020).
9. See https://pubmed.ncbi.nlm.nih.gov/32425657/ (accessed May 26, 2020).
10. See https://www.nytimes.com/2020/05/04/health/fda-antibody-tests-coronavirus.html (accessed May 26, 2020).
11. See https://www.fda.gov/medical-devices/emergency-situations-medical-devices/faqs-testing-sars-cov-2#serology (accessed May 26, 2020).

Visit 4sight.com/insights to read more commentary on COVID-19 and including Part I of this series, Rogue Cops and Vigilantes: COVID-19’s Best Friends.