by Kurt B. Johnson
The Best Approach to SARS-CoV2 Testing
Some of the key questions Americans want answered right now about the novel coronavirus do not require the hundreds of thousands to millions of tests being touted by the media and epidemiologists. How many people have COVID-19 or its antibodies in the United States? What is the true positivity rate? How many tests does it take before we can safely reopen?
Given that the positivity rate has been reported at about 20% thus far, it should buttress the hopes of many out there to learn that we can determine whether the true positivity rate for the population is 1% or less by conducting only 459 random tests. If 1% is too high, consider that only 4,603 random tests would be needed to determine if the rate is 0.1% or less and only 45,999 for a rate of 0.01%. For reference, the United States has 242 total cases per 100,000 people for a 0.242% rate. A state would only need to conduct this many tests on a weekly basis to determine their status in controlling the disease.
The states with sample results that do not meet their target rate would need to increase restrictions. The states that meet their target rate can reduce restrictions. We can do this with a technique in longstanding use in audit: attribute sampling. Attribute sampling is used to show whether the proportion of a population is below a certain amount called the materiality threshold. Incidence (new cases only) and prevalence (new and preexisting cases) are proportions of the population that have or had the coronavirus.
Indeed, the Unites States needs to implement attribute sampling studies on a state-by-state basis to audit each state’s population for incidence of coronavirus or possibly even prevalence if the test can detect those who have had the disease but were not tested. Given the United States already tests over a million people per week, critical knowledge could be gained from utilizing less than 1% of that test stock to randomly test the population.
Issues With Other Approaches
Articles by Keith Collins of the New York Times and Ashish K. Jha formerly of the Harvard Global Health Institute and now at Brown University have claimed that at least 500,000 tests would be needed every day to enable the United States to reopen its economy by easing social distancing restrictions. Dr. Jha employs the test positivity rate of 20% reported in an article in The Atlantic in the calculations made to arrive at the 500,000 test figure. This test positivity rate is a biased statistic because it is measured from those who have experienced severe enough symptoms to be tested. The Atlantic article explains that “if the U.S. were a jar of 330 million jellybeans, then over the course of the outbreak, the health-care system has reached in with a bigger and bigger scoop. But every day, 20 percent of the beans it pulls out are positive for COVID-19.”
Actually, the health-care system has not reached in with a scoop. Instead 150,000 of the 330 million jellybeans are hopping out of the jar and saying, “test me”. The mathematics employed in the 500,000 tests per day calculation involves go nowhere logic. They start by saying the positivity rate is 20% for the 150,000 tests they conduct per day because they observe about 30,000 COVID-19 positives out of the 150,000. Then they say that 20% of those 30,000 will have serious symptoms. This means 6,000 will have serious symptoms. Somehow this is supposed to imply that there are 150,000 new cases in the United States per day. Why one would divide the 20% into the 30,000 is a mystery.
The calculations go further to say we must cast a wider net by testing more of the people who self-report symptoms. This recommendation essentially continues to build on this biased test statistic. Instead, we can gain better knowledge and cast a much wider net by more efficiently using only a fraction of our test capacity on a random sample of the population.
What remains a mystery is the actual prevalence of SARS-CoV2. As Daniel Westreich, an epidemiology professor states the Atlantic article: “If you want to interpret [the positivity rate] as a hint to prevalence in a particular location, you have to assume lots of other things stay constant. We just haven’t tested enough people yet,” he said. “If you were doing random screening of the whole population, we just don’t know what you’d see. We don’t know how many asymptomatic viral shedders are out there.”
The Solution
We would know that the number of asymptomatic viral shedders is less than 1% if we randomly test just 459 people and they all come up clean. And you would have 99% certainty that your conclusion is correct. Furthermore, this sample size is virtually independent of population size. If you want to know if the number is less than 1% for the entire United States, you only need to randomly test 459 people from the entire country. If you want to know this for a particular state in the United States, the answer is almost the same whether its Nevada or California: 459 for California and 458 for Nevada. The numbers stay roughly the same for counties/parishes or cities within states. This number changes slightly if you are talking about counties or cities with less than 100,000 in population. Then it ranges down to 450.
Auditors have been using attribute sampling to test for population proportions for decades. Typically, it is used to test if procedures and controls are followed and measure the population error rate associated with them. For example, did the Medicaid agency verify the person’s income and social security number before granting their Medicaid application? Did someone squeeze in through the cracks somewhere and get grandfathered in who does not meet the qualifications? Attribute sampling allows us to measure this proportion and test to see if it is less than an amount the auditor is willing to tolerate: the materiality threshold. All a state needs to do is set this threshold.
How Attribute Sampling Works
Let us say Stephen Curry claims he can shoot 90% from the free throw line. That means an error rate of 10%. To test whether Stephen Curry’s free throw error rate is 10% or less, he would need to sink 44 free throws in a row to be 99% confident that he can indeed shoot 90% from the free throw line. The number is 44 because 0.944 is about 0.97%, just under the 1% we need to be 99% confident. Note that 43 free throws are too few because 0.943 is about 1.08%.
What if he misses a free throw? In that case, if he makes 63 out of 64 that can prove with 99% confidence that he is a 90% free throw shooter. That is because 0.964 + 64*0.1*0.963 is 0.96%. Note that 63 free throws are too few because 0.963 + 63*0.1*0.962 is 1.05%. It is too high to subtract from 100% and still be above 99%. Because we are sampling with replacement from Steph Curry, we are using the cumulative Binomial distribution to perform our statistics calculations. The calculations are even easier for a 50% free throw shooter like Shaquille O’Neal. Shaq can prove his free throw shooting error rate is 50% or less by making 7 free throws in a row or by making 10 out of 11 (0.57 = 0.78% and 0.511 + 11*0.5*0.510 = 0.59%).
Testing for coronavirus requires sampling without replacement because we are only going to test a person one time. For this we employ the hypergeometric distribution in the same fashion as we did the Binomial distribution for free throws. In attribute sampling audit parlance, an error would be testing positive for the coronavirus.
The following table shows the sample sizes that would be required to test for incidence/prevalence rates of 1%, 0.1% and 0.01%. The first three columns show the equivalent of missing no free throws. If you randomly test 459 people in California and none test positive for the coronavirus, you can be 99% confident the true rate is less than 1%. Test 4,603 and none test positive then you can be 99% confident the true rate is less than 0.1%. If one person out of the 4,603 tested positive, then California could test 2,032 (=6,635-4,603) additional random people. If they all come up negative for coronavirus, then California passes as having a true rate less than 0.1%. The value 6.56 million is used to represent the average state population. As you can see, the numbers are barely any different for any state population.
These calculations can be made as shown using ADA’s Attribute Sample Planning module:
Conclusion
Government officials, health experts and the public at large should recognize that employing attribute sampling can provide great levels of insight into the country’s coronavirus predicament. It provides the core technique to allow health experts and epidemiologists to learn the fundamental statistics they need to make better predictions about how to go about combating this disease. Attribute sampling can complement the work of a suppress and lift strategy like that discussed by Gabriel Leung. Stanford scientists in California have performed the kind of sampling study suggested here with 1.5% testing positive. However, instead of 3,300, only 460 people needed to be sampled to reach a conclusion that the rate is not less than 1%.
Without absorbing needed test resources from ill patients, a small, manageable amount of random sampling can provide critical information. This information can be updated on a weekly basis to allow a state, region, county or city to keep tabs on its progress in its fight against the coronavirus. It can be used to prevent the sudden wave of new patients that many are afraid will unexpectedly begin arriving at hospitals if restrictions are lifted.
It is important to make a distinction here. This sampling should done in addition to the needed tests for detecting the virus in patients showing COVID-19 symptoms as well as to conduct contact trace testing. Both are required to combat the illness and bring the level of incidence down if a state needs to increase its restrictions.
