I continue to be disturbed about the way we are driving the COVID-19 testing machine. I have been critical previously over the increased attention to testing asymptomatic individuals. The UK Government recently announced what I have dubbed their “Cuddles for Christmas” approach of rapid testing in care homes in England to allow relatives to visit their incarcerated loved ones without the need for social distancing. On the face of it, this sounds a wonderful idea but my worry is the results of these tests are taken as absolutes and nothing could be further from the truth.
Nature Reviews Microbiology recently quoted the sensitivity of the PCR tests as 63% for nasopharyngeal swabs. The article did not cite a specificity but advised, “No test is 100% specific.” For the PCR, most thinking suggests a specificity of 99.96%. What this means in practice is determined by the number of cases in a given population. In a recent meeting in our health board, it was announced that there had been 800 tests on one day. To therefore give a sense of what this means in practice from an asymptomatic testing perspective, I calculated what results we might expect for 800 such tests. The prevalence in the Hywel Dda Health Board three counties is currently around 200 cases per 100,000 population or 0.2%.
What this shows is that if 800 asymptomatic relatives had a test before seeing their loved ones, 798 would be true negatives and safe to visit. However, at this prevalence, there will be two people positive for the virus. One will be detected and turned away to self isolate for 10 days, the other will not. As this is missing half the positive tests, is this a good way to spend an awful lot of money? These are not low cost tests and we will still be allowing virus into our care homes.
The same argument can be levied at the testing we do when we are bringing people in for planned surgery or chemotherapy. Indeed, in the health board paper I wrote and linked to previously to illustrate these effects of sensitivity and specificity, I proposed rather than the huge costs associated with this asymptomatic testing, it would be just as reasonable to not test but rely on patients self-isolating for the two weeks before their procedure. The same arguments apply: we will detect 1 case in 800 but also miss 1 case in 800.
This week, the Welsh Government has announced routine testing of health and social care staff to begin in December. This is a further distraction that we really do not need when the vast majority of staff will always test negative and based on the same figures above, we will only detect half the positives and therefore while we can remove one positive person, there will be another we will miss. The Welsh NHS Confederation advised there were 89,079 people employed in our NHS in September 2016. At the prevalence above, that would mean there would be 178 positives but we would only detect half of them, while 89 would continue to work. One company advised they were making their tests available “at cost” at £225. I tend to think that our lowest kit cost is around £40, which for 89,079 staff in Wales, would cost £3,563,160 each time we tested them. The reality of course is we have established a huge machine of staff to do these tests and if we took that figure of £225 per test, then each time we tested the NHS workforce, we would need over £20million.
What is becoming clear however, is these newer tests do not perform as well as the PCR tests we have been using up to now. So if we conservatively drop the sensitivity to 60% and the specificity to 99%, then the tests become even more dubious:
In a low prevalence population with a reduced specificity, among our 800 we still have 2 people with the virus. As before, we are only going to detect one of those true positives and allow the other one to stay at work but still pose a risk of starting an outbreak. But we are also sending a further 8 people with false positives home for 10 days.
That means with our 89,079 staff we get:
So do this exercise once (and that is not the plan, I would remind you) and we will be asking 996 of our staff across Wales to self-isolate for 10 days. A band 5 health care worker currently earns around £30,000 per year so 10 days salary is about £820 or £816,720 for all these staff. If we can backfill these posts, then we have to find another £816,720. And we still have 71 positive people at work with the virus and potentially liable to start an outbreak in a closed setting.
(Some may be wondering why in the table above the number of true positives and false negative are not the same, compared to the previous table. This is because the figures are rounded up or down as people do not come as 0.71 or 1.07 – these are statistical calculations)
There is no getting away from the fact that as I write, the worldwide number of people who have had this virus is now over 66 million and the number of deaths over 1.6 million. These are horrendous numbers and are responsible for the emotional responses we have seen to a lot of the actions we have taken. I also will continue to commend all colours of politicians who have to make very difficult decisions on our behalf, some of which like many commentators I may not like. But as highlighted above I think we have got the whole issue of testing terribly wrong. I know the World Health Organization said, “Test, test, test!” but I wish they had said, “Test, test, test symptomatic cases.” Now if you check the link above, the Director General said, “Test, test, test,” twice. The first time however, he did expand and said, “We have a simple message for all countries: test, test, test. Test every suspected case” Did he mean we should be doing more asymptomatic testing than symptomatic? I would venture to suggest not because the only tests we have available, are not suitable as screening tests because of the poor sensitivity illustrated above.
So lets look at how Symptomatic testing performs:
The 800 people I mentioned as my starting point was the number associated with symptomatic testing on one day in my health board area, who had booked themselves to be tested via the gov.uk portal. So I also did the calculations for a symptomatic group of individuals. From our own laboratory tests on paediatric admissions, ITU admissions and symptomatic care home residents, for those with respiratory symptoms who tested negative for COVID, the other virus we have detected is mainly rhinoviruses with a few adenoviruses. So I based my calculations on a symptomatic population with a prevalence of 50% and this is how the numbers then look:
Again at this specificity, using the PCR, we can rely on the positive tests all being true. Among the 800, there are 148 false negatives and here is where my second worry comes into play. Because of the reliance on available testing, we are having a further detrimental effect on the behaviour of our healthcare complex adaptive system. Indeed, the advice we are giving is totally wrong in my view. This “easy read” document from the Welsh Government says:
“You will be told your result by phone, text or
email.
• If your test is negative, you and the people
you live with no longer need to stay at home.
• If your test is positive, someone from the NHS
will contact you.”
This is reiterated in the formal guidance. It has to be said, the England guidance is more in line with the approach advocated at the end of this blog post.
Of our 800 people tested with symptoms, the Welsh Government are telling 148 people with false negatives that they no longer need to self isolate! Now we know that one of the main amplifiers of any respiratory virus is the household, so we are likely to be releasing more than 148 people with the virus into the community to infect others while they glibly tell all their friends and neighbors over a pint in the pub that they tested negative.
It has to be said, the newer rapid tests perform in a similar vein and will only find 4 false positives who would be asked to self-isolate with their households inappropriately:
So how should we be doing this?
As I illustrate above, I think we have got this totally wrong. Testing asymptomatic individuals to find the positives is as good as a coin toss: at the current prevalence, 2 people in our 800 will have COVID: toss a coin and we detect 1 but not the other. There is no value to testing asymptomatic people with a diagnostic test.
A diagnostic test is designed to confirm a clinical suspicion i.e. this person has respiratory symptoms, could it be COVID? Our PCR test will confirm 252 of the 400 do indeed have COVID. It will miss 160 but they still have symptoms either COVID or another respiratory virus! All respiratory viruses are capable of causing severe disease and death, especially in vulnerable groups.
Lets compare with how we normally manage a respiratory outbreak in say a care home or ward, which we have done every year that I have been a microbiologist i.e. 34 out of the last 35 years. We take a throat swab from every symptomatic individual and start putting through our test systems. In the early days it was expensive cell cultures and I cannot remember the statistical rules we used to determine how many we would test. But since the advent of molecular PCR technology, we start testing them in small batches, usually bathes of three. Once we have three positive tests all with the same virus, we stop testing. From here on in, anyone who meets a case definition that the outbreak control team agrees is declared a case. They do not need to be tested unless they do not meet the case definition for some reason i.e. does the person have a different infection?
We always assume there will be asymptomatic individuals (I said that in my very first blog post) but as we have seen with all the preamble above, we will never detect them all and we have to accept that as the reality.
We manage individuals based on their symptoms. Anyone with symptoms is a risk to others. A previous blog post offered a way of showing the massive first peak of cases. I now have a sneaky feeling that access to testing is one of the drivers behind the amplification of the current case load and that if we did not have mass testing available, we would be seeing a quicker decline as in the period May to July. The rules then were different, because we had no mass testing. “If you have symptoms, you and your household must isolate.” Now we allow 148 of 400 with the virus to carry on spreading their virus, rather than waiting for their symptoms to get better.
Our normal rules around respiratory outbreaks are clinically driven. We advise people to self isolate while they have symptoms. When they are better, we ask them to wait two days before they re-establish their normal work-life patterns. Where we are dealing with a person with symptoms but who has tested negative, there is no reason for us not to do the same with this virus. This is supported by a pre-print review paper still undergoing peer-review which identified a number of papers where viable virus cannot be detected beyond 8 days from symptom onset, often despite the PCR remaining positive. The PCR can detect fragments of viral RNA but viable virus is required for transmission. This also exposes the futility of requiring patients who are to be transferred from hospitals to care homes to have a negative test, another example of the waste associated with asymptomatic testing.
There is a better way:
- Test symptomatic people only but offer the test to those with all the wide range of respiratory symptoms (I’m repeating myself now.)
- Ask all people with respiratory symptoms to get a test and self-isolate with their household contacts.
For those where the index case in the household tests positive:
- Ask them to self-isolate for a minimum of 10 days. If they are still unwell at 8 days from when symptoms began they should continue to isolate for 2 days after they feel better.
- All household contacts should self-isolate for 14 days unless they develop symptoms. Any new person with new symptoms can be tested if they wish but regardless of the result, their symptoms should be presumed to be due to COVID because of their link with the first household member and they should restart their personal self-isolation count to the day their symptoms began and self-isolate for a minimum of 10 days as discussed above.
For those where the index case in the household tests negative:
- Ask them to self-isolate while they continue to have symptoms and for a further 2 days after their have ceased.
- Household contacts can be released from quarantine, so long as none have developed symptoms.
- Household contacts who develop symptoms should undergo testing and all others should remain in quarantine, including the original index case if they had not completed the 10 days minimum. If at any point one tests positive, then any symptomatic cases in the household should also be presumed positive and be subject to the minimum 10 day self isolation.
Where there is a suspicion of a local cluster/outbreak in a closed environment:
- Test symptomatic cases first to establish the causative virus.
- This should be the only situation where asymptomatic testing may have a role in managing the outbreak. This is because in such a situation the tests will be performing in a high prevalence area, where the results may prove effective as part of the control measures.
Stop testing asymptomatic people in low prevalence situations:
As illustrated above, the test fails to find half the low numbers of positive individuals and is no better than a coin-toss. It is associated with a huge financial resource and is indeed wasting money on an industrial scale. This includes such screening as a means of visiting care homes. The current approach makes people think that if they are negative, they are “safe.” This is just not the case and it would be better to allow relatives to self-assess themselves through personal reflection about any respiratory symptoms before they visit.
At a time when we need to be moving as much of our precious NHS resources to allow for mass vaccination of the population, stopping this waste will mean funding and more importantly, human resources can be channeled into this next important task on our collective journey.
Thank you. I cannot replicate part of your analysis. In the case of 60 sens / 99 spec for 89,079 tests, at 0.2% prev, we expect 178 positives. In your figure, 996 are observed positive after test. Given that only max 178 can be positive, I cannot understand how 996 would come positive from the test. Thank you for your clarification, Fred
Many thanks for taking the time to digest the post, Fred.
An ideal screening test needs to have as near to 100% sensitivity and specificity as possible to avoid false positives and false negatives.
You are referring to the third table, where I have done the calculations for the 2016 estimate for the number of staff in the Welsh NHS. This is where I have reduced the specificity from 99.96% to 99% and the sensitivity from 63% to 60%. Probably easier to see this by comparing the first and second tables. The first uses 99.96% specificity and 63% sensitivity. The second uses the lower 99% specificity and 60% sensitivity and because I have set the number of tests at 800 for the reasons outlined in the post, is approximately 10% of the NHS population figure. Therefore the third table has 996 total positives, table two has 9 positives. Now compare these positives with those in the first table where we are at the higher specificity and instead of 9 total positives, we have 1. This illustrates what happens as the specificity falls, we begin to see the false positives increase. Therefore, when looking at the population of NHS employees the number of true positives is 107. The remaining 889 are false positives. This is why I get so exercised about the waste associated with this proposal. For every 9 people who test positive with these new tests, 8 will be false positives and could still be at work.
As I note at the end of the blog, if we are trying to manage an outbreak where the prevalence is much higher, there is some value in screening asymptomatic staff, particularly if we use the normal PCR where false positives are rare.
My spreadsheet is available to download as an embedded Excel spreadsheet in the health board paper I originally set this at 98% sensitivity and specificity but you can manipulate the figures and see the effects discussed above and the Word document explains how to work with the spreadsheet.
Best wishes, Mike
Thanks Mike, good reading and analysis
Thank you so much Mike for this post. It deserves to be widely read but it probably won’t in the “testing frenzy” we are in.
Ill considered testing, like fresh air ventilation and lots of long hair draped around face masks (which appears to be the reverse of hygiene for me) seem to be some of the elephants in the room of Covid-19 situation.
Your contributions to debate are much appreciated.