Care For A Mint?

A short note on PCR test specificity and false positive dishonesty

One of the trickier issues that can clobber any analysis is taking data at face value. By now I hope regular subscribers to my newsletters will have noted how weird little details and definitions sitting down in the fine print can snag the unwary. My favourite has got to be the daily count for COVID admissions to hospital, where we learn that ‘COVID’ does not necessarily mean sick with flu-like symptoms or even carrying any virus (false positive), and where ‘admission’ includes re-admissions as well as folk turning up in A&E under their own steam (and does not necessarily end in bed use, whether day or night bed), and where ‘hospital’ does not necessarily mean to a service provider that actually treats COVID cases (e.g. maternity or mental health unit). Those who have read some of my more formal reports will also recall that positive-testing admissions with flu-like symptoms may also be sent home the same day! People usually stand with an open mouth and puzzled expression when I relate the reality of the daily count - what it actually means - and to this end I produced this popular slide:

…but beyond such groupspeak and double-talk lies an even bigger problem in that analysts (including myself) assume that our counting of positive test results is a true count. It most certainly is not, for any diagnostic test will yield false positive results as well as true positive results, and false negative results as well as true negative results. Wiki has a handy little entry on this subject which may be found here.

In COVID-world false negatives become an issue if infection of the population is going through the roof for it means healthcare professionals will be missing infectious cases, thus adding fuel to the fire. In contrast, false positives become an issue if infection rates within the population are minimal for it means that healthcare professionals will be telling folk they are carrying the virus when they are most likely not. Unfortunately, healthcare professionals are so bound up with protocol and hitting management targets that they forget they can be administering a test whose results are likely to be erroneous, which leads the service into deeper and deeper water.

No Test Is Perfect

In the engine room of the pandemic sit the PCR and lateral flow tests. Originally developed for the replication of DNA from measly samples this mass production method somehow seems to have been flogged to health authorities as a fabulous diagnostic tool. No test is perfect and neither is the PCR test. If we rummage in the Office for National Statistics’ pantry we find this report on methods that spell out what we may expect of the PCR test in terms of sensitivity and specificity. The driver of false positives is poor test specificity so let us take a look at what this document actually states:

We know the specificity of our test must be very close to 100% as the low number of positive tests in our study over the summer of 2020 means that specificity would be very high even if all positives were false. For example, in the six-week period from 31 July to 10 September 2020, 159 of the 208,730 total samples tested positive. Even if all these positives were false, specificity would still be 99.92%.

As a former NHS suit my first concerned query in response to this rather woolly paragraph is why we don’t have robust figures to hand that have been reported in multiple calibration studies undertaken by independent laboratories. The short answer is that no lab possesses isolated and purified virus sufficient for test calibration; all they have are centrifugal isolates (not the same thing) and computer simulations. Yes, really.

Impeccable Logic

But let us not get distracted by the complete and utter failure of the industry on the most critical single issue in all of COVID-world; let us instead consider the impeccable logic of 159 out of 208,730 samples testing positive, this giving rise to 99.92% specificity. Whilst we can’t argue with that, what we can usefully do is pull down some fresh data from UK GOV just to check.

We can also go one better by junking the narrow analytical window of 31 Jul - 10 Sep 2020 and calculating rolling 7-day totals for PCR tests undertaken and cases detected using PCR for the period 30 Jan 2020 - 7 Jun 2022, thence to derive a curve for the minimum theoretical test specificity (MINTS) since COVID-world began. Here’s what this looks like:

What should immediately strike folk is that the calculation offered up by bods at the ONS gives very different estimates of test specificity over time and this is because the goalposts are constantly changing - something they omit to admit!

I’ve marked their 6 week reference window with dashed lines and we may note that they chose a time when disease prevalence would have been minimal and people generally happy in the sunshine. The figure I obtain for the period 31 Jul - 10 Sep using the rolling 7-day method bounces between a minimum specificity of 98.32% and maximum specificity of 99.33%, with a mean of 99.12%.

If I abandon rolling 7-day figures and simply count what UK GOV provide on a daily basis for the period 31 Jul - 10 Sep 2020 I obtain 58,080 cases detected using PCR and 5,910,541 PCR tests undertaken, thus giving an estimate of 99.02% specificity. But note the massive difference in tests and cases. This tells us that the authorities concerned were using a sample of test results for their report and not the national tally; the question then is whether that sample was representative. Case rate comparison for these discrepancies of the order of 0.9% reveal that it wasn’t¹.

But Does That Piddling 0.9% Matter?

Oh yes.

In their report the ONS claim a range of 85% - 98% for test sensitivity, so let us split the difference and go for 91.5%. Let us also suppose that 1% of the population are infected (1 in 100 people infected, a.k.a. pre-test probability). If we assume a test specificity of 99.92% as claimed then the false positive reporting rate is down at a modest 8.0%. If we now lower our estimate of test specificity just a weeny fraction to, say, my rolling 7-day mean of 99.12% then the false positive reporting rate climbs to a whopping great 48.8%. Scorchio!

For a test sensitivity of 91.50% and infection rate of 1% then the false positive reporting rate rises to 50.0% (1 in 2 folk told they are COVID cases when they are not) if test specificity drops to precisely 99.08%.

If the infection rate is down at 0.1% (1 in 1,000 people infected) and test specificity up at the claimed 99.92% then the false positive reporting rate soars to 46.6%. If test specificity is down at my 99.12% then the false positive reporting rate is a rather giddy 90.6%. When it comes to PCR test specificity teeny weeny fractions matter terribly!

N.B. You can check this out using the online calculator provided by the BMJ or you can try out my DIY calculator which sits on this Google drive.

In Plain English

Assessment of PCR test performance at the national level reveals 1 out of every 2 people are being told they ‘have COVID’ when they do not whenever the infection rate drops to 1% of the population (1 in 100 genuinely infected). If the infection rate drops to 0.1% (1 in 1,000 genuinely infected) then 9 out of every 10 people are being told they ‘have COVID’ when they do not.

In Plainer English

We are being shafted. Not only are we being shafted but our addiction with continually poking our noses for no good reason is making a dire situation even worse.

1. 159 cases in 208,730 tests yields a rate of 0.0008 cases per test whereas 58,080 cases in 5,910,541 tests yields a rate of 0.0098 cases per test, this rate difference being highly statistically different (p<0.001; comparison of Poisson mean rate ). No surprises here, then, for the official national case rate for England for the same time period is 12.9x higher than that ONS reported rate. Sure smells like fudge to me!