Randomised Vaccine Benefit

Lessons from an undisclosed NHS Trust

Back on 31st October I penned my 18th report entitled Assessing Vaccine Benefit: A Simulation Study Using In-hospital Death. This summarised a body of work I had undertaken in an attempt to understand a curious result; namely that vaccine benefit (in terms of reduced likelihood of in-hospital COVID death) could be established for some months but not others.  This simply did not make sense so I took 9,831 deaths occurring over the period 8/12/20 – 22/9/21 for an unknown NHS Trust and subject them to a simulation study.

Simulation studies are a handy technique whereby we deliberately mangle real world data, or synthesise data to understand what may or may not be going on in a complex situation. For this study I decided to try different ways of interfering with EPR data to see how this impacted on estimates of vaccine benefit. Thus, vaccination status was initially lagged by 1, 10, 100, 200 and 400 records with respect to COVID status within the EPR. Sliding records by just one case should be sufficient to annul benefit (think of this as mixing Mr Smith’s vaccination status at death with Mrs Jones’ COVID status at death) but to my great surprise vaccine benefit was still apparent even if patient records were knocked out of kilter by 400 cases!

I then started fully randomising the EPR month by month. Even with the months Dec 2020 – Apr 2021 fully randomised (58% randomisation) I was still obtaining apparent benefit. This should not be possible. The process was repeated for May – Sep 2021 (42% randomisation) and still the result persisted. Startled by this I flipped to comparing second dose benefit with initial dose benefit for non-randomised data and found another topsy-turvy result; namely, second dose benefit was apparent for Dec 2020 – Apr 2021 but not for May – Sep 2021.

These highly irregular results were discussed with Professor Norman Fenton and Professor Martin Neil of Queen Mary, University of London and we came to the conclusion that systemic bias in a complex care setting is forcing an illusion of vaccine efficacy when there is likely none: it should not be possible to randomise diagnostic and treatment records and still obtain estimates of benefit! Subsequently, Professor Martin Neil led a team to produce an excellent paper exploring the issue of hidden bias.

This is a monumental finding, for if quasi-randomised EPR data can still show an apparent benefit of vaccination, how can we then glibly publish analyses claiming the apparent benefit we do see is genuine? The answer is we cannot, and it was for this very reason I stopped analysing EPR data: if we cannot accurately assess benefit, neither can we accurately assess harm. Never has a report depressed me so much!

..and this is before we consider the curve ball of natural immunity from prior infection being labelled as vaccine-induced immunity!