COVID vs. COVID (part 2)

Further investigation of differences in COVID diagnoses as derived from application of machine learning (MLP) and as recorded in the EPR of an unknown NHS Trust

In part 1 of this mini-series we took a quick look at how the COVID diagnosis as registered in the EPR of 19,457 in-hospital deaths stacked up against the predicted COVID diagnosis I have derived using machine learning applied to a matrix of symptoms, age, sex and disease prevalence. We observed a decent overlap of 86% of cases with just 14% falling into the ‘we-better-query-these’ category. This morning I’m going to follow through on the ‘we-better-query-these’ category by… er… running some queries, and I shall start by creating an indicator variable (COVagree) that identifies the 2,767 cases that were not in diagnostic agreement.

At this point I could sit with umpteen tables and a bucket of slides and work my way through trying to ascertain which, if any, factors stand out in distinguishing diagnostic agreement or I could rely on machine learning again and use multilayer perceptron to tell me in 6 seconds flat what it thinks is important. Herewith its final decision:

Case Complexity

The total number of diagnoses made in the EPR comes out top dog in distinguishing cases so we better have a look at a table of means:

Aha! We now see that disagreement tends to happen with the more complex cases, which makes sense. Sitting in second place are injury and trauma cases with 60.3% normalised importance.

NOTE: Folk are bound to ask what ‘importance’ means and how it is calculated so I shall respond by saying this parameter drops out of a sensitivity analysis. In essence the model is run multiple times, each time omitting one of the independent variables. Quite what calculations are made is not revealed but I am going to guess we’re talking minimization of sums of squares or cross-entropy error.

Injury & Trauma

Let’s crosstabulate COVagree against injury & trauma cases:

Again this is making total sense because disagreement tends to occur when staff are not dealing with injury & trauma cases. It must be pretty hard to categorise somebody with lacerations as a COVID case but then again I’ve seen some pretty loopy social media commentary over the last three years! The good news is that this sort of nonsense doesn’t apply to the EPR I’m using since it merely lists up to ten diagnoses.

Case Detection Rate

Case Detection Rate (CDR) is a variable of my own baking, being the rolling 7-day sum of daily new cases divided by the rolling 7-day sum of daily new viral tests as gleaned from the UK GOV coronavirus dashboard for the nation of England. I’m using this as a proxy for disease prevalence and you’ll find it cropping up in many analyses that I’ve undertaken. Admittedly this is not the best measure I could use and ideally I’d have data permitting me to calculate prevalence for the NHS Trust in question since COVID is nosocomial in nature. However that must remain a pipe dream, so let us carry on and produce a table of means for CDR factored by COVagree:

Now this is rather fascinating! Disagreement between EPR and MLP-derived diagnosis tends to occur when disease prevalence is higher. This is national disease prevalence, though, and so it is a great shame I cannot derive a local measure. This also points to a time-based factor that we ought to squizz at next:

I say! Now that is an utterly gobsmacking finding. My eyeballs went straight to that whopping great red peak during the first 10 weeks of 2021 that isn’t supported by a corresponding hike in cases detected using machine learning. Since the machine learning approach is multidimensional in nature and is based upon a large array of factors for a large sample of deaths (see this article), then we must ask just what the heck was going on within this Trust to yield a surplus of dodgy COVID diagnoses over such a short space of time.

In fact, if I’m going to get all rough-puff pastry about this, I’d say that’s the nubbins of the diagnostic problem right there - a window in time when common clinical sense didn’t prevail and hyped-up test results did. I most certainly did not expect this, so I shall retire with a box of fresh cream éclairs to cogitate properly on the matter.

Summary

• Machine learning (multilayer perceptron) was used to provide a shortlist of factors that were associated with diagnostic disagreement.

• The three factors rated as the most important were case complexity, indication of injury & trauma, and a proxy for national disease prevalence (case detection rate).

• Diagnostic disagreement was found to peak significantly during the first 10 weeks of 2021, with in-patients being assessed as being COVID-19 positive when a neural network model suggests otherwise.

Kettle On!