Primary Clinical Outcomes For A Single Emergency Department 2017 - 2021 (part 2)

SMLR modelling of 1.9 million admissions records for the emergency departments of an undisclosed NHS Trust for the period Jan 2017 – Sep 2021: predicting decision to treat.

So here I am at the beginning of an attempt to build another staged multivariate logistic regression (SMLR) model in order to understand the world. Sometimes this stats stuff is all rather surreal. In part 1 of this series I fleshed out two primary clinical outcomes for the emergency department (ED) these being decision to treat (treated; not treated) and disposal method (discharged; hospitalised). I’ve a shortlist of 13 independent and 2 dependent variables to consider (mostly in binary form) so let us start out by eyeballing the summary stats for the 1,530,522 adults that were aged 18 years and over on admission to the ED over the period 2017/w1 - 2021/w37:

As before the mean of any binary indicator variable coded as [0,1] provides a sample estimate of the proportion of cases. Thus we see that adult females slightly outgunned males over this period with a proportion of 0.52 (52%). We also see that the dominant condition was physical injury (including intoxication and poisoning) at 0.36 (36%). This is all as expected and matches my experience back when I was a suited thing annoying ED consultants.

Some readers may be surprised at a mean admission age of 49.31 years. As I’ve pointed out before front door medicine is largely for youngsters doing young things and coming a cropper. Oldsters tend to be referred to clinics and diagnostic services.

The arrival mode figure of 0.29 (29%) refers to the proportion of those arriving by emergency ambulance, as opposed to those making their own way. Readers who glue themselves to daily scaremongering (a.k.a. ‘the news’) might need to readjust their blue-light thinking.

The ED disposal route proportion of 0.22 (22%) refers to those who were hospitalised instead of being discharged home or discharged with referral to another service provider (e.g. physiotherapy). Thus we discover that only 1 in 5 admissions require hospitalisation, this being one of my two dependent variables I shall shortly attempt to model. Treatment Status is another of my dependent variables and we see that 0.59 (59%) of admissions received treatment of some kind in the department.

Modelling Strategy

As always there’s a zillion ways we can go about building a predictive model. In this instance I need to build 2 predictive models - one for decision to treat and another for disposal. Retrospective observational studies like this invariably fall foul of confounding factors and whilst I cannot hope to trap many of these down I can at least try and trap down what can be trapped down. In this regard, and after stroking my beard many times, I decided to start by splitting the data into three epochs: pre-pandemic (2017/w1 – 2020/w10, n=1,082,962); COVID (2020/w11 – 2020/w49, n=198,052); post-vaccine (2020/w50 – 2021/w37, n=248,805).

The first of these epochs, being COVID- and vaccine-free, can be used to derive risk of treatment/hospitalisation scores under normal service conditions. Retrospective observational studies that don’t utilise some form of risk/propensity score need to be sneered at, with ripe words thrown at the authors daily until they repent. Everybody knows that older folk tend to be sicker, and that folk with serious conditions like atherosclerotic heart disease and cancer tend to require extra care and attention. Everybody also knows that ED intake varies with the seasons, and will vary in the longer term; so they must also realise that the playing field is not only not level, but is changing from week-to-week, with surges of high risk cases one month and little the next. The knock-on effect in terms of my two dependent variables is that both decision to treat and disposal route will be affected over time in a non-random manner. A surge of cardiac cases will elevate the likelihood of both, and a surge of hay fever will serve to deflate the likelihood of both. We thus have to find some way of accounting for these changing case profiles and a jolly good way is to derive a modelled risk score. If this sounds a bit fruity let me put it in the form of two parallel questions:

• What is the risk/propensity for an admission to be treated within the department given their age, sex, method of arrival, number of diagnoses and diagnostic profile?

• What is the risk/propensity for an admission to be hospitalised given their age, sex, method of arrival, number of diagnoses, diagnostic profile and decision to treat?

If this were a qualitative study we might categorise individual admissions into high, medium and low risk groups and match cases that way (stratification), but since it is a quantitative study we need some decent numbers and we’ll get these by running a model across the pre-pandemic data of 1,082,962 adult admissions for the period 2017/w1 – 2020/w10 to obtain a set of coefficients that pertain to ‘normal’ service.

With some very basic risk scores in our pot we can move on to the second phase of the study. What I’m looking for is whether the likelihood/risk/propensity of treatment within the department was elevated over the period 2020/w11 – 2020/w49 compared to ‘normal’. To minimise seasonal trickery I decided to set the control period to 2019/w11 – 2019/w49, this providing a matched week analysis. This approach generates a rather useful binary indicator for control vs. study periods that can be utilised as an independent variable. Likewise, the control period for the vaccine period was set to 2018/w50 – 2019/w37 to match the post-vaccine data sample of 2020/w50 – 2021/w37.

I think it best to stop wittering there and to turn the handle on some funky SMLR since I can explain more of my flamboyant strategy as puddings pile out of the oven…

Decision To Treat

Risk Score

Just to recap that the risk/propensity score for decision to treat was derived using the pre-pandemic period of 2017/w1 – 2020/w10, for we want to know how things normally are for adult admissions to the ED. To generate the score a single stage multivariate regression was run using decision to treat as the dependent variable along with the 13 independent variables listed in the table above. As it so happens 13 independent variables generate no less than 78 two-way interactions and 286 three-way interactions so a full factorial model to the level of three-way complexity requires submission of 377 terms in total. This is crazy, and largely pointless, since the CDS 010 dataset in real-life use was geared to recording one primary condition per patient, so there’s no point chasing an interaction between, say, cardiac, cancer and respiratory conditions.

After some thought (and more beard stroking) I settled on a model with 13 main effects and 46 two-way effects by way of clinical compromise. This took a fair time to crunch, even on my quad core, and the resulting model structure is way too large to present as a pasted table. In fact, out of those 59 terms, no less than 44 were incorporated into the final model during a forward (conditional) selection procedure. Despite a sizeable model classification performance wasn’t stonking, as may be seen from the ROC analysis below:

We’ve got an area under the curve of 0.675 which may be compared with a pure guess at 0.500 and utter predictive perfection at 1.000. This might sound dismal but we are trying to second guess what medics in the ED are going to do when a person comes through the door. I can add additional terms to the model but experience shows these are going to yield a rapidly dwindling return. This is what we are stuck with, so we must lump it or leave it! Herewith a summary table of the risk score for all adult admissions from Jan 2017 – Sep 2021 (n = 1,350,032):

Scores spanned the range 0.144 – 0.985, with a mean score of 0.608 and median of 0.570. We might like to compare the modelled mean and median risk score with the Treatment status mean value of 0.59 given in the very first table to discover nothing dodgy is going on! As regards the distribution of these scores among the admission population then there is another story to tell but to tell that I need to trim the sample to just those 321,193 scores derived for the calendar year of 2019 otherwise my graphing module would grind to a halt:

There we go – as complex as you like. This distribution doesn’t just reflect the case profile of admissions but every single factor you can think of that will impact on the decision to treat. So we’re not just talking clinical symptoms and protocols but bed availability, workload, management targets and all the rest. If we take the mode then there’s a 50:50 ‘risk’ of being treated, though perhaps we should call it an opportunity! That bump up at 0.78 or thereabouts will represent cardiac admissions, though serious injury will dominate as we have seen.

Comparison of Periods: Cross-tabulation

Before we launch into the deep end I thought we might consider a very simple cross-tabulation that attempts to determine whether the likelihood of treatment on admission was elevated during the COVID period:

Here we have 227,015 adult admissions during the control period of 2019/w11 – 2019/w49, 142,655 of which were treated in the department (62.8%), compared with 162,149 adult admissions during the COVID period of 2020/w11 – 2020/w49, 102,968 of which were treated in the department (63.5%). If we run a Fisher’s Exact Test on this 2 x 2 table we arrive at a highly statistically significant p-value of p<0.001, but with a sample size up at n=389,164, then p-values were always going to be off-the-scale significant: an issue often referred to as the large sample size fallacy. What we’ve got in the real world is an effect size that amounts to a 0.7% increase in the likelihood of treatment during the COVID era, which is not a lot given the dramatic changes in provision of healthcare and patient profile that took place. In odds ratio terms we’re looking at a factor increase of 1.011, indicating a 1.1% increase in risk of treatment, which is pithering.

The BIG problem here is that, in terms of confounding factors, we’ve only matched weeks across 2019 and 2020 and haven’t paid any attention to the differences in the admission profile in terms of age, sex and diagnosis. We might be comparing chalk with cheese for all we know! This is where that risk score comes in, for that score has attempted to account for age, sex, number of diagnoses made, arrival mode, and presence of nine medical categories, as well as a few key interactions. It’s not perfect by any means but it’s going to be an improvement on the crude tabulation provided above.

Comparison of Periods: Mean Risk Score

What we can do now is ask if the mean risk score for the COVID period differed from the mean risk score for the control period to get a handle on potential changes in case profile (and thus likelihood of treatment). I shall start with a very boring summary table:

All looks pretty samey, doesn’t it? What I rather like are the highly similar standard deviations which tells me variance hasn’t really changed, which in turn tells me that the variety of admissions across each period is similar, this being a good thing ‘coz we are going around assuming a great deal. As for the means and medians then there’s a hint of a hike in the overall risk profile offering a factor increase of x1.013/x1.014 respectively but it’s nothing to write home about.

Missing Mash-Up

Just in case some eagle-eyed reader has spotted a drop from n=389,164 for the cross-tabulation to n=352,629 for the comparison of mean scores then in all this we must remember that missing data is going to play havoc with sample sizes. What I ought to do is get disciplined and restrict the data sample to adult admissions where a risk score has been generated *and* a treatment outcome has been recorded. Here’s how the cross-tabulation now looks:

Well isn’t that darned interesting? The crude odds ratio has now dropped from 1.011 to 0.989 simply because the sample has changed slightly. This is not exactly inspiring confidence and I’m going to stick my spoon out and say that there’s no genuine and significant difference in the likelihood of treatment between control and study periods. A bit odd for a novel and deadly pandemic, don’t you think?

If we take these basic analyses at face value we’ve a very modest hike in the risk score during the COVID era and a very modest drop in likelihood of treatment for adult admissions. But… as we’ve seen, the distribution of scores is complex such that the meaning of means is largely meaningless. What I better do, then, is pull out the logistic spanner once more to settle the score…

Settling The Score

The logic of this logistic is very simple: use the risk score as a covariate in a basic model of predicting treatment outcomes on an individual basis. In many ways this is akin to the concept of a propensity score; except I’m not case matching but fiddling with covariance space. This is not unlike using a hunk of bread to mop up the gravy in order to see the peas. Try this:

Here we have a very simple model with just two main effects and their interaction. We expect the risk score to mop up a load of gravy in the prediction of treatments dished out, so an astronomic odds ratio of 44.996 (p<0.001) shouldn’t come as a surprise. What does come as a surprise is the odds ratio of 1.743 (p<0.001) for the period indicator that indicates that treatments within the ED were 1.742 times more likely to occur in the COVID period compared to the control period once base risk has been taken care of. It’s not all plain sailing, though, for the interaction term of OR = 0.364 (p<0.001) serves to moderate this main effect with gusto. I suspect a slide is required:

Ain’t that pretty? We’ve got a flip point sitting just under the overall mean risk score of 0.624 that indicates an elevated likelihood of treatment within the ED for those with lower risk scores that turned up during the COVID period. Equally, there appears to be a reluctance to treat within the department for those turning up in a bad way during the COVID period. Whoops!

This is not exactly honest-to-goodness evidence-based medicine as I know it, and would suggest that protocol and senior management ruled during the COVID era. This is not new news to those who’ve followed the sorry tale of healthcare failure over these last four years but it is good to see the numbers dance a dance in support of what we’ve all witnessed. From my point of view as cruncher of numbers it feels good to produce something that makes sense of what would otherwise be a quagmire of confounding factors and nonsense comparisons. I shall proceed by repeating this exercise again for likelihood of hospitalisation - get the biscuits in!

Kettle On!