[Previously posted over here]
What these guys looked at is a real challenge. How do i tell if the kid in front of me has just “the snuffles” or is in the early hours of something terrifying like a pneumococcal sepsis?
They do what everyone does these days and try to come up with a prediction “rule” that you can type into your iPhone and tell you what to do with your patient.
This could be a poster child for a badly done derivation set. Or let me take that back, the derivation was well done, the variables they chose to look at where silly.
They used terms like SBI=α+β1X1+β2X2+β3X3 so they must know what they’re doing right… right?
Serious Bacterial Illness (SBI) was defined kind of weirdly. They suggested an SBI was a hosptial admission PLUS one of the following. Before getting to the following…
How can a hospital admission be neccessary in a defnition of SBI? The kids that got admitted got admitted because someone thought they were sick enough – the reasons why (which are likely many) are not recorded and as a result it becomes useless in trying to “derive” a rule.
Anyhow.
So you had to be admitted PLUS some of the usual sensible things like pneumonia or pus or something like that, but they also included CRP>120 or WCC>20. So if you got admitted to hospital with a CRP>120 you apparently had an SBI. This is qute frankly nonsense. How can you have a definiton of Serious Bacterial Illnes that needs no reference to bacteria!!! You could be counted as SBI in this study if you had Still’s disease…
Sorry for getting all high-pitched and exasperated here but this stuff is really important. No matter what you do with your logisitic regression after this, you’re not gonna be able to answer the question you started with.
They recruited 2000 kids to this study. Wow that’s lots of kids surely they’ll find lots of cool things?
Unfortunately not.
Only 74 (or 3.8%) of the kids had SBI (by their definition) and remember that their definition will tend to oversetimate the SBI.
With an event rate this low it’s hard to say anything meaningful in terms of useful identifying features. Not that that stops them doing just that.
In terms of the 74 SBI kids, most had pneumonia – that’s pretty much expected. What is a little bit odd is the low rate of UTI’s. In lots of these kiddy sepsis studies UTI is way up the list and makes up lots of their numbers (and remember UTI, even a sick kid with a UTI isn’t the same as pneumonia or meningococcal sepsis) yet here (with a generous definition) it wasn’t.
As mentioned above, they calculate sens/spec and AUC and all kinds of numbers that are “accurate” but not in the slightest bit useful.
One bit is worth a quote
Apart from tachypnoea (sensitivity 71.6%), the sensi- tivity of most clinical signs was poor.
Does this mean they think a sensitivity of 71.6% is actually good?
The come up with a “rule” and I’ll spare you the details but guess what? A sick kid looks just like you’d expect a sick kid to look like.
We could do with putting our energy into teaching ourselves how to spot a sick kid with the much derided “clinical judgement”.
This isn’t much use to us.
PS by their numbers and definitions, if i just sent home all 2000 of those kids without doing a thing i would have been right 96.2% of the time. Worth noting.