problems with the IHME virus model

Here's the link to which I'm reacting: https://arxiv.org/abs/2004.04734

 

 

I didn't bother to read the entire paper, but the link is above if you're so inclined.  The abstract alone answers a couple of questions:

What is the confidence interval for the IHME model?  Turns out it's 95 percent, which is interesting and about what I expected.

How good is the model? Not very, even with all the updates.  Here's the damning sentence from the abstract - "...the true number of next day deaths fell outside the IHME prediction intervals as much as 70% of the time, in comparison to the expected value of 5%."

So if the stated confidence interval is 95 percent, even with a huge +/- margin, the results should fall within those margins about 95 percent of the time.  Instead, the actual results are outside the generous margins 70 percent of the time.  That translates to a 30 percent confidence interval, which is absolutely ridiculous.  Standard statistical work is usually done at either 99, 97, 95 or in really uncertain cases, 90 percent confidence intervals.  And work at 90 percent CI is usually pretty speculative.  30 percent is unheard of.

I'll admit that the model looked good five or six weeks ago because it was "scientific" lending it credence in the swirling uncertainty we were all operating under.  However, tracking actual deaths in Georgia and comparing to the model, even as it's updated time and time again, it just doesn't pan out long term.  Now somebody presumably smart (since it's Cornell University) has done the math and found that the stated accuracy is nowhere near the actual accuracy. 

This is a damning indictment of depending too much on one model, failing to take in a variety of opinions and predictions and then using some judgment to make a decision.  It looks like our national leaders especially, just grabbed the model and used it to guide ALL decisions.  Worse, it looks like they're still using it, even after it's been proven inaccurate.

Part of the problem as I see it is the extremely variable nature of the data.  New York City is not Atlanta.  Atlanta is not Blairsville.  Blairsville is not Dougherty County.  Wildly different rates happening should be compared and contrasted, not lumped together and averaged out to some impossible one size fits all response.

And by the way, Georgia's death rate is still climbing, not decreasing.  That's the real data speaking, not Kemp or his advisors, and not a prediction.