problems with the IHME virus model

Here's the link to which I'm reacting: https://arxiv.org/abs/2004.04734

 

 

I didn't bother to read the entire paper, but the link is above if you're so inclined.  The abstract alone answers a couple of questions:

What is the confidence interval for the IHME model?  Turns out it's 95 percent, which is interesting and about what I expected.

How good is the model? Not very, even with all the updates.  Here's the damning sentence from the abstract - "...the true number of next day deaths fell outside the IHME prediction intervals as much as 70% of the time, in comparison to the expected value of 5%."

So if the stated confidence interval is 95 percent, even with a huge +/- margin, the results should fall within those margins about 95 percent of the time.  Instead, the actual results are outside the generous margins 70 percent of the time.  That translates to a 30 percent confidence interval, which is absolutely ridiculous.  Standard statistical work is usually done at either 99, 97, 95 or in really uncertain cases, 90 percent confidence intervals.  And work at 90 percent CI is usually pretty speculative.  30 percent is unheard of.

I'll admit that the model looked good five or six weeks ago because it was "scientific" lending it credence in the swirling uncertainty we were all operating under.  However, tracking actual deaths in Georgia and comparing to the model, even as it's updated time and time again, it just doesn't pan out long term.  Now somebody presumably smart (since it's Cornell University) has done the math and found that the stated accuracy is nowhere near the actual accuracy. 

This is a damning indictment of depending too much on one model, failing to take in a variety of opinions and predictions and then using some judgment to make a decision.  It looks like our national leaders especially, just grabbed the model and used it to guide ALL decisions.  Worse, it looks like they're still using it, even after it's been proven inaccurate.

Part of the problem as I see it is the extremely variable nature of the data.  New York City is not Atlanta.  Atlanta is not Blairsville.  Blairsville is not Dougherty County.  Wildly different rates happening should be compared and contrasted, not lumped together and averaged out to some impossible one size fits all response.

And by the way, Georgia's death rate is still climbing, not decreasing.  That's the real data speaking, not Kemp or his advisors, and not a prediction. 

It's the taxes, stupid

Almost no one in most of the online or broadcast media seems to be talking about this, so I'll just pose a few questions.

Are our governments about to go broke?

One effect, not too far in the future, of the economic shutdown is the drying up of tax revenues for all levels of government from the federal system on down.  With crazy high unemployment, income taxes are going to fall.  The collapse of the tourism industry means that hotel/motel taxes will not be collected.  Locally those pay for all sorts of things.  Gas taxes aren't being collected in the amounts they normally are since people aren't driving.  With the cutback in spending, sales taxes collections will be lower; that means every damned SPLOST in the nation will bring in less money than thought.

There are other effects.  The economic downturn we forced upon ourselves means that property values may stagnate or drop.  That means that every local or state government that depends on property taxes will see less coming in.  Services paid for by those taxes will be cut.  School systems will be hard hit, at least here in Georgia where they're paid for by property taxes.

What's worse is that for the last few years property values have been increasing, and so property taxes have gone up.  Those yearly increases in revenue were budgeted in, the school system assuming that each year there would be more money to spend.  That's no longer the case.  Expect budget cuts, wage freezes, maybe even layoffs in the school systems of Georgia.

Government services at all levels, city, county, state, national, will be starving.  Either taxes will go up or services cut back, probably both.  Get ready because it's coming, and we've done it to ourselves by overreacting to the virus with draconian economic damage that hasn't stopped people dying.

Risk assessment part 5

Picking up from Part 4, here's a few more selected emails I've sent to friends regarding the pandemic.

April 13, 2020

https://www.outsideonline.com/2411617/surfers-climbers-break-social-distancing-rules

As a counterpoint of sorts, read the article linked above.  I agree that a total economic shutdown is a bad idea, but I do support social distancing, not because of the risk to myself, but because of the risk to others more vulnerable than me.  I don't think the author of the American Thinker article really considers that.  This article from Outside Online is a thoughtful response.

=========

April 14, 2020

As for legit data, I'll go back to what I said weeks ago: the only thing that matters is deaths.  Yes, I know there's some controversy over dying with versus dying of, but it's still the closest thing to impartial that we have.  In Georgia it looks like the old and infirm are hit hard and sadly we're finding out that a lot, and I mean an awful lot, of nursing homes have this stuff spreading in them.  Just around here we have cases in Greensboro and Union Point at three different facilities.  Clarke County's deaths are largely old people in nursing homes.

 

==========

 

April 17, 2020

 

I think I basically agree with what's outlined in this short article, but I'm leery of the numbers since they're mostly relying on China data.  Regardless, the truth is that the virus is not going to disappear anytime soon, so how do we live with it in the two or three years it'll take to develop a vaccine?  We'll have to abandon the current lockdown, either officially or in large incidents of civil disobedience.  I'd rather do it officially to avoid hurting vulnerable populations.

Link is below.

Following the herd in COVID-19 exit strategies - Research - J.P. Morgan Markets

 

=========

 

April 18, 2020

 

Exactly what I see in the data and exactly what I saw them do today - drop it from 3.6k deaths in Georgia to 1.7k deaths.  Story linked below, but the parts that match what I've been saying: "...many locations are now predicted to have longer peaks and are taking longer to move down the epidemic curve to zero death..."

This is what I have been saying the data for Georgia shows.  What they say is a longer peak is what I see as a plateau, same thing.  And I said that the downward curve would be different from the upward curve.  Now they're saying it too, but I said it first.

The fact that they've changed it so much means their model is pretty poor as a source for policy decisions, and that I'm right when I say look at the deaths and only the deaths.  Predictions have been really shit, while me looking at the data day to day has presented a clearer picture.  It's scary how many smart people are only now coming around to what I've been seeing for a while.  Maybe they're not so smart.

========

 

April 20, 2020

 

From a paper I can barely understand that's highly critical of all these models and estimates there's this.  I've underlined the parts that bolster my argument that what I'm looking at with my little bit of real world practical data analysis from an industrial setting is probably a good way to deal with all this noisy and incomplete data.  I've left out major chunks of mostly math to focus on the implications.

“...Estimating R from noisy real-world data when you don’t know the underlying model is fundamentally difficult...often be badly (and confidently) wrong about that because it fails to account for how the confirmed count data it’s based on is noisy enough to be mostly garbage. (Many serious modelers have given up on case counts and just model death counts.)

...Digging into the data and the math, you can see that a few days of falling case counts will make the system confident of a very low R, and a few days of rising counts will make it confident of a very high one, but we know from other sources that both can and do happen due to changes in test and test processing availability. (There are additional serious methodological problems with rt.live, but trying to nowcast R from observed case counts is already garbage-in so will be garbage-out.)

 

...it’s just figuring out whether what you’re observing is growing or shrinking. Many folks would actually be better off not trying to forecast R and just looking carefully at whether they believe the thing they’re observing is growing or shrinking and how quickly."

 

And since that's what I've been doing, I've probably got as good a handle on the trends as anybody else, and better than some who don't know what they don't know.  I'm just concentrating on the death counts and not trying to estimate anything at all about the underlying mechanics (since I don't know how), and looking at the shape of the data, techniques I've used in industrial settings to make decisions in the absence of complete data.

 

========

 

April 21, 2020

 

That was an excellent summary.  Points two and three I especially agree with wholeheartedly: what happens in the second wave?  And what are we going to do about it?

The virus is here.  It's not going away.  The ball is in our court; what's our next move?  I hope we eventually stumble into doing what we probably should have done from the start: go all in on protecting our vulnerable folks better than we have so far, while everybody else takes reasonable precautions and gets on with their lives.

 

======

 

April 23, 2020

 

Not at all surprised by this, other than the fact that it was nearly universal.  This says our quarantine and stay home measures should be a lot more tightly focused on people who are obese, diabetic, elderly or otherwise prone to sickness.  The rest of us should stay away from those people we can hurt by infecting them, but if we're otherwise healthy, we can probably survive the virus. 

This gets back to that variolation proposal where we incentivize younger and stronger populations to catch the virus under the assumption that each person who does and survives it protects something like 3 other people.  So as Georgia opens up, we'll keep in mind who we're coming in contact with and continue to limit exposure to the elderly and sickly.  Hopefully that will work to knock the virus down by denying it hosts eventually.

========

 

 

shutting down the shutdown

Yesterday Georgia Governor Brian Kemp said the state is ready to begin phase one of opening the economy, in part because deaths from the virus are beginning to decline.  While I support wholeheartedly getting people back to work, I firmly disagree that Georgia's death rate is tapering off.  It is not declining; it is increasing.

Here's a chart I made from the data posted daily on the website of the Georgia Department of Public Health, the agency tracking cases and deaths from the Wuhan virus.  The daily deaths are the yellow line.  The 5-day median is the red line, recalculated every 5 days.  Every 5 days, it's gone up so far.  It has not gone down.  In other words, every five days more people die than the previous five days.  I'm using median as the best measure of central tendency given the huge variability in the data. 

While I am all for opening up the state, I have no frakking clue what they're looking at to say that any rate is declining when their own numbers, released daily, are telling me the death rate is still climbing.

 

I hope that while the rest of us get back to work, we take extraordinary measures to protect the most vulnerable Georgians.  Up to now it looks to me like we've not done enough to protect our nursing homes, while doing too much to lock down everybody else.  Maybe we can change some of that going forward.

and a repost from woodswomyn

In a longer post about hiking there's this:

"I love hiking and I love a challenge. But what I am addicted to is wearing myself down to these raw moments of weakness so I can prove to myself that I come back.⁣⁣"

That's a damned good bit of writing and so relatable if you're a hiker or runner who likes to challenge yourself.  More from woodswomyn over on her Instagram page: https://www.instagram.com/woodswomyn/

Reposting from John F. Harris

From a post by John F. Harris comes this succinct bit of writing -

The pandemic response arguably could represent liberal values at their best. Government, guided by scientific expertise, protected vulnerable people through a noble exercise of shared sacrifice for shared benefit.

The pandemic response arguably could represent a caricature of what critics disdain about liberalism. Government, responding in a panicky way to headlines and hysteria, ran roughshod over individual freedom and the private sector, a problem whose only remedy was even more remorseless expansion of government. 

----------

That's good stuff, and there's more at the link below:

https://www.politico.com/news/magazine/2020/04/16/the-shutdown-backlash-is-coming-soonwith-a-vengeance-189809

What's a proper response?

In response to a slippery slope argument against pandemic laws...

Ironically (or hypocritically), I'd argue that it's up to each person to voluntarily decide whether their actions risk harming others.  I'm all for guidance from respected medical/economic voices on precautions to take.  I disagree with coercion in the form of laws that curtail my Constitutional liberties.  I'll give Kemp some grudging respect for the way he's (mostly) tried to use persuasion rather than force, especially with church services.  I don't like to see even the current "rules" in place very long, lest we get used to them and forget we have a right to move about freely.

One problem with my approach of voluntary assumption of temporary limitations, though, is that it relies on having good information on which to base one's actions.  That is absolutely NOT the case today, mostly because our news media is so horribly incompetent.  If we all had more or less that same factual information, we could all more or less make good decisions for ourselves and for our immediate communities, friends, and families.  We'd probably come to similar conclusions and there wouldn't be any necessity for laws to coerce us into anything.

But we don't have good information at all.  We get offered opinions, guesses, models, poorly written, poorly researched and blatantly slanted stories designed to generate ad revenue while whipping us into a frenzy.  You and I know that the press has, for the last three years, been exposed as liars of the worst type, unable and unwilling to report factually about a president they viscerally hate.  Instead, they're willing to write absolute untruths because they believe they're right and we are wrong.  Do we think they've magically abandoned that to suddenly report THIS story the right way?  Not a chance.

So if we're in a situation where most of the information we have is suspect, what do we do?  I don't really know, other than to fall back on core principles of behavior and try to maintain a sense of humor (research says people with the ability to see the humor in otherwise serious situations like a pandemic tend to live longer).  I guess in my case, I am sitting zazen every day to maintain my sanity and my humor, and trying to remember that I took a vow that includes what's called the three pure precepts:

1 - Don't do harm (or don't be a jerk), 2 - Do good (whatever that may be), 3 - Help others do good (I'm not real good at that last one).

As one response I believe satire would be appropriate.  If you've seen my Instagram page, you've seen my public face and public response, which I hope pokes a little fun at things from time to time.  Maybe that's a poor or insufficient response to our current situation, but it's the best I can do today.  I reserve the right to be wrong and start tossing bombs tomorrow.

How the predictions panned out

First, my own predictions:

On March 29, 2020 I predicted 300 deaths (+/- 50) in Georgia by Easter.  On April 4, 2020 I updated that to 400 +/- 50.  The actual death toll in Georgia as of Easter midday is 433.  My original estimate was based on the data we all had at that time.  I simply used a linear regression model and then a fudge factor of four standard deviations.  The death curve, unfortunately, was not linear and so I underestimated by about 24 percent, though my updated estimate was within my margin of error.

How did others do?

Around that same time the Georgia Municipal Association (GMA) gave a presentation to every Mayor in the state.  Their model predicted a range of values for three months from the date of the presentation, which was, I believe, March 25, 2020.  The low figure if shelter in place was done, was 6,000 deaths by the end of June.  If no action was taken, the figure was 27,000 deaths.  A little less than a month into their timeline and we are still under 500 deaths in Georgia.

The IHME model, widely used nationally, has changed several times as more data becomes available, but on March 29th, their predicted death toll for Georgia for Easter was 558 deaths.  Their prediction as of today, for today, is 552 deaths.  They are about 22 percent over the actual number.

So it appears that I under estimated the death toll by about the same amount that IHME over estimated it.  GMA's numbers can't really be tested yet, but look way over estimated so far.

My April 4 estimate was much, much closer than the IHME's for the same time frame.  They missed by about 80; I missed by 33, though I gave myself +/- 50 as a range.  By contrast, the IHME model has a +/- of nearly 200!  My estimate is both more accurate and has much less wide of a range.

I'm not smarter than IHME and I have access to less data than they do, so how did I come in with the better estimate for Georgia?  Skepticism, rational thought, paying no attention to news hysteria, and admittedly, some luck.

My basis for decision making from the first has been to follow the deaths.  Forget the number of cases; look only at the deaths.  Forget modeling, because complex iterative models have unexamined assumptions that can lead you astray.  Look at real world data, graph it to see the shape, and use rational thought, not panic, to make decisions.

In the end, I will admit to being somewhat too optimistic with my initial prediction, but considering the fact that I'm just a guy who likes numbers, not a statistician, not an epidemiologist, I think my analysis is at least as good as some of the so called "experts."

risk assessment part 4

More posts I've made regarding the Wuhan virus:

April 6, 2020 - While it's probably not feasible for a lot of reasons (cost, lack of qualified medical personnel) here's a response that I think I might favor. It could be summed up as, infect as many young and healthy volunteers as we can, while isolating the most vulnerable, and then let herd immunity do the rest. The volunteers would be rewarded (paid) for getting infected under the assumption that each recovered person increases herd immunity in the community and is thus saving lives by risking their own. I like the fact that some people are at least thinking about these things as an antidote to the usual media scrum.

April 7, 2020 - I'm attaching a spreadsheet with some of the Georgia stats on deaths.  It's updated as of noon today (4/7).  The total deaths are over 300 right now and will be updated again tonite at 7pm.  The rate of dying is accelerating now, after dropping a bit this past weekend.  My original prediction of 300 by 4/12 will be low, obviously...The age range is disproportionately older people, some quite old.  On the other hand, I would imagine that under "normal" circumstances, that particular chart would look much the same, i.e. skewed right.

April 7, 2020 - Unfortunately the numbers of people dying look to be accelerating.  As an aside, Greene County now has 23 confirmed cases and 1 death.  Union County has only 4 cases and nobody dead.  They're continuing to die in Dougherty County, leading the state with 56 dead.  Next closest is Fulton with 39.  I'm reluctantly beginning to believe we haven't reached the peak yet.

April 9, 2020 - That IHME page is one I've been tracking, and yes, it's changed pretty significantly for Georgia, which is the only part of it I pay attention to.  In their defense, they're continuing to update their predictions as conditions change, rather than slavishly adhere to their original numbers.

What I saw was that they were actually LOWER than the actual numbers of deaths early in April, not a whole lot, but about 3-5 percent under what the real deaths were day to day.  Longer term they were projecting something like total 3. 5K deaths in Georgia by June, with a shortage of hospital beds and so forth.  The general shape of their data, though, and the ramping up, was tracking pretty well unfortunately.

After Kemp came out with the statewide shelter in place proclamation, their projections changed.  Rather than a shortage of hospital beds, there's now enough that no shortage is projected at all.  The number of ICU beds is still showing a shortage, but a smaller one than before.  And the total projected deaths is now down.  I think the peak date may have moved a day or so and the peak is now smaller, based on the newer data and changing conditions. 

So it appears that the model is being refined as more and more data comes in, which makes sense. They're also changing it based on factors like whether a shelter in place order is in effect or not.  I suppose they take into account the number of hospital beds available, too. Their projected deaths per day is now slightly higher than actual numbers, I think.  But of several models that have been thrown at us, the IHME one seems to be the closest to reality for Georgia.  The percent hospitalized seems important as to whether we get a runaway pandemic or not, and Georgia has remarkably stayed under 35 percent the whole time.

April 9, 2020 - ...let me flip a switch and point you to a couple of news stories out of New York that support two things we've talked about:

This first one says that new data indicates the virus has been extant in the city a lot longer than first thought, and that it came from Europe.  That matches perfectly with the theory that after the Milan fashion show, all the jet setters left Milan and headed all over Europe and to America where they landed mostly in New York.  The east coast mostly got the virus from European travelers, not Chinese since we had already cut off China (Yay, Trump).  I'd be interested in seeing the same type of research done on the virus on the west coast.  Dollars to donuts it came from China, maybe as early as November.

 

April 9, 2020 - That's a good argument for shelter in place, at least if you're part of the high risk population. Unfortunately, there was a huge exodus from New York (and Atlanta) some weeks back, which spread the virus around.  We need social distancing measures, but we also need to get the country back to work full time.  I don't know how to juggle those two things, but I still like the variolation method of purposely infecting the young and strong to build up herd immunity.  I'd just hate to try that and have it make things worse.

I've been toying with the theoretical idea of purposely infecting myself.  Would I do it if it were offered?  I dunno.  I'd hate asking younger, stronger people to do something I'm not willing to do.  On the other hand, am I a good candidate to survive it if I get it?  I don't know that answer either.  There are just too many unknowns still to make hard choices.

 

April 10, 2020 - The point of my post was that the problem is not the model (which is actually pretty good by now); the problem is the reaction to the model. 

I guess what I'm saying is that I'd like decision-making to be based less on predictions of the future and more on analysis of the past.  By this point, we are collecting enough data on how this virus behaves in the wild to analyze what is really happening, not what is projected to happen down the road.  I'd rather decisions be based on what we know and have seen the virus do. 

For one limited example, we've now seen over and over that if this gets loose in a nursing home it's horrible and people die who probably shouldn't.  What's the proper response to that?  Me staying home?  Bars closing?  Probably not.  Isolate the nursing homes with super stringent hygiene and very limited in and out visitation?  Probably so. 

Somewhere between total societal lockdown and let-her-rip there's a middle ground that acknowledges we all have some social responsibility not to hurt the vulnerable among us by insisting on our own free reign, while at the same time the government backs off on trampling my Constitutional rights in the name of safety.

 

 

not horseshit but it's horseshit

From my email response to a friend:

I'm pretty sure you don't mean that the virus itself is horseshit, but that the response to it may be overblown in places?  There are certainly spots in the country where it's killing huge numbers of folks, like NYC and causing a problem with overloading the available bed space, like down in Dougherty County around Albany here in Georgia.  Savannah Court Nursing Home here in Greene County definitely has multiple cases, and some of them may die.

With that caveat, what is horseshit is the media treatment of this pandemic.  Short on facts, long on speculation and doomsday scenarios, with a liberal sprinkling of politics, the media has almost completely failed to keep us informed with anything approaching what we need to deal with it.  I'd include the CDC and the FDA among those agencies who have failed us as well.  The CDC was late and has been inconsistent.  The FDA dragged its bureaucratic feet on approving a test for the virus and we are still paying that price.

What I saw happen was that first, the media blew this thing up because they were certain it would hurt Trump's reelection chances.  However, the fact that the virus was a real threat and people really started dying tripped up that narrative thread.  By the time the media stories stampeded people into a panic, it was too late.  They had lost control of the story and it no longer served their purposes.  At a time when the strategy might have been more nuanced with an emphasis on isolating vulnerable old people and everyone else just being careful, the panic had already set in.

And so the response became "shut it down," an overly blunt instrument.  The truth is that there is a lot of uncertainty about this thing and what it can do.  But nobody wants to honestly report THAT because it serves no agenda.  The agenda driven reporting of this (and damn near every other story for that matter) does a terrible disservice to our country.  It means that a lot of people who don't trust the media don't believe the virus can harm them.  It also means that other people who swallow media whole believe we're living through the last days on planet earth.  Neither of those things is correct.

This is a bad thing, a cause for concern, but not, I think, a cause for panic and overreaction.  Measured, thoughtful reporting would have helped us make better choices, but it's too late for that now.  We have to live within what's happening now, deal with the reality rather than wishing it were different.  I still like the strategy of that link I sent you the other day of suppress, then test, trace and isolate.  Unfortunately, it's too late for that, though we might still get a variation of it.

I don't see any way that this virus doesn't eventually pass through the entire population.  That's a reality that I also don't see reported very much.  Given that that's gonna happen, what's the best way to deal with it?  That's what I want to know.  In other words, what happens AFTER we flatten the curve, as the saying goes?  The virus doesn't just give up and go home, never to be seen again.  I see it coming back this November and that makes me worried for my parents and others who manage to not be exposed during this first wave.  What protects them next time?  Shutting down the country again is not an option (and shouldn't have been an option this time, IMO).

updated prediction

Originally on March 29th I projected 300 deaths in Georgia by Easter from the Wuhan virus.  If I were making that prediction today, based on data that's been released since the date of my original prediction, I'd make it closer to 400 deaths.  The +/- 50 I'd keep the same.

Thus my updated prediction based on the latest data is 400 deaths in Georgia by Easter, +/- 50.

What is somewhat troubling is that the official prediction for April 2 was lower than the actual deaths for that date, and the same for April 3.  Their model has many, many more people dying than I am predicting.  So far they are actually lower than the real death toll.  I hope they are wrong, else we are in for a horrific few weeks this month.

we're shut down

Yesterday Kemp declared a stay at home order taking effect from April 3 through April 9th.  Depressing and scary.

What bothers me more than the virus itself are two things:

1) Our loss of Constitutional rights and how unthinkingly most people are allowing it without even a whimper of protest.  We are becoming a true nation of sheep.

2) The economic carnage, which will linger like an oil spill over the pristine beauty that was our hard charging nation. 

Both of these together will give rise to a clamor for more government solutions, when we just need the government to get the fuck out of our way.  As the great depression gave us the beginnings of the welfare state, aided with Johnson's war on poverty in the 1960s, this mostly media whipped crisis will see the true birth of socialism as the generation of 20-somethings, scared by this experience and poorly educated, begin to vote for the nanny state to take care of them.

There's no shortage of officious asses willing to enforce the current Unconstitutional orders.  What bothers me is the clear joy these jack-booted nancy-boys take in ordering (with bullhorns!) people to "Disperse ye rebels."