AI/Biz

COVID Broke Every Model You Own. None of Them Failed.

In March, COVID-19 shut down most of the world economy in about a fortnight. Shops closed, borders closed, offices emptied, and hundreds of millions of people changed how they bought almost everything, all in the same week.

Let me say the obvious thing first, because I think it gets lost in write-ups like this one. This was not primarily a machine learning event. It was a public health catastrophe and the sharpest economic contraction in living memory. Output fell by a fifth in a single quarter here. Entire sectors — aviation, hospitality, live events — went to roughly zero revenue overnight, and a great many people lost work they are not going to get back.

Machine learning sits inside that economy. When the economy moved, everything built on top of it moved too, and that included a very large number of models.

Demand forecasts predicted normal Tuesdays. Fraud models flagged legitimate transactions as suspicious because nobody had ever bought that much of anything online before. Credit models scored applicants against an economy that had ceased to exist. Staffing models recommended rotas for footfall that never arrived.

Here is the part I want to be blunt about: none of those systems failed. Every one of them worked exactly as designed, right through the whole thing, at full confidence. That is not a comforting sentence and it is not meant to be.

What Actually Happened

Every predictive model rests on one assumption, usually unstated: the future will resemble the past it was trained on.

That is not a flaw. It is the definition of the exercise. There is no way to learn from history without assuming history is relevant.

In March that assumption became false for essentially every consumer-facing business on earth, simultaneously.

If you are new to this: a model of this kind is not a set of rules somebody wrote down. It is a pattern extracted from historical data — here is what people bought, here is who repaid, here is what a fraudulent transaction looked like — which is then used to make guesses about cases it has not seen. The catch is that it has no idea what any of it means. It cannot tell that a lockdown has been announced. It only knows that the numbers arriving today should look roughly like the numbers it learned from, and when they stop looking like that it goes on answering anyway, in the same confident tone, with no way of telling you it has left familiar ground. The industry term for that gap opening up is distribution shift.

Normally distribution shift is gradual. It happens over quarters, in one segment at a time, and you have the luxury of catching it in a review. This year it happened everywhere at once, overnight, in every industry simultaneously.

“The models didn’t break. Reality moved, and the models kept faithfully describing where it used to be.” — Sameer Gupta

The Failure Was Silent, and That’s the Real Story

Here’s what I’d want a board to take from this year.

Almost nobody found out from their monitoring. They found out from a human noticing something was wrong, days or weeks later, usually because a number in a report looked strange or a customer complained.

Because monitoring, where it existed at all, was watching the wrong things. It watched whether the service was up, whether the job completed, whether latency was acceptable. All green. Meanwhile the model was producing confident, well-formatted, completely wrong output at full speed.

I wrote last year about the six things missing between a model and production, and monitoring was one of them. This year supplied the demonstration.

What Monitoring Should Have Caught

Three layers, and most organisations have none of them:

  • Input distribution. Are the features arriving today distributed like the features the model trained on? Transaction volumes, basket sizes, times of day, channel mix. A sharp change here is your earliest warning and it arrives before any outcome data.
  • Output distribution. Is the model’s own behaviour changing? If your fraud model’s flag rate triples in a week, something is wrong regardless of whether you know which cases were genuinely fraudulent yet.
  • Actual accuracy, when outcomes arrive. This lags, sometimes by months for things like credit default. But it’s the ground truth and it needs to be tracked against the model’s predictions permanently.

The first two require no labels and no waiting. They’re cheap. Almost nobody does them.

The Harder Question: What Do You Retrain On?

This is where the year got genuinely difficult, and I don’t think there’s a clean answer.

The obvious response is to retrain on recent data. But recent data is anomalous. A demand model trained on the spring is a model that thinks a national lockdown is normal, and it will be exactly as wrong when things normalise, in the opposite direction.

The approaches I’ve seen, none of them satisfying:

  • Shorten the training window and retrain more often. Responsive, and noisy, and it never learns anything about seasonality.
  • Weight recent data more heavily without discarding history. Sensible, and now you have a decay parameter that nobody can justify.
  • Add explicit features for the disruption. Honest, and it requires you to know when the disruption ends.
  • Fall back to simpler models during instability. Unfashionable and frequently the right call, because a simple model degrades predictably while a complex one degrades in ways nobody can anticipate.
  • Widen the human review band. Accept that the model is less trustworthy and route more cases to people. Expensive, and it’s the only one that reliably works.

“The uncomfortable answer this year was that the correct amount of automation went down for a while, and organisations that could dial it back coped better than ones that couldn’t.” — Sameer Gupta

What I’d Institutionalise Now

While this is fresh and the budget conversation is still easy:

  • Every production model gets input and output monitoring. Not eventually. It’s a week of work per model and it’s the highest-return engineering in this entire field.
  • Every model has a documented fallback. What runs if we turn this off? If the answer is “nothing, the process stops,” you have a resilience problem that has nothing to do with machine learning.
  • Every model has a stated shelf life and a scheduled review, even when nothing appears wrong.
  • Somebody owns the question “is this still valid.” Not as part of a job. As the job, for a defined portfolio of models.
  • Write down what happened this year, model by model. Which broke, how you found out, how long it took, what it cost. That document is the business case for all of the above, and its persuasive power decays fast once things feel normal again.

Final Thoughts

The lesson of this year isn’t that machine learning is fragile. It’s that we deployed a lot of systems that assume a stable world, and never built the instrumentation to tell us when that assumption stopped holding.

COVID is an unusually clean example, because it hit everything at once and the cause was on the front page. Nobody had to be persuaded that something had changed. The ordinary version is worse: one model, one segment, drifting quietly over eight months, with nobody watching, and no dramatic event to prompt anyone to look.

That’s happening right now, somewhere in your estate. This year just made it visible for a moment.