Every Model Trained Before 2020 Is Now Wrong

Somewhere around March, a very large number of machine learning systems stopped working, all at once, all over the world.

Demand forecasts predicted normal Tuesdays. Fraud models flagged legitimate transactions as suspicious because nobody had ever bought that much of anything online before. Credit models scored applicants against patterns from an economy that no longer existed. Staffing models recommended rotas for footfall that didn’t arrive.

None of these systems failed. Every one of them worked exactly as designed. That’s the part worth understanding.

What Actually Happened

Every predictive model rests on one assumption, usually unstated: the future will resemble the past it was trained on.

That’s not a flaw. It’s the definition of the exercise. There’s no way to learn from history without assuming history is relevant.

In March that assumption became false for essentially every consumer-facing business simultaneously. The technical term is distribution shift, and it’s normally something that happens gradually, over quarters, in one segment at a time. This year it happened everywhere, overnight.

“The models didn’t break. Reality moved, and the models kept faithfully describing where it used to be.” — Sameer Gupta

The Failure Was Silent, and That’s the Real Story

Here’s what I’d want a board to take from this year.

Almost nobody found out from their monitoring. They found out from a human noticing something was wrong, days or weeks later, usually because a number in a report looked strange or a customer complained.

Because monitoring, where it existed at all, was watching the wrong things. It watched whether the service was up, whether the job completed, whether latency was acceptable. All green. Meanwhile the model was producing confident, well-formatted, completely wrong output at full speed.

I wrote last year about the six things missing between a model and production, and monitoring was one of them. This year supplied the demonstration.

What Monitoring Should Have Caught

Three layers, and most organisations have none of them:

Input distribution. Are the features arriving today distributed like the features the model trained on? Transaction volumes, basket sizes, times of day, channel mix. A sharp change here is your earliest warning and it arrives before any outcome data.
Output distribution. Is the model’s own behaviour changing? If your fraud model’s flag rate triples in a week, something is wrong regardless of whether you know which cases were genuinely fraudulent yet.
Actual accuracy, when outcomes arrive. This lags, sometimes by months for things like credit default. But it’s the ground truth and it needs to be tracked against the model’s predictions permanently.

The first two require no labels and no waiting. They’re cheap. Almost nobody does them.

The Harder Question: What Do You Retrain On?

This is where the year got genuinely difficult, and I don’t think there’s a clean answer.

The obvious response is to retrain on recent data. But recent data is anomalous. A demand model trained on the spring is a model that thinks a national lockdown is normal, and it will be exactly as wrong when things normalise, in the opposite direction.

The approaches I’ve seen, none of them satisfying:

Shorten the training window and retrain more often. Responsive, and noisy, and it never learns anything about seasonality.
Weight recent data more heavily without discarding history. Sensible, and now you have a decay parameter that nobody can justify.
Add explicit features for the disruption. Honest, and it requires you to know when the disruption ends.
Fall back to simpler models during instability. Unfashionable and frequently the right call, because a simple model degrades predictably while a complex one degrades in ways nobody can anticipate.
Widen the human review band. Accept that the model is less trustworthy and route more cases to people. Expensive, and it’s the only one that reliably works.

“The uncomfortable answer this year was that the correct amount of automation went down for a while, and organisations that could dial it back coped better than ones that couldn’t.” — Sameer Gupta

What I’d Institutionalise Now

While this is fresh and the budget conversation is still easy:

Every production model gets input and output monitoring. Not eventually. It’s a week of work per model and it’s the highest-return engineering in this entire field.
Every model has a documented fallback. What runs if we turn this off? If the answer is “nothing, the process stops,” you have a resilience problem that has nothing to do with machine learning.
Every model has a stated shelf life and a scheduled review, even when nothing appears wrong.
Somebody owns the question “is this still valid.” Not as part of a job. As the job, for a defined portfolio of models.
Write down what happened this year, model by model. Which broke, how you found out, how long it took, what it cost. That document is the business case for all of the above, and its persuasive power decays fast once things feel normal again.

Final Thoughts

The lesson of this year isn’t that machine learning is fragile. It’s that we deployed a lot of systems that assume a stable world, and never built the instrumentation to tell us when that assumption stopped holding.

The pandemic is an unusually clean example because it hit everything at once and the cause was obvious. The ordinary version is worse: one model, one segment, drifting quietly over eight months, with nobody watching, and no dramatic event to prompt anyone to look.

That’s happening right now, somewhere in your estate. This year just made it visible for a moment.