AI/Biz

The Data Lake Is Where Data Goes to Drown

There’s a term going around and I’ve now heard it in enough meetings to want to write about it.

The pitch for a data lake goes like this. Traditional warehouses require you to define a schema before you load anything, which is slow and rigid and means you throw away detail you might later want. A lake takes raw data in its native form, keeps everything, and applies structure at the point you read it rather than the point you write it. Schema-on-read. Flexibility.

I want to take this seriously, because there’s a real idea in there. And then I want to explain why I think most implementations of it will fail.

The Legitimate Part

The criticism of the traditional warehouse is fair. Defining a schema up front does force decisions before you know enough to make them, and it does discard information. Anyone who has tried to answer a new question and discovered the relevant field was collapsed into a category three years ago knows the pain.

Keeping raw data and deciding later is genuinely more flexible. That part is not marketing.

What Actually Happens

Here’s the failure mode, and it’s consistent enough that I’d almost call it inevitable.

Schema-on-write is annoying because it forces an argument. Before you can load the customer table, somebody has to decide what a customer is. Is a cancelled account still a customer? Is a household one customer or four? Does the number in this column mean pounds or pence?

Those arguments are tedious. They’re also the actual work. When you defer the schema, you don’t remove the arguments. You just move them to the point where twelve different teams each have their own answer, encoded in their own query, and nobody has compared them.

“Schema-on-read doesn’t mean you decide later. It means everybody decides separately, and you find out at the board meeting when two teams present different revenue numbers.” — Sameer Gupta

The lake fills up. Nobody knows what most of it means. The people who loaded a given feed have moved on. There’s no catalogue, because building a catalogue was the discipline you skipped when you chose the lake in the first place.

At which point you have a swamp, and the term is not mine, it’s already in circulation among people who have lived through this.

The Four Things That Are Actually Missing

I don’t think the lake concept is wrong. I think it’s incomplete in ways the vendors are not emphasising, and each gap is a real project:

  • A catalogue. What datasets exist, where they came from, what the fields mean, when they last updated. Without this, discovery is asking a colleague, which doesn’t scale past about thirty people.
  • Ownership. Every feed needs a named human who is accountable for its quality. Unowned data rots silently, and you learn it rotted when a decision goes wrong.
  • Lineage. When a number looks odd, you need to trace it back through every transformation to the source. Without lineage, debugging a figure takes days and often ends in a shrug.
  • Quality monitoring. Row counts, null rates, distributions, watched over time. A feed that silently starts arriving half-empty is worse than a feed that stops, because nobody notices.

Notice that none of these are storage problems. They’re all governance problems, and the lake pitch is a storage pitch. You are being sold the easy half.

What I’d Do Instead

I’m not against keeping raw data. I’m against keeping raw data as a substitute for deciding what it means.

Keep the raw layer, but treat it as a landing zone, not a product. It’s where things arrive. It is not where anyone should be running business queries.

Curate a small number of datasets properly. Pick the ten that matter, agree what the fields mean, document it, assign an owner, and monitor the quality. Ten well-understood datasets beat four hundred mysterious ones, and the ratio isn’t close.

Make the catalogue a first-class deliverable, funded and staffed, not something to do after the platform lands. It never gets done after.

Charge the cost back. The reason lakes fill with junk is that adding a feed is free to the team that adds it. Make retention visible in somebody’s budget and you’d be amazed how quickly the useless feeds get switched off.

“The question isn’t whether you can store it. Storage was solved. The question is whether anyone will be able to tell what it means in three years, and the honest answer is usually no.” — Sameer Gupta

Final Thoughts

I wrote a few years ago that big data was mostly a storage bill, and I’d say the lake is the same argument wearing better clothes. The technology improved. The organisational discipline it requires did not, and that discipline was always the binding constraint.

If your company is considering one, the question I’d ask in the meeting is not about the platform. It’s: who will own the meaning of each dataset, and what happens to them when they’re wrong?

If nobody has an answer, you’re not building a lake. You’re building a very large place to lose things.