AIAn Afternoon at Kepler Labs
I’ve been writing around the edges of machine learning for three years now. Reading papers, working out what the terms mean, forming opinions about what it means for businesses.
Last month I spent an afternoon with the team at Kepler Labs, where my friend Ajay Vishnu works, watching people actually do it. I’ve been thinking about it ever since, and the honest summary is that reading about this and watching it happen are not the same activity at all.
What They Were Working On
They’re teaching a computer to recognise hand-drawn doodles.
I want to sit with how odd that sounds for a moment, because my first reaction was that it seemed like a toy. A person draws a rough sketch of something, a house, a bicycle, a fish, and the machine says what it is.
It is not a toy, and here’s why. A photograph of a bicycle contains enormous amounts of information: texture, shading, background, colour, the specific bicycle. A five-second doodle of a bicycle contains almost none of that. Two circles and some lines. Whatever makes it recognisable as a bicycle is the abstract structure alone, stripped of everything else.
If a machine can do that, it’s not matching pixels. It’s got hold of something closer to the concept.
“A photograph tells you what a thing looked like. A doodle tells you what somebody thinks a thing essentially is. Teaching a machine the second one is a much stranger problem.” — Sameer Gupta
Watching the Work
Here is what I had not appreciated from reading papers: most of it is waiting, and the rest is judgement under uncertainty.
The training run had been going for two days on a pair of graphics cards, and it would run for a few more. Someone had a chart on a second monitor showing the loss curve, the number that’s supposed to go down as the network gets better.
It had been flat for about six hours.
The conversation around that flat line was the most educational part of the day. Was the learning rate wrong? Had it found a local minimum it couldn’t escape? Was there something wrong with the data augmentation? Was it actually fine and just slow? Four experienced people, genuinely uncertain, reasoning from experience rather than from theory, proposing experiments that would each cost another day.
I had assumed, from the papers, that people knew what they were doing and the results followed. The papers report the run that worked. Nobody publishes the fortnight of runs that didn’t, or the argument about whether to kill one at hour thirty.
The Confusion Matrix
The thing that actually got me was a table Ajay pulled up near the end of the afternoon.
A confusion matrix shows, for every category, which other categories the model mistakes it for. Most of it was unremarkable. And then there were the cells where the model was consistently, systematically wrong, and every single one of those mistakes made sense.
I don’t want to overclaim here, and I’m aware I’m describing a statistical artefact in emotional terms. But the errors weren’t random. They were the mistakes a distracted person would make. Things that genuinely do look alike when you reduce them to a few strokes got confused with each other. Things that are conceptually related but visually different did not.
The model had learned something about shape that lined up, imperfectly, with how people categorise things. Nobody had told it to. It emerged from being shown enough examples.
Ajay’s reaction to my reaction was the reaction of someone who sees this every day: yes, that’s normal, and here’s the case where it goes badly wrong. Which was also useful.
What Changed for Me
I’ve been comfortable in a particular role for a while: the person who reads the research and translates it into what it means commercially. That’s a legitimate job and I intend to keep doing it.
But I left that afternoon with an uncomfortable realisation. I was writing about a practice I had never performed. I could explain what a convolutional layer does. I had never watched one fail to converge and had to decide what to do about it. Those are different kinds of knowing, and the second one is where the actual difficulty lives.
“I could describe the technique accurately and I had no idea what it feels like when it doesn’t work. That gap is where all the real decisions are.” — Sameer Gupta
Everything expensive about a machine learning project happens in that gap. How long to persist with an approach. Whether a disappointing result means the idea is wrong or the setup is. Whether the data is the problem. I had been writing about the economics of these projects without understanding the activity that consumes the budget.
So I’m Going to Learn It Properly
Not to become a researcher. That ship has sailed and I have no ambitions there. But I want to be able to train a model, badly, myself, and feel what goes wrong.
The plan, such as it is:
- Start with the fundamentals rather than the frameworks. I’ve been told repeatedly that people who start with a library and no statistics end up unable to tell a real result from an artefact. That warning has come from too many directions to ignore.
- Do the mathematics. Linear algebra and probability properly, not skimmed. I’ve been avoiding this and the avoidance is showing.
- Build things that fail. Small projects on public datasets where I can see the whole pipeline and make every mistake personally.
- Find structure. Left to my own devices I’ll read interesting papers and skip the tedious foundations, which is exactly the wrong order.
Final Thoughts
My thanks to Ajay and the team for a genuinely generous afternoon, and for answering questions that must have seemed extremely basic with more patience than they were owed.
I went in expecting to write something about applications and business models. I came out with a reading list and a slightly embarrassed sense that I’d been commenting on a craft from a comfortable distance.
The next few years of this blog are probably going to be a bit more technical. Consider yourself warned.