AI/Biz

A Neural Network Just Won ImageNet — and It Wasn't Close

Every year there’s a computer vision competition you’ve probably never heard of. This year a team from Toronto didn’t just win it. They broke it.

The competition is ILSVRC, the ImageNet Large Scale Visual Recognition Challenge. Roughly 1.2 million labelled photographs, a thousand categories, and one question: can your software tell a beagle from a basset hound. Progress in this contest is normally measured in fractions of a percent. Teams grind out a point of accuracy and publish a paper about it.

This year Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton entered a deep convolutional neural network and finished with a top-5 error rate of 15.3%. The next best entry came in at 26.2%.

That is not a win. That is a different sport.

“Progress in this field is normally measured in fractions of a percent. This was measured in double digits. When a number moves that far, the method behind it matters more than the number.” — Sameer Gupta

What They Actually Built

The model is eight layers deep: five convolutional layers followed by three fully-connected ones, about 60 million parameters and 650,000 neurons. It was trained on two NVIDIA GTX 580 cards with 3GB of memory each, for roughly six days.

Read that again. Two consumer graphics cards. Six days. This was not a supercomputer.

Three engineering choices did most of the heavy lifting:

  • ReLU activations. Instead of the smooth tanh function everyone had been using, they used a rectified linear unit, which is just “output the input if it’s positive, otherwise output zero.” Crude, and about six times faster to converge. Sometimes the dumb thing is the right thing.
  • Dropout. During training, half the neurons in the fully-connected layers are randomly switched off on every pass. It sounds like sabotage. It stops the network from over-relying on any single path, and it is the reason a 60-million-parameter model trained on 1.2 million images doesn’t simply memorise them.
  • Data augmentation. They generated extra training images by cropping, flipping, and shifting the colour balance of the originals. Free data, more or less.

The Part That Should Get Your Attention

Here is what I keep coming back to, and it has nothing to do with dogs.

For the last decade, computer vision has meant a clever person designing features by hand. SIFT. HOG. Colour histograms. Edge detectors. A researcher would spend years working out what mathematical description of an image best captures “there is a car here,” then feed those hand-built descriptors into a classifier.

This model was given raw pixels. It worked out the features itself.

“For ten years, computer vision meant a smart person designing features by hand. This year a machine learned them from pixels, and beat every one of those smart people at once.” — Sameer Gupta

That’s the shift. Not that a computer can label photographs, but that the expensive, artisanal, human part of the pipeline just got automated.

So What Does This Mean for a Business?

I want to be careful here, because I’m still working through this material myself and I’d rather be useful than breathless.

What’s genuinely changed:

  • If your company generates images and you’ve been ignoring them because analysing them required a research team, that calculation is shifting. Think quality inspection photos on a production line, scanned documents in a back office, shelf photography in retail, damage assessment in insurance claims.
  • The hardware is affordable. Two gaming cards is a rounding error on most IT budgets. The barrier was never really the machines.

What hasn’t changed, and you need to hear this part:

  • You need labelled data, and a lot of it. ImageNet is 1.2 million hand-labelled images and it took years of human effort to build. If you have four hundred photos of defective parts, this technique will not save you.
  • You need people who can do this. There are perhaps a few dozen groups in the world who can train one of these networks reliably today. That will change, but not this quarter.
  • Nobody can fully explain why it works this well. The theory is genuinely behind the practice right now. If you’re in a regulated industry and someone asks why the model rejected a claim, “the network decided” is not an answer.

What I’m Watching Next

A few things I’d bet on, with the usual caveat that I’m learning this alongside everyone else:

The techniques will get packaged. Right now, training one of these is a craft. Within a couple of years I’d expect libraries that make it routine, the way statistical packages made regression routine.

The labelled data problem becomes the real bottleneck. Once the modelling is commoditised, whoever owns a large, clean, well-labelled dataset in their domain holds the actual asset. That’s a strategic question, not a technical one, and it’s the one I’d be asking in a board meeting.

And the GPU stops being a gamer’s accessory. If this approach generalises past images, the ability to do enormous amounts of parallel arithmetic cheaply becomes the constraint on the whole field.

Final Thoughts

I don’t think this result means computers can see. It means one narrow, well-defined, richly-labelled problem yielded to a method that learns its own features, given enough data and enough parallel compute.

That’s a smaller claim than the headlines will make. It’s also a more useful one, because it tells you exactly what conditions you need to reproduce it: a problem you can define crisply, data you can label at volume, and patience.

Most companies have the third and not the first two. Start there.