Attention Is All You Need — and All You Should Budget For

Every so often a paper arrives with a title so confident it’s almost rude. In June, eight researchers at Google published one called “Attention Is All You Need,” and having spent a few weeks with it, I think the title is fair.

They threw out the recurrent neural network, which has been the foundation of machine translation and most serious language work for years, and replaced it with something simpler that trains faster and works better.

For anyone paying GPU bills, the second half of that sentence is the one that matters.

The Problem With Reading One Word at a Time

To understand why this is a big deal, you have to understand what was wrong with the old approach.

A recurrent network reads a sentence the way you’d read it aloud: one word, then the next, carrying a running summary in its head. To process word fifty, it must first have processed words one through forty-nine.

That’s a reasonable model of reading. It is a catastrophic model for a modern GPU.

A graphics card is a machine for doing thousands of arithmetic operations simultaneously. Handing it a task that must be done strictly in order is like buying a thousand-seat call centre and taking one call at a time. You’ve paid for the parallelism and you cannot use it.

“The bottleneck in language modelling was never really the mathematics. It was that we’d built a sequential algorithm and then bought hardware that is only fast when nothing is sequential.” — Sameer Gupta

There was a second problem. Information from the start of a long sentence has to survive being passed through fifty successive steps to reach the end. In practice a lot of it doesn’t. Long-range relationships get lost.

What They Did Instead

The Transformer, as they call it, reads the entire sentence at once.

The mechanism is self-attention. For every word, the model computes how much each other word in the sentence should influence its interpretation. In “the trophy didn’t fit in the suitcase because it was too big,” working out what “it” refers to requires looking directly at “trophy,” seven words back. Self-attention does that in one operation rather than seven.

They run this eight times in parallel with different learned parameters, which they call multi-head attention. Different heads end up specialising, some tracking grammatical structure, others tracking meaning. Because word order matters and the model no longer processes words in order, they inject position information directly into the input as a set of sine and cosine signals.

There is no recurrence anywhere in the architecture. Every word is processed at the same time as every other word.

The Number That Should Get a Budget Holder’s Attention

Here is the result that made me put down my coffee.

The big Transformer reached a new state of the art on English-to-German translation after three and a half days on eight GPUs. The previous best results came from large ensembles that consumed, by the paper’s own accounting, more than an order of magnitude more compute to train.

Better output. Roughly a tenth of the training cost.

“When something gets both better and dramatically cheaper at the same time, that isn’t an improvement to the existing approach. That’s a replacement, and everyone will switch inside two years.” — Sameer Gupta

I’ve been making a version of this argument for a while in a different context. The interesting frontier in this field isn’t what the largest possible model can do. It’s what a normal team with a normal budget can now afford to train. This paper moves that line further than anything I’ve read this year.

Why This Matters Outside a Research Lab

Language work gets cheaper, which means it gets attempted. Plenty of businesses have a text problem they’ve never touched because building a translation, summarisation, or classification system meant a research team and a month of training. Cut the training cost by an order of magnitude and that calculation changes for a lot of organisations at once.

Sequence models aren’t only about language. Anything ordered has the same shape. Clickstreams. Transaction histories. Sensor readings. Event logs. If self-attention beats recurrence on translation because of parallelism and long-range dependencies, the same argument applies to a year of a customer’s purchases.

Hardware planning shifts. Recurrent models were latency-bound by their own structure. Transformers are throughput-bound, which is what modern accelerators are good at. If you’re forecasting infrastructure for language work, the shape of the demand is about to change.

Fair Warnings

Self-attention is quadratic in sequence length. Every word attends to every other word, so doubling the sentence length quadruples the work. It’s brilliant on sentences and expensive on documents. Somebody will need to solve that.
Memory, not compute, will bite you first. Those attention matrices are large and they live on the GPU. Training the big model in the paper needs serious cards.
This is a translation paper. The authors are appropriately careful about that. The extensions to other tasks are speculation, mine included, until someone publishes them.

Final Thoughts

I try to be sceptical about anything with an emphatic title, and I’ve argued often enough that the field overstates its own pace.

This one I’d take seriously. Not because the translation scores went up, but because the architecture removes the specific constraint that made language models slow to train, and it does so by making the whole thing more parallel rather than more complicated.

Simpler, faster, and better is a rare combination. When it turns up, it usually wins.