AI/Biz

BERT Reads Both Ways

In August I wrote about OpenAI’s pre-training paper and flagged one thing that seemed obviously fixable. The model read left to right, so when interpreting any given word it could see what came before but never what came after.

That felt like a handicap somebody would attack. In October, Google attacked it. The paper is called BERT and the results are not close.

Covering Half the Sentence

Consider: “I went to the bank to withdraw some cash.”

By the time a left-to-right model reaches “bank,” it has seen “I went to the.” That’s it. It has to commit to an interpretation without the word “cash,” which is the one word that resolves the ambiguity.

You read the whole sentence. The model was reading it with a hand covering the right half.

The reason nobody had simply removed the hand is that it breaks the training objective. If the task is “predict the next word” and the model can see all the words, the answer is right there. The model learns nothing except how to copy.

The Trick

Google’s solution, from Jacob Devlin and colleagues, is the masked language model, and it’s the kind of idea that seems obvious only afterwards.

Take a sentence. Hide about 15% of the words at random. Ask the model to fill in the blanks, using everything on both sides.

That’s it. The model now sees full context in both directions, and the task remains genuinely hard because it can’t see the specific words it’s being asked to produce. It’s a fill-in-the-blank exercise rather than a guess-the-next-word one.

“The obstacle was never the architecture. It was that our training objective quietly required the model to be half blind, and it took someone asking a different question to notice.” — Sameer Gupta

They added a second objective, next sentence prediction, where the model is shown two passages and asked whether the second actually followed the first. That teaches it something about relationships between sentences, which matters for question answering.

The Results

BERT-Large is 340 million parameters, 24 layers. BERT-Base is 110 million, which matters because it will fit places the larger one won’t.

It set new state of the art on eleven language tasks. The GLUE benchmark score went from 75.1 to 80.5, which in a field that fights over fractions is not an improvement so much as a reset. On SQuAD question answering it passed the human performance baseline.

Training took four days on a substantial pod of Google’s TPUs. Call it tens of thousands of dollars of compute if you were renting it.

Why the Release Matters More Than the Score

Here is the thing I’d actually put in front of a business audience, and it isn’t in the benchmark table.

Google published the trained weights.

You do not pay the tens of thousands of dollars. You do not need a TPU pod, a research team, or a corpus of Wikipedia and 11,000 books. You download a model that already understands English, then fine-tune it on your own labelled data.

  • Fine-tuning runs in hours on a single GPU.
  • It typically needs thousands of labelled examples, not millions.
  • The same base model serves classification, extraction, question answering, and similarity, with a different small head on top.

I made this argument earlier in the year about the second-hand frontier in computer vision, where fine-tuning somebody else’s enormous pre-trained network became the standard way to work. Language has now arrived at the same place, and it took about three years less than I’d have guessed.

Things I’d look at immediately if I had a text-heavy operation:

  • Support ticket classification and routing. The classic keyword-rule system that breaks whenever a customer phrases something new.
  • Contract and document extraction. Pulling dates, parties, and obligations out of unstructured agreements.
  • Search over internal knowledge. Matching a question to an answer that shares no keywords with it.
  • Compliance review triage. Not deciding, but ranking what a human should look at first.

Where I’d Be Careful

  • 340 million parameters is heavy to serve. Training is a one-off. Inference is forever. If you need answers in milliseconds at high volume, budget properly, or use the base model and accept the accuracy cost.
  • Benchmarks are not your data. BERT is superb on clean, well-formed English. Your customer emails contain typos, three languages, and pasted log files.
  • It is completely opaque. We now have a far more capable language model with no more insight into its reasoning than we had before. GDPR came into force in May and the questions regulators are asking about automated decisions are getting sharper, not softer. Plan for having to explain something you cannot inspect.
  • It absorbed whatever was in the corpus. Wikipedia and a pile of books carry assumptions about the world. Those assumptions are now in your ticket router.

Final Thoughts

Two papers this year, four months apart, have between them moved language processing from something you needed a research team to attempt to something a good engineer can fine-tune in an afternoon.

I’ve written repeatedly that the pace of this field is overstated in the press and understated in the places that matter. This is one of the places that matters. Nothing about BERT will make a newspaper. It will quietly become the default starting point for every serious text project inside about eighteen months.

If you have unstructured text and a business problem attached to it, this is the moment the tooling caught up with the ambition.