Words as Numbers: What word2vec Means for Your Business

Computers don’t understand words. They never have. For decades our workaround has been to treat a document as a bag of tokens, count them, and hope the counting tells us something useful.

It mostly does. Spam filters work. Search works, more or less. But the approach has an obvious hole: to a counting machine, “physician” and “doctor” are two entirely unrelated strings. They share no letters in common position, so they share nothing at all.

In January, a team at Google led by Tomas Mikolov published a paper that closes that hole cheaply enough to matter.

The Idea

The technique is called word2vec, and the concept is disarmingly simple. Every word gets assigned a list of numbers, a position in a space of a few hundred dimensions. Words that show up in similar company end up in similar places.

That’s it. No dictionary, no grammar rules, no linguists. You point it at a large pile of ordinary text and it works out the positions by trying to predict which words surround which.

The results are strange and a bit delightful. Because words are now coordinates, you can do arithmetic on them. The paper’s famous example: take the vector for “king,” subtract “man,” add “woman,” and the nearest word to where you land is “queen.” Take “Paris,” subtract “France,” add “Italy,” and you get “Rome.”

“Nobody taught it that Paris is to France what Rome is to Italy. It read enough text that the relationship fell out of the geometry.” — Sameer Gupta

The model didn’t learn what a capital city is. It learned that a particular direction in the space means “is the capital of,” because that pattern is sitting in the statistics of ordinary sentences, waiting for someone to measure it.

The Number That Actually Matters

The linguistics are fun. The engineering is what makes this a business story.

Previous attempts at neural language models were academically interesting and commercially hopeless, because they took weeks to train on far smaller vocabularies. Mikolov’s team trained on 1.6 billion words in under a day, on ordinary hardware.

They got there by throwing away almost everything. The two architectures in the paper, continuous bag-of-words and skip-gram, are shallow. There’s no deep stack of hidden layers doing expensive matrix multiplication. It is close to the simplest thing that could possibly work, and it turns out that at this scale, the simplest thing that could possibly work is enough.

That is a pattern worth internalising. A method that is 90% as good and 100 times cheaper doesn’t compete with the expensive method. It replaces it, because it’s the only one anybody can actually run.

Where I’d Use This Tomorrow

Unusually for anything in machine learning right now, this requires no labelled data. You need text. You almost certainly have text.

Search that understands synonyms. A customer searches your catalogue for “laptop bag” and you have it listed as “notebook sleeve.” Today you miss the sale. Train vectors on your own product descriptions and support logs, and the two land next to each other without anyone maintaining a synonym list by hand.
Support ticket routing. Tickets that use different vocabulary for the same underlying problem cluster together. You stop routing on keyword rules that break every time a customer phrases something new.
Product matching across suppliers. Anyone who has tried to reconcile two catalogues from two vendors knows this pain. Vectors give you a similarity score that survives different naming conventions.
Domain vocabulary that nobody has written down. Train on your own corpus and you get your industry’s jargon, your product names, your internal shorthand, all positioned correctly. A general-purpose thesaurus will never contain your part numbers.

That last one is the one I’d push hardest. The value here isn’t in the model, it’s in running the model on your own text. Your competitors can read the same paper. They can’t read your ten years of support tickets.

The Honest Caveats

I’ve been reading around this for a few weeks and I want to be straight about the limits, because the demos are seductive.

It has no idea what a word means. It knows what a word is near. Those are different things, and the difference will bite you at some point.
It learns whatever is in your corpus, including the parts you’d rather it didn’t. If your historical text associates certain job titles with certain kinds of people, the vectors will encode that association faithfully and hand it back to you as a recommendation. Nobody has a good answer for this yet.
One word, one vector. “Bank” as in river and “bank” as in money get averaged into a single confused point in space. Context-sensitivity is an unsolved problem here.
You need real volume. Not a billion words necessarily, but tens of millions. A few thousand documents won’t give you stable positions.

Final Thoughts

Most of what gets written about machine learning describes things you can’t do yet, with data you don’t have, on hardware you’d have to buy.

This is the opposite. It’s a cheap technique, published with the numbers to back it up, that runs on text you’re already storing and pays off in systems you already operate. If you have an engineer with a spare week and a corpus of company documents, this is a genuinely good use of both.

Start with search. It’s the easiest place to measure whether it worked.