The bigram model

When we're small, nobody hands us a manual for learning to talk. We learn by living: we hear the people around us, we tie a tone of voice to a smiling face, and little by little the meaning of things sinks in.

But how do you teach writing to a machine that has never lived a single day? To a box of wires and silicon, the word apple isn't sweet or red. It means nothing. If it can't understand the world, writing about it looks impossible.

So before we build anything, a small experiment.

You probably filled every blank without much effort. But look at the last one: you had no idea what Fli fli fla means. You just looked at what came before, sensed the logic, and guessed what came next.

You just found the trick. Since engineers couldn't teach machines to understand the world the way we do, they changed the rules of the game. Instead of teaching them to reflect, they taught them to predict.

Today, the big models do this with whole sentences. But to really understand the magic behind it, we'll go to the most basic thing of all: predicting what the next letter is.

That's the goal of everything that follows: a machine that looks at what you've written and bets on the letter that comes next. Then another. And another. Until it writes on its own.

And there's the awkward question: how do you build that from scratch, when the machine can't even read?

Is there a pattern between the letters of a language? What usually comes after a «t»? What about after a «v»?

You have never stopped to think about it, and you have been writing your whole life. Let's look at one sentence, two letters at a time.

We write following an invisible structure. Nobody ever explained that 70% of the time you write a «v», the letter right behind it is an «e»; or that some pairs never happen at all: in a whole book there isn't a single «aa», nor a «uu», nor a «dt». Your brain just absorbed it from reading and listening. Mash the keyboard, asdfghjkl, and none of that turns up.

A short sentence has room for only a handful of repeats. Let's give it a longer one. And to keep from going mad, we'll narrow to a single letter, the «t»: let's see which letters tend to keep it company.

Depending on the text you give it, it learns a different rule. With so little to read, what it learns tells you more about that sentence than about the language: its view of the world is a pinch of it, and a biased pinch at that.

So let's get serious. Let's have our machine read all of Shakespeare.

That row is everything there is about the «t» in all of Shakespeare. The machine pulled out every one of its connections on its own, just by counting.

This process —handing a machine a giant text so it reads, counts, and builds its own tables of rules— is called training data. You just watched, first-hand, how a model is trained. (One catch: it learned from Shakespeare, so it'll talk like it's 400 years ago. Change the book and you change the machine.)

We have 7,071. Seven thousand and seventy-one times that, in all of Shakespeare, an «h» came right after a «t». A huge number. Now tell me: what do we do with it?

Think about it for a second. 7,071 would be enormous if the «t» showed up 8,000 times in the whole book… and next to nothing if it showed up a million. On its own, the number says nothing: it only means something next to the total. And we have that total — it's sitting in the same row.

So it's time for the only piece of arithmetic in this entire chapter. And it's the kind you learned at school: a division.

7,071 divided by 19,763. That's 36 %. And that figure, said the way you'd say it out loud: out of every 100 times you write a «t», 36 of them are followed by an «h». No more, no less.

Run the same sum down the rest of the row and you no longer have counts: the space takes 29 %, the «o» 10 %, the «e» 7 %, and all the others share what's left. The counts were dead data. These are bets: hand the machine a letter and it answers, in percentages, what it's wagering comes next.

And a machine that bets is nearly a machine that writes. It's only missing the hard part: deciding.

The careful thing would be to never gamble: look at the row and always take the biggest number. It's the most likely one — what could go wrong? Try it.

Always «h». And «h» again. And again. However many times you ask, as long as the rule is "keep the biggest number", nothing else will ever come after a «t». Not the space, not an «o», not an «e». They're all still there in the row, and 64 % of the die never gets used.

Always picking the likeliest doesn't give you the safest answer: it gives you the only answer. The safe move comes out dead.

It needs randomness. But not coin-toss randomness, which would treat the «h», turning up 36 % of the time, exactly like the «g», which followed a «t» once in all of Shakespeare. That would throw away everything the machine just read.

It needs randomness that respects what was learned. The engineers' idea was a die: a loaded one, of course, with loads of «h» faces, plenty of space, the odd «o», and almost none of the rare letters. So the likely thing usually comes up, but every so often it surprises you. Roll it yourself and see.

Roll it a few times. Mostly the «h», the widest stretch on the bar; every so often the space, an «o», something rarer. The die invents nothing: every letter takes up exactly the room it earned by being counted.

One last question — and it's the one the products you already use actually run on. What if we don't want that much surprise? What if we want far more?

You can rig the die again. Squeeze the percentages toward the winner, until the «h» eats the whole bar and we're back where we started: the same letter every time. Or flatten them, so the rare letters start getting the same chances as the common ones. That dial exists and it has a name: temperature. Move it yourself.

Low temperature: obedient, repetitive, dull. High: creative, surprising and, past a point, unhinged. When a model lets you pick between «precise» and «creative», underneath it is almost always doing exactly this: moving this dial. There's no more magic to it.

We've got the whole trick for the «t»: count, divide, and pick with a spark of randomness (more spark or less, as it suits us). And the question asks itself: what if we did exactly this for every letter at once?

We have a row for «t». What about «a»? And «h»? And all the rest?

Any cell reads the same way: the letter on the left is the one you start from, the letter on top is what comes next, and the number is how many times it happened. That's all. And the thing that just came out has a name of its own: a transition table.

That came out of a five-word phrase. With a whole book in front of it, nobody has to count by hand.

And that is still one corner of the language: lowercase. Count the capitals, the periods, the commas and the numbers too, and the table grows to its real size. The one below is that one: it looks like a mess of light, but every lit cell is a rule of the language and every dark gap a pair that almost never happens. Nobody taught it any of them.

With the table in front of us there is no special letter any more. What we did by hand with the «t» row now holds for all twenty-seven: give it whichever letter you like and the machine answers with what comes next.

And picking one we already know how to do: look at the row, roll the die. Here is that step in slow motion, and each letter that lands is the starting point for the next.

And there it is: a machine that writes on its own. Nobody taught it spelling, or grammar, or a single rule. It just counted pairs of letters in a pile of text, and everything came out of that. You built it, from scratch.

You've seen it in slow motion. At full speed it's this: one letter after another, no brakes, whole phrases pour out at once.

Before writing it off, a fair question: bad compared to what? Here it is between the two things it isn't — the chance of mashing a keyboard, and the book it read — with two lenses for looking at all three.

And now the bad news. Read it again and it almost sounds like a real language: the letters fit together… but they aren't words. Babble with a good accent. We pulled it off, it writes on its own. But what a mess, right? Why does it write so badly?

Before we fix it, let's understand why it writes so badly. What comes after «th»? The machine couldn't care less about the «t»: it only looks at the «h». To it, «th», «sh» and «wh» are exactly the same thing.

It's not forgetful. It's blind from birth. No matter how much text you give it, it will never tell «th» from «sh». This isn't a bug you fix with more data. It's the ceiling of the model.

What if it could see more than one letter? Your turn: a word reveals itself one letter at a time, and you bet on the next.

Did you feel it? With one letter you were guessing blind. With almost the whole word in front of you, almost certain. More context, better prediction. That's exactly what our model is missing: it only sees one piece back. Just like you with «hi»: it reacts to the last thing it heard, with no idea about the rest.

What if we teach it to look at two letters? Three? Five? That's already a different model. And it's the next one.