Before ChatGPT, a mathematician counted 20,000 letters of a poem

In 1913, Andrey Markov counted the vowels of a Russian classic by hand. In 1948, Claude Shannon used the idea to make a machine write. The grandparents of chatty AI.

You start typing “good” on your phone and the keyboard already suggests “morning”. You write a question to an AI and it answers in seconds, with sentences that sound human. It feels like recent magic, something from the last few years.

But the question driving all of it is old. It was first asked with pencil, paper and a book of poetry, in 1913, by a very patient Russian mathematician.

A mathematician, a poem and a lot of patience

The mathematician was Andrey Markov. The book was Eugene Onegin, a novel in verse by Alexander Pushkin, a classic of Russian literature.

According to the engineering magazine IEEE Spectrum, Markov took the first 20,000 letters of the book and wrote them all out as a single line, with no spaces and no punctuation. Then he split that line into 200 grids of 10 by 10 letters and counted the vowels in each row and each column.

No computer, no spreadsheet. Just him, the text and the arithmetic.

Diagram showing 20,000 letters arranged in 200 grids of 10 by 10, highlighting the result: 43% vowels and 57% consonants.
Markov's method in 1913, according to IEEE Spectrum (labels in Portuguese). Illustration: Escovando Bytes

What the letters revealed

The first result was simple: 43% of the letters were vowels and 57% were consonants.

The interesting result came when he looked at pairs, that is, each letter together with the next one. Vowel followed by vowel appeared 1,104 times. Consonant followed by consonant, 3,827 times. And the mixed pairs, vowel with consonant or consonant with vowel, added up to 15,069.

Do some quick math with me. If the letters landed on the page at random, with no connection between them, and 43% of them were vowels, the vowel-vowel pair should show up about 3,700 times in 20,000. It appeared less than a third of that.

In other words: the next letter isn’t random. It depends on the one before it. After a vowel, a consonant is much more likely, and vice versa.

Bar chart of the letter pairs Markov counted: 15,069 mixed pairs, 3,827 consonant-consonant and 1,104 vowel-vowel, compared with about 3,700 you would expect by chance.
Markov's numbers, via IEEE Spectrum. The random-draw estimate is a simple calculation based on them (labels in Portuguese). Illustration: Escovando Bytes

That idea, a sequence in which each step depends on the previous one, took its author’s name: today people talk about “Markov chains”. His paper, however, was rarely cited at the time and was only translated into English in 2006, almost a century later.

1948: what if the machine wrote?

Cut to 1948. Engineer Claude Shannon publishes “A Mathematical Theory of Communication”, a work about how to measure and transmit information. In the middle of it, Shannon runs an experiment that closely resembles Markov’s, only in reverse.

Markov counted letters to understand a text. Shannon used the counts to manufacture a text.

He did it in steps, as the math outreach site Plus Magazine recounts:

  • Step 1: draw letters with equal odds for all of them. The result is alphabet soup that makes no sense.
  • Step 2: draw letters following the frequency of English, in which E appears far more than W. Still meaningless, but it starts to look like a language.
  • Step 3: choose each letter based on the previous one and, in the next step, on the previous two. Real words begin to appear among the made-up ones.
  • Step 4: swap letters for whole words, each chosen based on the previous word.

At the last step, the drawing machine produced a sentence that begins like this: “THE HEAD AND IN FRONTAL ATTACK ON AN ENGLISH WRITER”. It means nothing. But, read quickly, it looks like English.

Staircase with the steps of Shannon's experiment, from drawing letters with no rule to choosing words based on the previous one, with a snippet of the generated text at each step.
The texts Shannon generated, as reproduced by Plus Magazine (labels in Portuguese). Illustration: Escovando Bytes

Plus Magazine notes that Shannon seems to have stopped at the word step, and that by looking at more previous words, at some point the text would start to read well. Keep that sentence in mind.

What this has to do with today’s AI

IEEE Spectrum calls Markov and Shannon the authors of the first language generation models. That’s what a language model is: a program that, given the start of a text, calculates what probably comes next.

The AI assistants that chat with you today, like ChatGPT, are distant descendants of that idea. The difference is scale and technique. Instead of looking at one previous letter or word, they look at the whole text of the conversation. Instead of counts done by hand in a book, they learn from an enormous amount of text and very powerful computers. But the underlying question is the same as in 1913: what comes next?

I started programming when I was 11, and one of the things that still fascinates me is seeing how simple ideas travel across decades. Counting vowels in a poem seems like the most useless thing in the world. It ended up becoming one of the foundations of the technology in your phone.

Why this is worth knowing

Understanding where AI comes from helps you use it better. If the machine writes by choosing the most likely continuation, it’s great at producing text that sounds right. Sounding right and being right are different things.

Shannon’s sentence about the English writer looked like English and said nothing. Today’s AIs go far beyond that, but the lesson still holds: fluent text is not proof of truth. When in doubt, check the source.

And next time your phone’s keyboard finishes your word, think of a Russian mathematician hunched over 20,000 letters, counting vowels by hand.

Keep readingSee all →