How ChatGPT Reads a Sentence

The answer is that an AI never actually sees the letters. Before a single word reaches the model, the text is chopped into pieces called tokens — and the model only ever sees those pieces. Let’s look at the text the way it does.

Type anything below. Each colored chunk is one token. Try the presets — watch what happens to a long word, a big number, and poor “strawberry.”

A few patterns jump out once you play with it:

Common words stay whole. “the,” “sentence,” “read” — each is a single token, because the model has seen them a billion times.

Rare or long words shatter. Something like “tokenization” becomes “token” + “ization.” The model stitches meaning back together from fragments.

Numbers break into chunks, and a space is usually glued to the front of the next word. To the AI, “ cat” (with a space) and “cat” are different tokens entirely.

Tokens aren’t just trivia — they’re the unit AI is measured in. A model’s “context window” (how much it can read at once) and your bill from the API are both counted in tokens, not words. And because English packs more words per token than most languages, the same paragraph in, say, Turkish or Japanese can cost noticeably more to process. The humble token quietly sets the price and the memory of modern AI.

So why chop text into these odd pieces at all? The obvious idea — give every whole word its own token — falls apart fast. There are millions of words across the world’s languages, plus names, typos, slang, and brand-new coinages every day. A vocabulary that big would be unwieldy, and it would still choke the first time it met a word it had never seen. The opposite extreme — one token per single character — never hits an unknown letter, but it makes every sentence enormously long, and the model has to work much harder to find meaning in a soup of individual letters.

The winning compromise is subword tokenization. The most common recipe is Byte-Pair Encoding (BPE): start with plain characters, then repeatedly find the most frequent adjacent pair and merge it into a new chunk. Do this thousands of times and you grow a vocabulary of handy building blocks — pieces like “ing,” “tion,” and “ the.” Words you see constantly end up as a single token; rare ones get spelled out from smaller pieces. Best of both worlds: short sequences for common text, and nothing is ever truly unknown.

A useful rule of thumb: $1\ \text{token} \approx \tfrac{3}{4}\ \text{English word}$, so roughly 100 tokens ≈ 75 words. It’s only an average — short common words are one token, while a clunky technical term may be three or four — but it’s close enough to estimate with.

Every time you chat with an AI, write a prompt, or pay for an API call, tokens are the currency changing hands. Knowing they exist is the first real step from “AI is magic” to “AI is a system I can reason about” — and it’s the doorway into everything that follows: embeddings, attention, and how these models actually predict what comes next.