Tokenization

What Is Tokenization?

Tokenization is the process of splitting text into tokens, the small chunks a language model actually reads, using a fixed vocabulary learned from training data. Common words become single tokens while rare words break into fragments, and every model API meters cost and context limits in the tokens this process produces. As a reference point, Anthropic's tokenizer maps roughly 3.5 English characters to one token, with the ratio varying across languages [1].

Key Takeaways

• Tokenization happens before the model sees anything. Text goes in, a sequence of vocabulary IDs comes out, and those IDs are the model's entire view of your input.
• The vocabulary is learned by frequency, which is why English prose tokenizes efficiently while code, JSON, and non-English languages can cost two to four times more tokens for equivalent content.
• Different model families use different tokenizers, so the same prompt has different token counts, and different costs, across providers.
• Several classic LLM failures trace back to tokenization, including weak letter counting and brittle arithmetic, because the model sees chunks rather than characters.

How It Works

Modern models use subword tokenization, most commonly byte-pair encoding (BPE) or a close variant. Vocabulary construction starts from raw bytes and repeatedly merges the most frequent adjacent pairs found in a training corpus, until the vocabulary reaches a target size, typically 50,000 to 200,000 entries. Vocabulary size is a real efficiency lever: Meta reported in 2024 that Llama 3's 128K-token vocabulary yields up to 15% fewer tokens on the same text compared to Llama 2 [2]. Frequent strings like "function" or " the" end up as single entries. Rare strings survive only as fragments: a made-up word like "flimbork" might tokenize as "fl", "imb", "ork". Because the base units are bytes, any input can be tokenized, including emoji, typos, and binary-looking strings, with no unknown-word failures.

At request time the tokenizer greedily matches your text against this vocabulary and hands the model a list of integer IDs. Inside the model each ID is mapped to a vector embedding before the transformer layers process it. Generation reverses the pipeline: the model picks a token ID, the tokenizer decodes it back to text, and streaming APIs ship those decoded fragments to your terminal as they arrive.

The engineering consequences are mostly economic. Token count, not character count, decides what fits in a context window and what a request costs. Whitespace handling matters too: deeply indented code, pretty-printed JSON, and long UUIDs are all token-expensive, which is why minifying payloads and normalizing indentation are real cost optimizations in agent pipelines.

Example

A team builds an agent that reviews API responses for anomalies and initially passes pretty-printed JSON into the prompt. A typical 80 KB payload comes to about 30,000 tokens. An engineer changes the pipeline to strip whitespace, shorten repeated keys, and drop fields the reviewer never uses. The same payload now tokenizes at around 9,000 tokens. Across a few thousand runs a day, that one change cuts input spend by roughly two-thirds and leaves headroom in the context window for longer conversation history, with no model or prompt changes at all.

What People Get Wrong

The classic misconception is that models read characters or whole words. They read tokens, and that explains a family of otherwise baffling failures. Ask a model how many times the letter "r" appears in "strawberry" and it may miscount, because it perceives the word as two or three opaque chunks, not eleven letters. The same applies to reversing strings, precise character offsets, and digit-level arithmetic. When an agent needs character-exact operations, the reliable pattern is to have it call a tool or write code for the operation instead of answering from perception.

FAQ

Is tokenization the same as embedding? No, they are consecutive steps. Tokenization splits text into vocabulary IDs. Embedding then maps each ID to a numeric vector the network can compute with. Tokenization is lossless and reversible; embeddings are learned representations of meaning.

Why do the same words cost different amounts on different models? Each provider trains its own tokenizer on its own corpus, so vocabularies differ. A prompt that is 1,000 tokens on one model may be 1,150 on another. When comparing model pricing, compare cost for your actual workload, not the per-token sticker price.

Can I count tokens before sending a request? Yes. Providers ship tokenizer libraries and token-counting endpoints that return exact counts for a given model. Use them to enforce budgets, size chunks for retrieval, and predict costs, rather than estimating from word counts.

Sources

1. Anthropic. "Claude glossary: one token approximates 3.5 English characters." https://platform.claude.com/docs/en/about-claude/glossary. Accessed August 2026.
2. Meta AI. "Introducing Meta Llama 3: 128K-token vocabulary with up to 15% fewer tokens than Llama 2." https://ai.meta.com/blog/meta-llama-3/. Accessed August 2026.