Tokenization is the step that converts text into the numbered pieces, called tokens, that a language model actually processes. A token is usually a whole common word, part of a rarer word, or a punctuation mark, chosen from a fixed vocabulary learned before the model is trained. The method used by GPT-style models, byte-pair encoding (BPE), starts from single characters and repeatedly merges the most frequent neighboring pair. Understanding it explains why models miscount letters, why some languages cost more to process, and why prices and context limits are quoted in tokens.
Why models need tokens
A neural network works with numbers, not letters. Something has to map text to numbers and back. The simplest options both have drawbacks. Splitting by character gives a tiny vocabulary but very long sequences, and each character carries little meaning. Splitting by word gives short sequences but a huge vocabulary, and any word the system has not seen becomes an “unknown” symbol.
Subword tokenization sits between the two. Common words get a single token, while rare words are built from reusable pieces. The 2015 paper “Neural Machine Translation of Rare Words with Subword Units” adapted byte-pair encoding, originally a text-compression method, to build such vocabularies for translation, which let models handle rare and new words without an unknown-word symbol. The Hugging Face course notes that BPE was initially developed as a compression algorithm and was then used by OpenAI for tokenization when pretraining GPT; it is used by many transformer models, including GPT-2, RoBERTa, BART and DeBERTa.
How byte-pair encoding is trained
The Hugging Face course walks through a small example. Imagine a training text containing the words “hug,” “pug,” “pun,” “bun” and “hugs.” The process goes as follows.
- Start with characters. The base vocabulary is every symbol used: b, g, h, n, p, s, u.
- Count adjacent pairs in the corpus, weighted by how often each word appears.
- Merge the most frequent pair into a new token. In the course’s example, “u” and “g” are merged first into “ug”, then “u” and “n” into “un”, then “h” and “ug” into “hug”.
- Repeat until the vocabulary reaches a chosen size.
The merge rules are saved in order. To tokenize new text, the tokenizer splits it into characters and applies those rules in the same order. The word “bug” becomes “b” and “ug”. Because the toy vocabulary has no letter “m”, the course shows “mug” falling back to an unknown token. That is exactly why production systems use a trick: GPT-2 and RoBERTa tokenizers work on bytes, giving a base vocabulary of 256 and guaranteeing that any character can be represented. This is called byte-level BPE.
Other tokenizer families
BPE is not the only approach. WordPiece, developed by Google for BERT, learns merges too but chooses pairs using a score that divides the pair’s frequency by the frequencies of its two parts, rather than picking the most frequent pair. It marks word-internal pieces with a prefix such as “##”. The Hugging Face course adds that Google never open-sourced its training implementation, so public descriptions are best guesses.
Unigram works in the opposite direction. It starts with a large vocabulary and removes the tokens whose absence increases the overall loss the least until it reaches the target size. It is typically paired with SentencePiece, which treats input as a raw stream including spaces, so it works for languages that do not separate words with spaces. Models such as T5 and XLNet use it, according to the course.
What this means in practice
Counting letters is hard. A model sees “strawberry” as one or a few tokens, not ten separate letters. Questions about spelling or letter counts ask it about structure it never directly observed, which is one reason such tasks trip up chatbots.
Cost and limits are in tokens. Context windows and API prices are quoted in tokens, not words. As our glossary notes, 1,000 English tokens is roughly 750 words, but the ratio varies. See our explainer on what a context window is and the economics of inference costs.
Languages are not equal. A vocabulary trained mostly on English text splits other languages into more pieces. The same sentence in another language can need more tokens, so it costs more and fills the context window faster.
Numbers, code and spacing matter. Whether “2026” is one token or several, or whether a leading space belongs to the word, depends on the tokenizer. Prompts that look identical to a human can tokenize differently, which can nudge model behavior.
Tokenizers are tied to models. A model must be used with the tokenizer it was trained with. Counts from one vendor’s tokenizer do not transfer exactly to another’s.
How to check token counts
Most model providers publish tokenizer tools or libraries, and open-source libraries such as OpenAI’s tiktoken and Hugging Face’s tokenizers let you encode text and see the pieces. Use the one that matches your model before estimating cost or fitting text into a limit. For rough planning, a ratio like 750 words per 1,000 tokens in English is a start, but measure real data when it matters.
What to watch next
- Research on tokenizer-free or byte-level models that avoid fixed vocabularies.
- Larger multilingual vocabularies that reduce the cost gap between languages.
- Providers changing tokenizers between model versions, which alters costs for the same text.
- Tooling that shows token boundaries directly in chat and API interfaces.
Key takeaways
- Tokens are the units of text a model reads, drawn from a fixed learned vocabulary.
- Byte-pair encoding builds the vocabulary by merging the most frequent adjacent pairs.
- Byte-level BPE guarantees any text can be encoded without unknown symbols.
- WordPiece and Unigram are alternative training methods with different rules.
- Token counts drive cost, context limits and some quirks like poor letter counting.
Frequently asked questions
What is a token in AI?
A token is a chunk of text, such as a word, part of a word or punctuation mark, that a language model reads and writes. Each token maps to a number in the model’s vocabulary.
Why do chatbots struggle to count letters in a word?
They process tokens, not individual letters, so the letter-level structure is not directly visible to them.
Is one token equal to one word?
No. In English, 1,000 tokens is roughly 750 words, but rare words, other languages, code and numbers can take many more tokens.
