A tokenizer splits text into tokens from a fixed vocabulary; in English, 1,000 tokens is roughly 750 words. Model limits and prices are measured in tokens, with separate rates for input and output.

Languages with less training data often need more tokens per word, making them slower and more expensive to process.