A context window is the total amount of text an AI language model can take into account at one moment, including your messages, any documents you paste in and the model’s own replies. It is measured in tokens, which are small chunks of text, and it works like short-term working memory rather than a stored archive. When a chat grows beyond what the window can hold, or when the useful detail gets buried in a lot of text, the assistant can seem to forget what you said earlier. This guide explains how that works and what you can do about it.
The simple picture: a desk, not a library
Anthropic’s documentation describes the context window as all the text a model can reference when it writes a response, including the response itself. It also stresses that this is different from the large body of data the model was trained on. The documentation calls it a “working memory.”
A useful way to picture it is a desk. The model learned general knowledge during training, a bit like having read a huge library long ago. But when it answers you, it can only look at the pages lying on the desk. Everything in this conversation sits there; an earlier chat from last week is not visible unless the app brings it back.
What is a token?
Models do not read letters or whole words directly. They split text into tokens, which can be a whole short word, a piece of a longer word, a number or a punctuation mark. Context windows are counted in tokens, not words. Long or unusual words are often split into several tokens, and the exact split depends on the model. See also our glossary entries for token and context window.
How a conversation fills the window
Here is the part that surprises many beginners. A chatbot does not keep a private memory of your chat between messages. According to Anthropic’s documentation, each turn of an API conversation includes all the previous conversation history plus the new message, and the model’s reply then becomes part of the input for the next turn. Everything in the request counts: the system instructions, every message, attached documents and images, and the output the model writes.
So a long chat is really a growing pile of text that is sent again each time. Here is a worked example with made-up round numbers, only to show the idea. Imagine a tiny window of 1,000 tokens.
| Turn | What is added | Tokens on the desk |
|---|---|---|
| 1 | Your question (100) and the reply (200) | 300 |
| 2 | Your pasted article (400) and the reply (150) | 850 |
| 3 | Your follow-up (50) and a reply that needs 200 | 1,100, which is over the limit |
At turn 3 the pile no longer fits. Real windows are far larger: Anthropic’s documentation lists up to one million tokens for its newest models and 200,000 for others. The principle is identical: there is a ceiling.
What happens when the window is full
The answer depends on the product.
- Through an API: if the input alone is already too large, Anthropic’s documentation says the request returns an error (“prompt is too long”). Developers can also use compaction, which summarizes earlier parts of the conversation automatically.
- In some chat apps: the documentation notes that chat interfaces such as claude.ai can manage the window on a rolling “first in, first out” basis. In plain terms, the oldest material may drop off the desk to make room for the new.
This is why a chatbot may ignore the rules you set at the start of a very long session. The instruction was not deleted from its brain; it is no longer on the desk, or a summary left out the detail you needed.
Bigger window, better memory? Not always
A huge window helps, but it does not remove the problem. Anthropic’s documentation warns that more context is not automatically better: as the token count grows, accuracy and recall degrade, a phenomenon it calls context rot.
Research points the same way. A 2023 paper, “Lost in the Middle” by Nelson Liu and colleagues (a preprint on arXiv), tested language models on questions that required finding information in long inputs. Performance was typically highest when the relevant information was at the beginning or end and dropped significantly when it sat in the middle, even for models built for long contexts. Where you put information matters. Our short opinion piece, bigger context windows are not always better, makes the same point.
Five habits that help
- Put the key instruction near the end of a long prompt, or repeat it, so it is not lost in the middle.
- Start a fresh chat for a new topic. Old, unrelated history takes up space.
- Summarize and restart. Ask the assistant to write a short summary of decisions so far, then paste it into a new chat.
- Share only what is needed. Paste the relevant section of a document instead of the whole file. Our guide on summarizing long documents with AI shows practical ways to do this.
- Restate important facts such as names and numbers in long conversations.
What to watch next
- Whether new models keep accuracy high across their whole window, not only the size printed on the box.
- How chat apps handle automatic summarizing and memory features, and how clearly they tell you what was kept.
Key takeaways
- A context window is the amount of text, counted in tokens, that a model can consider at once, including its own replies.
- It acts like working memory or a desk, not a permanent record of everything you have ever said.
- A long chat is resent as a growing pile of text each turn, so it eventually hits the limit.
- Even inside the limit, accuracy can fall as the text grows, and details in the middle are the easiest to miss.
- Short, focused chats, summaries and well-placed instructions reduce forgetting.
Frequently asked questions
What is a context window in simple terms?
It is the amount of text an AI model can look at while it writes a reply, including your messages, pasted documents and its own earlier answers. Think of it as the model’s working memory for the current conversation.
Why does ChatGPT or Claude forget things I said earlier?
The model only works with the text currently in its context window. If a chat gets too long, older messages may be dropped or summarized, and details buried in a lot of text can be overlooked even when they are still there.
Does a bigger context window mean the AI remembers everything?
No. A larger window lets you include more text, but Anthropic’s documentation says accuracy and recall can degrade as the token count grows. Organizing what you include still matters.
