People who use AI chatbots every day still carry a handful of mistaken ideas about how they behave: that hallucinations are a bug waiting for a patch, that a long context window means perfect recall, that a chatbot “remembers” you the way a colleague does. Our earlier piece on seven common myths about AI covered the broad misconceptions. This one goes narrower and deeper, taking five myths about how chatbots work and checking each against published research. Where a claim depends on a particular product, we say so, because behavior differs from system to system and changes between versions.
Myth 1: “Hallucinations are just a bug that better engineering will remove”
A hallucination is a fluent statement that is not true. It is tempting to treat it as a defect, like a crash, that will eventually be fixed. A 2025 paper, “Why Language Models Hallucinate” by Adam Tauman Kalai, Ofir Nachum, Santosh Vempala and Edwin Zhang, argues the picture is more structural. The authors link hallucinations to the way models are trained and evaluated. During pretraining, a model learns to produce plausible text, and for facts that appear rarely in the data (a single mention of an obscure person’s birthday, for example) there is no pattern to learn, so a confident guess is the natural output. During later evaluation, most benchmarks give credit for a correct answer and nothing for saying “I don’t know,” which rewards guessing over abstaining.
The practical reading is that hallucination can be reduced but not wished away. Retrieval from trusted documents (see our guide to how retrieval-augmented generation works), training for calibrated uncertainty and grading schemes that penalize confident errors all help. None of them makes a system that predicts text incapable of producing a wrong sentence. Treat any claim from a chatbot that matters, such as a citation, a dosage or a legal reference, as something to verify.
Myth 2: “A bigger context window means the model uses everything you give it”
Context windows now reach hundreds of thousands of tokens or more, and it is natural to assume the model attends to every word equally. Research suggests otherwise. In “Lost in the Middle,” Nelson Liu and colleagues tested language models on question answering and key-value retrieval with long inputs and varied where the relevant information sat. Performance was often highest when the answer appeared at the very start or end of the input and degraded when it was buried in the middle, even for models built for long contexts. The paper was later published in the journal Transactions of the Association for Computational Linguistics.
The finding does not mean long contexts are useless, and newer models may behave differently from the ones tested. It does mean that pasting a whole archive into a prompt and expecting perfect recall is a bet, not a guarantee. Practical habits follow: put the most important instructions and documents at the beginning or end, trim irrelevant material, and test with your own data. We make a similar point in our quick take on why bigger context windows are not always better.
Myth 3: “The chatbot remembers our conversation like a person does”
Inside a single conversation, the model has no memory in the human sense. The underlying network is stateless: each reply is produced from the text it is handed at that moment, which the application assembles from your earlier messages. If the conversation grows beyond the context window, older parts must be dropped or summarized, and the model genuinely cannot see them. The architecture that makes this work, described in “Attention Is All You Need,” lets each token in the input be weighed against every other token in the window, but only the window. There is no hidden diary underneath. Our explainer on how transformers work walks through the mechanism.
Some products add memory features on top, storing notes about you in a separate system and inserting them into later prompts. That is an application design choice, and what is stored, for how long and whether you can inspect or delete it varies by vendor. Check the product’s settings and privacy documentation rather than assuming either extreme: it neither remembers everything nor forgets everything by default.
Myth 4: “When a model shows its reasoning, that is exactly how it reached the answer”
Chain-of-thought prompting, asking a model to think step by step, often improves accuracy, and many products now display reasoning text. It is easy to read that text as a faithful transcript of the computation. The paper “Language Models Don’t Always Say What They Think” by Miles Turpin, Julian Michael, Ethan Perez and Samuel Bowman tested this directly. The authors added biasing features to prompts, such as reordering multiple-choice options so the correct answer was always “A.” Models were swayed by the bias, but their written explanations rarely mentioned it, instead offering plausible-sounding justifications. The study covered particular models available at the time, so it is evidence of a risk, not a verdict on every current system.
The takeaway: a reasoning trace is useful evidence and often a good debugging aid, but it is not a guaranteed window into why a model answered as it did. If a decision matters, check the conclusion independently.
Myth 5: “Chatbots are trained to be correct”
Base language models are trained to predict the next token in text. The step that turned such models into helpful assistants was instruction tuning and reinforcement learning from human feedback, described in OpenAI’s InstructGPT paper, “Training language models to follow instructions with human feedback.” In that work, human labelers wrote example responses and ranked model outputs, and the model was tuned toward the responses people preferred. The authors reported that their smaller tuned models were preferred by human raters over a much larger untuned model, and also noted that tuned models still made simple mistakes.
The detail matters: the training target is “what humans rate highly,” which overlaps with correctness but is not identical to it. Answers that sound confident, complete and agreeable can score well even when wrong. This is one reason developers now also train on verifiable tasks and publish evaluations of factuality, and why independent testing of claims matters.
How to use this in practice
- Verify anything consequential, especially names, numbers, quotations and citations.
- Place critical information at the start or end of long prompts, and keep prompts lean.
- Check what memory features a product offers and where the data goes.
- Use reasoning traces as hints, not proof.
- Prefer systems that can cite sources you can open, and open them.
What to watch next
- New evaluation methods that reward calibrated uncertainty instead of confident guessing.
- Long-context benchmarks that test recall across the whole window, not just the ends.
- Clearer controls and disclosures for chatbot memory features.
- Research on whether displayed reasoning can be made more faithful.
- Vendor documentation that states known limitations plainly.
Key takeaways
- Hallucination is linked to how models are trained and scored, so it can be reduced but not guaranteed away.
- Long context windows do not guarantee equal use of every part of the input.
- A chatbot’s in-conversation “memory” is just the text it is given each turn; product memory is a separate feature.
- Displayed reasoning may not reflect the real cause of an answer.
- Tuning for human preference is not the same as tuning for truth.
Frequently asked questions
Do newer models still hallucinate?
Yes, though rates vary by model and task, and retrieval and better training have reduced them in many settings. Independent evaluations are the best guide for a specific use.
Should I put important instructions at the top or bottom of a long prompt?
Research on long contexts suggests the beginning and end are safer than the middle. Test with your own material, since results differ between models.
Is it safe to trust a cited source from a chatbot?
Only after you open it. Models can produce citations that look right but do not exist or do not support the claim, so check each link yourself.
