Why AI Chatbots Confidently Make Things Up
A lawyer in New York once submitted a legal brief filled with citations to court cases that sounded entirely plausible — correct case names, correct years, correct legal language. The problem was that none of the cases existed. A chatbot had invented them, and it had done so with the same fluent confidence it used for everything else.
This is not a rare glitch. Researchers now know that fabricated citations have slipped past expert reviewers at one of the world’s most prestigious AI conferences, appearing in dozens of published papers despite review by multiple specialists per submission. Government reports have needed corrections. Consulting firms have issued refunds. The pattern is always the same: an AI system states something false with the same tone of certainty it uses for something true.
The natural question is why. Why would a system trained on enormous amounts of accurate information so often produce confident nonsense? For years, the standard answer was vague — “it’s just how AI works” or “the data must have been bad.” But recent research, including a detailed 2025 study from OpenAI, offers a far more precise explanation. Hallucination is not a mysterious malfunction. It is a predictable consequence of how these systems are built and, just as importantly, how they are graded.
What “Hallucination” Actually Means
In AI research, a hallucination is a statement that sounds fluent and plausible but is factually wrong or entirely fabricated. The term can be misleading, since it suggests a perceptual glitch, like a person seeing something that isn’t there. A language model doesn’t perceive anything. It generates text by predicting, one word at a time, what is statistically likely to come next given everything it has processed so far.
That distinction matters. A hallucination isn’t the system “lying,” because lying requires knowing the truth and choosing to say otherwise. It also isn’t quite an “error” in the way a calculator error is an error, since the model has no internal ledger of facts it consults and misreads. It is closer to a highly capable autocomplete producing a smooth, confident continuation that happens not to correspond to reality.
The Classic Explanation: Noisy Data and Statistical Guessing
Before 2025, most explanations for hallucination pointed to three familiar culprits.
The first was training data. Large language models learn from enormous collections of text scraped from the internet, and that text inevitably contains errors, outdated information, and outright falsehoods. A model trained on flawed material will sometimes reproduce those flaws.
The second was the long tail of knowledge. Research on how these models absorb information found that facts mentioned rarely in the training data — an obscure historical date, a minor scientific detail, a little-known person’s biography — are learned far less reliably than facts repeated constantly across the internet. A model can be extremely confident about common knowledge and shaky on rare knowledge, without any obvious signal to the user about which is which.
The third was architecture. Language models generate text sequentially, one token at a time, without the ability to step back and verify an entire claim before committing to it. A small early error in a sentence can snowball, since the model keeps building on what it already wrote rather than checking it against outside reality.
These explanations are all still valid. But they treat hallucination as an unfortunate side effect — something that happens despite the training process. The newer research argues something more unsettling: hallucination happens because of the training process, in a way that is difficult to fully eliminate without changing how these systems are evaluated in the first place.
The Deeper Mechanism: Why Guessing Beats Admitting Uncertainty
The 2025 OpenAI paper, written by researchers including Adam Tauman Kalai and Santosh Vempala, reframes the entire problem in statistical terms. The researchers argue that hallucinations arise from the way most evaluations are graded, since language models are optimized to be good test-takers, and guessing when uncertain improves test performance. arXiv
Think about how students behave on a multiple-choice exam with no penalty for wrong answers. A cautious student who leaves difficult questions blank will score lower than a confident guesser who fills in every bubble, even guessing at random. Over thousands of questions, guessing wins. Language models are trained and evaluated in a strikingly similar way. Like students facing hard exam questions, large language models sometimes guess when uncertain, producing plausible yet incorrect statements instead of admitting uncertainty. arxiv
The researchers go further, framing the core problem as a kind of built-in statistical trap. They argue that hallucinations need not be mysterious, since they originate simply as errors in binary classification: if incorrect statements cannot be distinguished from facts, then hallucinations in pretrained language models will arise through natural statistical pressures. In plain terms, a model has to decide, word by word, whether a given continuation is valid. When the difference between a true statement and a false-but-plausible one is subtle — which it often is for obscure facts — some rate of confident error becomes mathematically unavoidable, not just likely. arxiv
This reframing changes where the responsibility lies. It isn’t only that the internet’s text is messy. It’s that the scoring systems used to train and rank these models — benchmarks, leaderboards, human preference ratings — nearly all reward a confident, specific answer over an honest “I’m not sure.” A model that says “I don’t know” a lot will look worse on paper than a model that guesses fluently, even if the guesser is wrong more often.
Why Simply “Being Smarter” Doesn’t Fix It
A common misconception is that hallucination is a temporary limitation that will disappear as models become more capable. There is real progress: newer models produce noticeably fewer fabrications, especially when they reason through a problem step by step. But more capability does not automatically mean more honesty about uncertainty — and it can even work against it.
Recent research examining advanced “reasoning” models has found that some of these systems hallucinate more on certain tasks, not less, as they generate longer chains of intermediate reasoning. Each additional inferential step is another opportunity to drift from a well-supported claim toward a plausible-sounding invention, particularly when the model is rewarded for producing a complete, decisive-looking answer rather than an honest, partial one. Separate work on unanswerable questions has found that even highly capable reasoning models often fail to recognize when a question has no good answer at all, preferring to manufacture one instead.
This is the uncomfortable core of the problem: raw intelligence and honest calibration are different properties. A system can become extraordinarily good at producing convincing text while remaining no better — or even worse — at knowing the boundary of its own knowledge.
The Real-World Cost of Confident Fabrication
Hallucination is easy to dismiss as a curiosity until it intersects with domains where accuracy matters. Academic peer review offers a striking case study. A 2026 analysis of the 2025 NeurIPS conference — one of the most competitive and rigorously reviewed venues in artificial intelligence research — found roughly one hundred fabricated citations that had slipped past expert reviewers, appearing in dozens of accepted papers despite review by several specialists per submission. A related check of papers submitted to another major AI conference found similar fabricated references in papers that reviewers had rated highly.
Citations are a particularly vulnerable target for hallucination because they demand precise, specific details — author names, publication years, journal titles — that a model can blend and recombine into something that looks exactly like a real reference without corresponding to any actual paper. The same pattern shows up in legal filings, government documents, and professional consulting reports, several of which have required formal corrections or financial refunds after AI-generated fabrications were discovered too late.
What makes these cases genuinely dangerous is not that the AI was wrong. Humans are wrong constantly. It’s that the wrongness carried no signal. A hesitant human expert hedges, qualifies, or visibly struggles when unsure. A hallucinating model does none of that. It produces the fabricated citation in the same fluent, assured register as the accurate one beside it.
What Can Actually Reduce Hallucination
If the core problem is an incentive structure that rewards guessing, the most direct fixes target that incentive structure rather than the model’s raw capability. Researchers have proposed changing how models are scored during training and evaluation, giving explicit credit for calibrated uncertainty — a model that says “I don’t have reliable information on this” when appropriate would be rewarded rather than penalized for it.
Outside of training changes, several practical techniques already reduce hallucination in real applications. Retrieval-augmented generation connects a model to an external, verifiable source — a database, document set, or live search — and asks it to ground its answer in that retrieved material rather than relying purely on its internal, compressed memory of training data. Fact-checking layers can flag claims for verification before they reach a user. Fine-tuning on domain-specific, carefully curated data can shrink the “long tail” problem within a narrow field, such as medicine or law, even if it doesn’t solve the general case.
None of these are complete solutions. A model can still hallucinate confidently about the very source material it was supposed to be grounded in, and no evaluation system has fully solved the deeper tension between rewarding correct guesses and rewarding honest uncertainty. Reducing hallucination remains a matter of degree, not elimination.
Why This Matters Beyond the Lab
The deeper implication of this research is a shift in how people should think about AI-generated text altogether. A chatbot’s confident tone is not evidence of accuracy. It is, at best, a reflection of how fluent and statistically probable a piece of text sounded during training — a property that has surprisingly little to do with whether the underlying claim is true.
This has consequences for anyone using these systems for research, writing, or decision-making. Verification cannot be treated as an optional final step reserved for suspicious-looking claims, because hallucinations are specifically designed — by the mechanics of how these models are trained — to be indistinguishable in tone from accurate statements. The NeurIPS case is instructive precisely because the reviewers were experts actively looking for problems, and the fabrications still passed.
The Bottom Line
AI systems don’t hallucinate because they are broken. They hallucinate because the entire pipeline that creates them — vast, imperfect training data; sequential, uncheckable generation; and evaluation systems that reward confident answers over honest uncertainty — makes a certain rate of fluent, false statement close to inevitable. The 2025 research reframes hallucination not as a bug waiting to be patched, but as the predictable output of optimizing for confident test performance rather than calibrated honesty.
Fixing that will require more than bigger models. It will require deciding, as a matter of design and incentive, that an AI system saying “I don’t know” is a feature worth rewarding — not a failure to be trained away.
Frequently Asked Questions
Is hallucination the same as an AI “lying”?
No. Lying requires knowing the truth and deliberately stating something else. A language model has no internal fact-check step; it generates the statistically most plausible continuation of text, which sometimes happens to be false.
Do more advanced AI models hallucinate less?
Generally yes for straightforward factual questions, but not uniformly. Some research on advanced reasoning models has found they can hallucinate more on certain tasks, since longer chains of reasoning create more opportunities to drift from well-supported claims.
Can hallucination be completely eliminated?
Current research suggests full elimination is unlikely given how these models are trained and scored. Techniques like retrieval-augmented generation and better calibration training can meaningfully reduce it, but not remove it entirely.
Why do AI models hallucinate more about obscure topics?
Facts that appear rarely in training data are learned far less reliably than facts repeated constantly across the internet, so the model has less statistical grounding to draw on for rare or specialized information.
How can I protect myself from AI hallucinations?
Treat confident-sounding AI answers, especially citations, statistics, and specific factual claims, as needing independent verification — particularly in academic, legal, medical, or professional contexts where accuracy carries real consequences.
