Why AI Invents Things With Total Calm and Confidence

A language model does not store answers. What it stores is a pattern, plus the ability to guess which fragment of text comes next. Every answer comes out of that guess, and guessing has one style: neat, fluent and confident. That is why a fabricated answer still sounds certain. The mechanism has not changed since 2024, even though 2026 models are far more capable than the first ones.

A Language Model Is Not a Filing Cabinet

Imagine someone hands you half a sentence and asks you to finish it. Thousands of endings are equally reasonable. The model does not pick the true one. It picks the ending that shows up most often in the data. For a human, guessing is gambling. For a model, guessing is the whole job.

People picture a language model as a digital library with the answers already on the shelves. Not so. Inside are statistical notes about which sentence tends to follow which other sentence.

If the name you asked about never appears in any newspaper, the model invents one anyway. Not out of deliberate deceit. Because there is no pattern there to repeat. One road is left: guessing.

Its Training Loss Scores Resemblance, Not Truth

Training has one narrow goal: get as close as possible to the real text. The loss function compares the model's output with the data and penalises it for not matching. The data can be wrong, and the data can be censored. What gets measured is still resemblance, never the truth of the world inside.

Picture a cook judged from a photo of the plate. The salt can be heavy or missing, and the judge never knows, because only the colour is graded. The same cook can plate three dishes with different salt levels and still score identically.

Nothing in that process touches the real world, and nothing asks who actually owns that name.

The Distribution Always Has to Produce Something

A model has no option to stay silent. When the context gives too little signal, the distribution still has to return the next fragment, and what gets picked is the most likely one statistically, not the true one.

Picture a kiosk with 150,000 words on its buttons and no way to say it does not know. It always hands over a word: the one that most often follows the last word typed.

The settings users touch most are temperature, top-k and top-p. The first sharpens or flattens the distribution. Top-k keeps the highest-probability candidates. Top-p keeps candidates until their probability clears a threshold. Holtzman's team proposed top-p in 2019, after showing that always taking the top candidate produces flat, repetitive text. It became known as nucleus sampling.

The lower the temperature, the narrower the draw. Text looks tidier and sounds more certain. Tidy is not correct. At maximum confidence the model is least likely to say it does not know.

The First Mistake Grows Into the Second

One kind of hallucination does not come from ignorance. It appears when the model is already wrong once and then writes a defence of that error.

Zhang's team examined this in 2024 with three datasets of 500 questions each, and found something odd: models can recognise their own false claims. Asked separately, GPT-3.5 recognised 67 percent of them, GPT-4 recognised 87 percent, and LLaMA2-70B-chat recognised 94 percent.

The knowledge needed to object is already inside the model. What is missing is a mechanism that uses it. The first error is not dismantled; it is wrapped in a fresh layer of reasons, like someone who answers wrong and then lectures on the topic.

The finding does not disappear with a few settings. Higher temperature, wider beam search and zero-shot chain of thought all leave it in place.

A Bigger Model Is No Guarantee of Fewer Hallucinations

Two research results are often treated as contradictions, though they measure different things. The first, from Kalai and Vempala in 2024, covers facts that cannot be inferred from data. The bound is firm: if 20 percent of birthday data appears exactly once, base models should hallucinate on at least 20 percent of birthdays. Part of the world has no pattern to learn.

The second, posted to arXiv in 2024, points the other way. On a fixed dataset, larger and longer-trained models hallucinate less.

The catch is the price of the last few percent. Cutting the error to 5 percent of the training data needs a model roughly ten times larger, plus far more compute than Hoffmann's team found optimal in 2022. Detection also gets harder as the model grows.

So scale does help, on a curve that is expensive at the end. The first percentage points are cheap, the last are not. Once size stops making hallucinations easy to detect, you lose the last tool for inspecting them.

RLHF and Retrieval Reduce Without Removing

The two fixes people mention most are alignment with human feedback and document retrieval at answer time. Both work. Neither is a cure.

Reinforcement learning from human feedback works like handing back graded papers. A teacher marks the messy answers and the model learns to avoid that shape. The effect is real. Models become less likely to blurt out obvious nonsense.

What disappears is a tendency, not the underlying capability. Closing one door does not empty the warehouse. Researchers in Singapore put it bluntly in their 2024 paper.

Document retrieval works like opening the book before answering, and it does cut hallucination heavily. But the sentences linking two retrieved passages are still written from statistics. The bridge is still invented, only now it is built over facts that are right.

The mistake is assuming one fix closes a hole elsewhere. With no source to check, retrieval just returns an answer with no evidence attached.

Formal Evidence That This Cannot Be Fully Patched

In 2024, Xu, Jain and Kankanhalli at the National University of Singapore published a paper whose title reads like an accusation: hallucination is an innate limitation of large language models. Read the claim carefully, because it is a mathematical proof rather than a measurement of thousands of questions.

The argument is elegant and slightly unnerving. They use a diagonalisation argument from learning theory, first formulated by Cantor. If a model learned every computable function, there would still be one function it misses. There always is.

The consequence is blunt. The result does not depend on architecture, learning algorithm, prompting style or training data. Kalai and Vempala's statistical bound is the sibling result. One speaks the language of computability, the other of probability.

There is a limit worth stating honestly. The proof shows that hallucinations must appear on some set of questions. It does not show that every answer will be wrong. That gap is large, and most circulating arguments skip past it.

No database holds every fact in the world, so the shape of the result still makes sense.

Analysis: The Numbers Behind the Disagreement

Claims about hallucination often sound like two people shouting. One side says 2026 models are much better, the other says nothing has changed. Both can be true, because they measure different things. Here are the figures you can trace to a source.

The read-through is short. The 87 percent figure means the problem is not missing knowledge. The knowledge is there; it is never used to cancel an answer the model has already committed to.

The 20 percent figure points at the limit people forget. Some facts genuinely appear once in the world, and no pattern can be inferred from them. There, a perfect model would still invent.

The last three rows show what was paid. Refusing cuts errors but also usefulness. Thinking longer cuts errors but adds tokens. The menu really is that narrow.

• Every hallucination number depends on how the task is defined.
• Bigger models help, but very slowly at the end.
• All of it points at one thing: inventing is built into the job.

Where Things Stand in 2026

As of 2026 the direction is clear. Models that think longer and use tools do lower the hallucination rate. The GPT-5 system card reports factual claim errors 26 percent lower than GPT-4o.

That reduction cost three things: more tokens, longer answers, and a higher chance of refusing. In the joint safety evaluation published by OpenAI and Anthropic, Claude models refused up to 70 percent of hallucination questions. Its error rate is very low, partly because many questions go unanswered.

On the same evaluation, OpenAI o3 produced more than twice as many fully correct answers as earlier models, paying with a higher hallucination rate.

Those bars are PersonQA hallucination rates from the o3 and o4-mini system card, April 2025. The model that thinks longer misses more often.

There is also detection rather than prevention. Farquhar's team published semantic entropy in Nature in June 2024: ask the same question repeatedly, group the answers by meaning, and treat a wide spread as low confidence.

The most useful caveat comes from the authors: the method does not catch a model that is wrong consistently. The most dangerous failures come from a system that fails confidently and systematically.

So the numbers move, and the answer does not. A 2026 model lies less often than a 2022 model, yet still invents answers where none exist. The question has not changed since 2024: if the most confident system is also the most often wrong, how do you teach it to notice that it is guessing?