Why AI makes things up, and how to catch it

AI models make things up because making things up is the same operation as getting it right. A language model predicts what text should come next. When the answer is well represented in what it learned, the prediction is correct. When it is not, the model does not stop, because nothing in the machinery knows the difference. It produces the most plausible continuation, which is a sentence shaped exactly like a true one.

That is why this is not a bug awaiting a fix. It is the cost of the method. Models have got much better at it and will keep getting better, and none of that changes the underlying situation: there is no moment inside a model where it consults something and finds nothing there.

Two different failures that look identical

Almost everything called a hallucination is one of two quite different problems, and telling them apart changes what you do about it.

Invention. The model produces something that never existed: a court case, a citation, a function in a library, a clause in a contract. This is the famous one, and it clusters in a predictable place. Anything that looks like a reference is at risk, because references have a strong shape and shape is what these models are good at. A fabricated case name looks precisely like a real one, because the model learned the pattern of case names.

Staleness. The model tells you something that was true. The rate changed, the API was deprecated, the pricing page moved, the threshold went up. This is far more common than invention and much harder to spot, because it is not nonsense, it is just out of date. A model trained on text up to some point will describe the world as of that point, in the present tense, without a hint that it is doing so.

The second one causes most of the real damage, and it is the one people are least on guard for.

Why it never sounds unsure

There is no confidence signal in the output. A model that is reconstructing something it saw a thousand times and a model that is producing a fluent guess are running the same process, and the text reads the same either way.

Worse, the places where a model is most likely to be wrong are places where it has learned the form very well. Legal citations, API signatures, statistics, dates. It has seen thousands of examples of the shape, so it produces the shape flawlessly and fills it with whatever fits. The fluency of the output is uncorrelated with its accuracy, and human beings are extremely bad at not reading fluency as competence.

When a model does hedge, that is a trained behaviour rather than a readout of an internal uncertainty. Useful, but do not mistake it for the model knowing what it does not know.

Where it goes wrong most

From use rather than theory, the risk clusters in a few places:

• Anything numeric and specific. Prices, limits, thresholds, dates, version numbers. Highest failure rate of anything.
• Anything cited. Papers, cases, standards, documentation pages. If it has a reference format, treat it as unverified.
• Anything recent. The model's training has an end. Events after it either do not exist or have been half-absorbed from later sources.
• Anything niche. The less written about a subject, the more the model interpolates. Obscure tools and small jurisdictions are where invention concentrates.
• Anything you asked leadingly. Ask "what are the three main criticisms of X" and you will get three, whether or not there are three.

Conversely, it is reliable at: explaining a well-known concept, restructuring text you supplied, translating between registers, writing code in a popular language that you can run, and summarising a document you pasted in. Notice that most of those involve you providing the material.

A method that actually works

Not "be careful", which is advice nobody can act on. Four habits, in order of how much they buy you.

Give it the source instead of asking it to remember. This is the single biggest improvement available and it is not close. A model summarising a document you pasted in is doing something it is genuinely good at. The same model recalling that document from training is doing something it is bad at. Paste the terms, the standard, the documentation page. The failure rate falls off a cliff.

Ask it to quote. "Quote the sentence that says that." A model working from a real source can. A model that reconstructed the claim will produce a quote that is not in the document, and now you can see the problem, which you could not before.

Check every reference, without exception. If a name, case, paper or documentation page is going to matter, open it. This takes a minute and catches the entire invention category. There is no shortcut and there is no model good enough to skip it.

Ask the same thing twice, in separate conversations. Facts the model actually has come back stable. Reconstructions wobble. Two different answers to the same factual question means neither is reliable, which is genuinely useful to know in ten seconds.

What does not work: asking the model whether it is sure. It will produce a confidence statement shaped like the rest of the answer, and it carries no more information.

The staleness problem, and what people do about it

Giving the model the source works. The difficulty is that the sources keep moving, and nobody maintains the copies.

The usual pattern is familiar to anyone who has tried it. You save the documentation as a PDF, attach it to the conversation, get a good answer. Next week, new conversation, attach it again. Three months later the API has changed and your saved copy has not, so you are back to a confident answer about a version that no longer exists, except now it looks verified because there was a document attached.

A few ways people handle it, honestly compared:

• Re-fetch each time. Assistants that can browse will fetch the page when asked. Free, no setup, and unreliable at exactly the wrong moment: many documentation sites block automated fetches, and coding tools frequently cannot reach the web at all.
• Keep a folder of saved pages by hand. Works, and decays. The problem was never the saving, it was the remembering to re-save.
• A documentation index the tool reads. Several services now maintain up-to-date documentation for popular libraries and expose it to coding assistants. Excellent when your source is a popular library, no help when it is your own handbook, a supplier's terms, or a niche API.
• A capture that re-runs itself. Point something at the sites you depend on, have it save them on a schedule into a folder or repository your assistant already reads. This is what I built Site to PDF to do, so treat that as a disclosure rather than a recommendation. It is a Chrome extension, so it needs the browser running, which is the honest limitation.

Which of those is right depends entirely on whether your sources are popular public documentation, in which case an index is easier, or your own scattered set of pages, in which case something scheduled is the only thing that stays current without you tending it.

Why giving it the source works, in one paragraph

The technique above has a name, retrieval, and understanding why it helps tells you where it stops helping.

When you paste a document in, the model is not recalling anything. The text is right there in what it is reading, so producing an accurate summary is a comprehension task rather than a memory task, and comprehension is what these models are genuinely good at. When you ask without the document, the model has to reconstruct it from statistical traces of everything similar it ever saw, which is a different and much worse task wearing the same clothes.

That is also the limit. Retrieval fixes the recall problem. It does nothing for the reasoning problem, which is the next section.

What cannot be fixed by any of this

Retrieval fixes the staleness problem. It does not fix the reasoning problem. A model given the right document can still draw the wrong conclusion from it, and will state that conclusion just as confidently. Every method above raises your odds; none of them turns the output into something you can act on without reading.

The practical stance that follows: use AI to find out what to check, not as the thing you checked. That single reframe removes most of the risk from most of the uses, and it costs you very little, because finding out what to check was the slow part anyway.

If you want the machinery underneath this, how large language models work covers prediction, training data and why the cutoff exists. For the related question of whether the tools claiming to detect AI writing are any good, the short answer is no, and that article is coming next in this guide.