Skip to content
AI-Native PM

A real incident

A fabricated-citation scandal became the most cited example of an AI knowledge failure, and the chapters next door build the layer that prevents it.

01 / 09
A real incident

A New York lawyer filed a federal brief citing six court decisions that did not exist, each one invented by ChatGPT and quoted as settled law.

The judge sanctioned the lawyers, the case made headlines, and the model was never the broken part. ChatGPT returned fluent text exactly as built; the failure was that nothing checked those citations against a real legal record before they reached a court.

Mata v. Avianca, S.D.N.Y., 2023, sanctions ruling

02 / 09
What happened

He asked the model whether the cases were real, and took the answer as confirmation.

FROM: CHATGPT

Is Varghese a real case? Yes, it is a real case, and it can be found on Westlaw and LexisNexis.

No such case existed.

ChatGPT does not check a legal database. It produced text that looked like a confirmation, and the lawyer treated a generator as a research tool. There was no lookup step between the question and the answer.

03 / 09
What actually failed

The model wrote fluent legal prose. Every decision around its knowledge was missing.

No lookup against a real case database before the citations were used.

No check on whether each cited source was current, or even existed at all.

No grounding test that would have flagged six invented cases before filing.

A better model would not have prevented this, because the gap was never in the model. It was in the knowledge layer nobody built.

04 / 09
The pattern

Even purpose-built legal tools with retrieval still hallucinate.

1 in 6

Lexis+ AI queries returned a hallucination

1 in 3

Westlaw AI-Assisted Research queries did

These are retrieval-augmented products built to look facts up, and they still miss. Bolting retrieval onto a model is not the same as building a knowledge layer you can trust.

Stanford RegLab, Hallucination-Free?, 2024

05 / 09
The one idea

Most wrong answers from an AI product are not the model failing; they are the wrong facts reaching it.

Build the knowledge layer, what your product looks up, remembers, and answers from, and most hallucinations never start.

06 / 09
The knowledge layer

A handful of decisions stand between a question and a grounded answer.

The lookup step

Retrieve the facts before answering, chosen one-shot or agentic by stakes.

The search route

Send each question to keyword, vector, hybrid, or structured lookup.

The memory policy

Remember only what users consent to, and let them see, edit, and delete it.

The grounding gate

Prove each answer rests on a source before the release ships.

07 / 09
The objection worth answering

"Newer models hallucinate less. Won't this fix itself?"

Hallucination rates fall with each model, but they do not reach zero, and the failures that reach production are the ones retrieval was supposed to catch. The tools in the Stanford study had retrieval built in and still returned invented citations.

A stronger model cannot know your latest price, a customer's record, or a document filed this morning. Those facts have to be fetched, and fetching them well is a product decision, not a model upgrade.

08 / 09
What actually fixes it

Each gap in the story maps to a chapter with a fix.

Fabricated sourcesRetrieval tool

Make the product look facts up before it answers, never from memory alone.

Wrong route to the factSearch stack

Route each question to the retrieval method that actually finds it.

Stale or unowned sourcesKnowledge freshness

Give every source an owner, a shelf life, and a recall path.

No proof it was groundedGrounding evals

Gate every release on retrieval hit rate and grounded-answer rate.

Six fabricated cases would not survive a single grounding eval.

09 / 09
From story to part

Every gap in this story is a chapter ahead.

Eight chapters build the knowledge layer end to end, from the first lookup step to a grounding eval that gates your release, and they close with the Knowledge Charter: one signed page recording every decision about what your product knows and remembers.

Start with chapter one