In the summer of 2023, the same GPT-4 that identified prime numbers correctly 84% of the time in March was, by June, down to a near coin flip, and users noticed before the vendor would admit it.

OpenAI's own product lead publicly denied any decline. The model was not broken in the usual sense, and nothing crashed. What failed was that a shipped product changed behavior between two dates with no instrument in place to catch it, so users became the test suite.

Stanford and UC Berkeley, 2023: GPT-4 prime-number accuracy fell from 84% to 51% between March and June.

Users reported the decline for weeks, and the vendor's public answer was to deny it.

Within days, researchers at Stanford and UC Berkeley published measurements showing the June version scored far lower than the March version on the same tasks. The disagreement was not about opinion, because one side had numbers and the other had a hunch.

The model still ran and still answered. The gap was that no one was measuring whether the answers held.

A better model would not have prevented this, because the gap was never in the model. It was in the missing instrument around it.

The same product, the same question, two months apart.

A 33-point drop on an unchanged task, invisible to anyone without a test set that reran and compared. If your only detector is user sentiment, you learn about a regression last.

Chen, Zaharia, and Zou, 'How Is ChatGPT's Behavior Changing over Time?', 2023.

A demo tests the inputs you thought of, and production sends everything else, so you need an instrument that measures quality before your users do.

An eval is that instrument: a written bar, real cases, graders, and a gate. It turns 'seems worse' into a number you can act on.

Four parts assemble into one instrument you run on a schedule.

"We ship fast and read the complaints. Isn't that our eval?"

Complaints are real signal, but they arrive after the damage, and only from the fraction of users who bother to write in rather than quietly leave. By the time a pattern is clear in the inbox, the regression has been live for days and has already reached the silent majority.

An eval moves that detection before the merge. The same failing input a customer would have hit becomes a case that blocks the change in private, so the regression never reaches production at all.

Each gap in the story maps to a chapter in this part.

The instrument that would have caught GPT-4's drift is the one this part builds.

Every gap in this story is a chapter waiting.

Eight chapters take you from watching the full range production sends to a running eval, a bar, cases, graders, and a gate, that you stand up on one page called the Quality Bar and run weekly against what you ship.