Change control: ship prompt and model changes like releases

Code earned its discipline long ago: a review, a test run, a deploy that can be reversed. The pieces of an AI product that steer behavior mostly skip all of that, because they look like text or configuration instead of logic. Why AI products fail on the org chart, not in the model traced these failures to the operating layer; this chapter builds one piece of it, change control: the organization's rule for what counts as a release, and what every release must pass.

The five changes that are all releases

Five kinds of change can alter what your product says or does in production:

• A prompt edit. One sentence retuned the assistant in the opening scene for every user.
• A model swap. Same prompt, new model, different behavior on the edge cases you never wrote down.
• An eval-set change. Adding, removing, or relabeling test cases quietly moves the bar every future change is measured against, which is why Test cases: build the set that samples reality versions the case set.
• A corpus update. New documents in the retrieval store change what the product reads; a routine upload can change answers as thoroughly as a deploy.
• A tool change. Adding a tool, or editing one tool description, changes which actions the model takes and when.

In an AI product, a prompt edit is a release: one changed sentence alters behavior for every user, so it ships the way code ships, through a gate, behind a canary, with a rollback ready.

The release gate, line by line

If you built the gate from The regression gate: no change ships blind, the mechanical layer exists: an eval suite runs on every change, and a failing score blocks the ship. Change control wraps that mechanism in the decisions only the organization can make. For one behavior change, the gate reads:

• The eval suite passes at or above the bar, including the regression cases from past incidents.
• The behavior's owner signs off, by name. The owner map from Ownership: one name on every model behavior says who; a change no one will approve is not ready.
• The canary plan is written: the slice of traffic that sees the change first, and the metrics you will watch while it runs.
• The rollback is written and tested: the last good prompt or model version is addressable by id, and someone has exercised the switch recently.
• A comms note is drafted if the change is user-visible, so support hears about it from you instead of from tickets.

Filed together, the checklist and its results are the release record: what changed, who approved it, what the gate showed, how to roll it back. When behavior drifts weeks later, that record turns an archaeology dig into a five-minute lookup.

A hole in a real gate: the sycophancy release

In April 2025, an update to GPT-4o made ChatGPT conspicuously sycophantic: within days of the rollout, screenshots circulated of the model cheering on bad plans, behavior OpenAI's postmortem described as "validating doubts, fueling anger, urging impulsive actions." OpenAI began the rollback three days later and published two postmortems within about a week.

The postmortems are unusually candid, and they describe a gate that existed and had a hole:

• The offline evals passed, clearing every check the team had written down.
• Small A/B tests said users liked it. Sycophancy inflates exactly the metrics that measure whether users like the answers.
• Expert testers reported the behavior "felt" slightly off, but the numbers won, and it shipped.
• One row was missing from the checklist. In OpenAI's words, "we also didn't have specific deployment evaluations tracking sycophancy."

The behavior traced back to a new reward signal built from thumbs-up and thumbs-down data, which the postmortem says "weakened the influence of our primary reward signal, which had been holding sycophancy in check."

The fixes OpenAI announced were process fixes, not model fixes: treat behavior issues as "launch-blocking" like other safety risks, add an opt-in "alpha" testing phase, and weight expert spot checks more heavily against the A/B numbers. And notice what contained the damage: the previous version was still addressable, and the rollback kept a bad release from becoming a lasting one.

The canary finishes what the evals start

The episode also shows a limit no bigger eval suite removes. An offline suite samples behavior: a few hundred cases stand in for millions of unscripted conversations, and a change can pass every written case and still fail in a pattern nobody thought to write down. So staged exposure belongs to the gate itself: a canary release gives the change to a small slice of real traffic first, internal tenants or a few percent of sessions, while you watch the metrics the change could hurt; the rollback stays armed until the canary has run long enough to earn trust.

Offline evals sample behavior, they cannot exhaust it, so the canary is part of the gate: a small slice of real traffic finishes the test the eval suite started.

The model-swap runbook

Model swaps get their own runbook because providers schedule them for you. Anthropic's docs promise at least 60 days of notice before a public model retires, and requests to a retired model simply fail; OpenAI gave three months of notice before removing GPT-4.5-preview from its API in July 2025. Run the swap as a scheduled release:

• Re-run the entire eval gate on the candidate model, not a convenient sample of it.
• Diff the failures, not the score. A candidate with the same score can fail different cases; the new failures show what you are trading away.
• Canary a slice and watch the same metrics as any behavior change.
• Keep the old model addressable until you trust the new one, and schedule the swap so the canary finishes before the retirement date.

The honest objection: this will slow us to a crawl

If your team improves prompts daily, a gate can sound like a release train, and the objection deserves its strongest form: prompt iteration speed is a real advantage, and a heavyweight process would spend it. So count what the gate costs: minutes of eval compute, one approver reading a diff and a result table, and a canary that runs unattended. The gate removes the unreviewed release, not your speed, and the checklist was never the slow part: the time goes to the public rollback, the pair of postmortems, and the news coverage that follows. OpenAI moved to launch-blocking reviews only after its incident; adopting the rule before your own is cheaper.

Try it now

This drill takes about 15 minutes and produces the release gate for one real change.

Pick one pending behavior change: a prompt tweak in review, a coming model swap, or a batch of documents waiting for the corpus.

Write the eval check. Name the suite or slice that must pass and the bar it must clear. If you have no suite yet, list the ten cases you would run by hand and note the gap.

Name the approver. From your owner map, write down who signs off on the behavior this change touches, and confirm they agree.

Write the canary and the rollback. Record the slice that sees the change first, the two metrics you will watch, the exact id of the last good version, and the step that puts production back on it. Scale it down: if a full suite run costs real money, gate this one change on your smoke slice and file the full run as a follow-up.

Keep the finished page. The incident runbook you write in Incident response: when your product says the wrong thing in public reuses its rollback step word for word, and your Operations Charter files the gate as standing policy.

Chapter Summary

• Prompt edits, model swaps, eval-set changes, corpus updates, and tool changes all alter production behavior, so each one is a release, however small the diff looks.
• A behavior change ships through a gate: the eval suite at its bar, sign-off from the behavior's owner, a written canary plan, a tested rollback, and a comms note if users will notice.
• File the gate's results as the release record: what changed, who approved it, what the gate showed, and how to roll it back.
• The GPT-4o sycophancy rollback shows a gate can exist and still have a hole: offline evals and A/B numbers looked good, testers who reported the model "felt" off were outweighed, and no deployed eval tracked the failure that shipped.
• The honest fix is organizational: add the missing check, and give expert review real weight against the metrics.
• Offline evals sample behavior rather than exhaust it, so a canary slice of real traffic is part of the gate, and the rollback stays armed until the canary earns trust.
• Providers retire models on their own calendars, so treat every swap as a scheduled release: full gate, failure diff, canary, old model addressable until you trust the new one.
• The gate costs minutes per change; shipping blind costs public rollbacks and postmortems.
• Your product's behavior also depends on everything it is allowed to read, and that store needs governing next: Govern the knowledge: what your product is allowed to read.

Sources

• OpenAI (2025). Sycophancy in GPT-4o: What happened and what we're doing about it. April 2025.
• OpenAI (2025). Expanding on what we missed with sycophancy. May 2025.
• Anthropic. Model deprecations, Claude API documentation: 60-day minimum retirement notice for public models, with dated deprecation and retirement tables (last verified July 2026).
• OpenAI. Deprecations, API documentation: gpt-4.5-preview deprecated April 14, 2025 and removed from the API July 14, 2025 (last verified July 2026).