A real incident
OpenAI priced its flagship plan on a guess and lost money on every subscriber; the chapters next door turn cost, speed, and price into decisions you make before that bill arrives.
The chapters
59 min total
- The cost cliff7 minYou see why margin is designed like quality, not discovered in the monthly invoice.
- Cost per task7 minYou cost one finished task across every call, retry, and judge pass it triggers.
- Latency: design the wait7 minYou make a slow answer feel fast with streaming and real progress, no pricier model.
- Routing and cascades8 minYou send each request to the cheapest model your evals admit, and watch the escalation rate.
- Caching: pay once7 minYou serve repeated reads and answers from a cache, and evict them when the facts change.
- Batch and background7 minYou sort jobs by who is waiting and queue the unwatched ones for the batch discount.
- Pricing: match your costs8 minYou charge in a currency that tracks the meter and test margin against your heaviest user.
- The Inference Budget8 minYou assemble every decision into one signed budget per feature, with alarms that page a name.
The sum
By the end you can turn a feature's economics from a monthly surprise into a designed system: you know what one task costs, how long the user waits, which model runs it, what repeats for free, and what the price has to be.
Fill the Inference Budget, one signed page per feature, and you leave with a cost target, a routing bar, a caching plan, and alarms with names on them.
The Inference BudgetFillable PDFDownload →