Skip to content
AI-Native PM
Field Notes

Field Note

When Fable 5's Great Idea Cost Us a Week's Tokens

Fable 5 offered to test our new Claude Skills in a way no earlier model had, and the test spent most of a week's model allowance. What it suggested, why it cost so much, and the habits we keep from it.

· 6 min read

We were using Claude, running on Fable 5, to write three small skills for our own use: one that audits a week of meetings, one that works out which chores are worth outsourcing, one that turns a household fix-it list into a brief a handyman can quote. A skill is a packaged set of instructions an assistant loads when a request matches its purpose. The three were written, tested by hand, and ready for a public repo. Then Claude offered something we had not asked for and had never been offered before: a measured test of each skill's trigger description, the one paragraph that controls whether the skill ever opens. It said the test would take "a few dozen small Claude calls in the background over tens of minutes." We said yes.

The test went on to spend most of a week's model allowance on the highest usage tier and filled the session's context several times over. This note covers the four parts that matter: the suggestion, which was genuinely good; the bill; why the bill was so far off the estimate; and how it was shut down. The failure taught us more than the skills did.

The suggestion was genuinely good

An installed skill is not loaded whole. The assistant loads only the skill's name and its one-paragraph description, and whether the skill opens rests on that paragraph alone; Anthropic's launch note describes Claude scanning "available skills to find relevant matches" and loading the rest only on a match. The paragraph can fail two ways. Someone types "my week is nothing but meetings, help," and the meeting-audit skill stays shut. That is the common failure, the silent miss. Or someone types "schedule a meeting with Sam," and the audit opens for a request that was never for it, the barge-in. Our skills were going to strangers, who phrase things their own way, so that paragraph mattered, and we had never tested one.

How a skill opens, and how its description failsLeft panel: what is loaded up front. A small card shows the skill name meeting-audit and its one-paragraph description; below it, a dashed box labeled the rest of the skill, which loads only after the description matches. Right panel: how the paragraph fails, in two directions. The silent miss, in clay: a request it should catch opens nothing. The barge-in, in gold: it opens on a request meant for something else. Caption: the description is all the assistant has loaded when a skill opens or stays shut.HOW A SKILL OPENSLOADED UP FRONTmeeting-auditAudit a calendar and reclaim the hours.Use when someone wants fewer meetings…THE REST OF THE SKILLloads only after the description matchesHOW THE PARAGRAPH FAILSThe silent miss“my week is nothing but meetings, help”should open the skill, and nothing opensThe barge-in“schedule a meeting with Sam”opens a skill that was never meant for itThe description is all the assistant has loaded when a skill opens or stays shut.

Fable 5's plan was easy to follow: write about twenty realistic prompts per skill, half that should open it and half tricky near-misses that should not; let us edit the list, since we define what should fire; then run a background loop that asks a fresh Claude each prompt several times, counts how often the skill opened when it should have, rewrites the description to fix the misses, and repeats. Some prompts are held back from the tuning so the rewrite cannot simply memorize the test.

To be clear about the good part: this is an eval, offered without being asked, for something the model itself had just written, aimed at the exact failure that breaks shared skills. A year of similar work on Opus never produced an offer like it. We approved it in one line.

What it actually cost

The proposal said a few dozen small calls. What we got was millions of tokens and, by the time we stopped it, more than three quarters of the week's allowance on the highest usage tier. We asked what had happened. The answer was straightforward, and it came down to multiplication.

Why: every test was a full session

Each test prompt was not a small call. It was a complete session in our real environment: the system prompt, the project instructions, the memory files, and the connected tools all load before the first word, about 30,000 tokens per prompt. (Tokens are the units of text a model reads and writes, and the unit it bills in.) Worse, a prompt that should not open the skill only finishes after the test model answers it in full: it writes the whole meal plan before the harness can record that nothing fired.

The tune-up arithmeticAt the top, the pitch, struck through: a few dozen small Claude calls in the background. Below it, four multiplied factors: one prompt is a full session with about thirty thousand tokens loading first, times sixty prompts per round, times five rounds per skill, times three skills in the plan. They resolve into a wide clay-bordered total: roughly three hundred full sessions and ten to fifteen million tokens per skill, the session's own estimate after we asked what happened. Caption: none of the numbers is scary on its own; multiplied together, they were the bill.THE PITCH, THEN THE ARITHMETICthe pitch: “a few dozen small Claude calls in the background”1 prompta full session:~30k tokens load first60prompts per round:36 tune, 24 held back5roundsper skill3skillsin the plan×××≈ 300 full sessions and 10 to 15 million tokens, per skillthe session's own estimate, after we asked what happenedNone of the numbers is scary on its own; multiplied together, they were the bill.

Each round ran sixty prompts, each skill was set for five rounds, and there were three skills. By the session's own later estimate, that comes to roughly 300 full sessions and 10 to 15 million tokens per skill. Its words on the cause: "That's the tool's design, not a fixable bug." None of the numbers is scary on its own; multiplied together, they were the bill. Our own chapter The bill of materials: cost the task, not the call teaches exactly this, and we still said yes without doing the multiplication, and without setting any cap that would stop the run on its own.

The score was flat the whole time

There was a second problem: the test could not measure what it was tuning. Recall, the share of should-open prompts that actually opened the skill, sat between 6 and 17 percent and barely moved across five rewrites of the description.

Recall across five rewrites of the descriptionA line chart. The vertical axis is recall from zero to one hundred percent. Five points, one per rewrite of the description, sit at nine, six, seventeen, eleven, and twelve percent, inside a shaded clay band marking the six to seventeen percent range. A dashed gold line far above them marks a score worth iterating on. Caption: a flat line means the test cannot tell the rewrites apart, and more rounds just buy noise.WHAT FIVE REWRITES BOUGHTrecall, the share of should-open prompts that opened the skill0%25%50%75%100%a score worth iterating onrecall stayed between 6% and 17%rewrite 1rewrite 2rewrite 3rewrite 4rewrite 5A flat line means the test cannot tell the rewrites apart; more rounds just buy noise.

A flat score does not mean every rewrite was equally bad. It means the measurement could not tell them apart. The harness tested the skills in a stripped-down, one-shot command-line setup where they competed with more than forty other tools, and that is not where the skills actually run, which is a desktop app, properly installed, in the middle of a real conversation. The session's own summary: "the optimizer is climbing a flat hill: more iterations buy noise, not signal." This is the rule Graders: deterministic, judges, and humans teaches: before a test scales, it has to be able to tell candidates apart, and ours could not.

How it ended

When we asked what happened, the reply did not defend the run. It laid out the arithmetic above, named the design as the cause, and put a number on continuing: about ten million more tokens for the third skill, spent on a question the measurement could not answer. Then it offered a way to stop without losing everything: write up the findings, keep the sixty test prompts for cheap spot checks in real chats, and copy the best lines from the rewritten descriptions by hand. We said abort. One of those lines is live today: the handyman skill's description now says to stay shut when someone plans to do a single repair themselves.

A more capable model makes bigger and better suggestions; it does not watch what they cost. That stayed our job the whole time.

What we do differently now

  • Ask for the total, not the plan. A background loop is prompts times context times rounds times items. Have the proposal multiply that out in tokens before you say yes.
  • Pilot one round of one item. One round of one skill would have shown the real cost for about a fifteenth of the planned spend. This is the pilot-first rule from Failure modes: how fleets go wrong together, applied to experiments.
  • Agree the stop line with the yes. A token cap that halts the run by itself is a control. A number you check afterward is just a receipt.
  • A flat score means the test is broken. When rewrites cannot move the number, stop paying for rounds and fix the test.
  • Shut a failed experiment down deliberately. Ours still left behind a reusable test set, one better description line, and this note.

Sources

  • Introducing Agent Skills, Anthropic, October 16, 2025: https://claude.com/blog/skills
  • The session transcript, July 2026; the quoted lines are verbatim.