The notice lands in your weekly release review: the analyst who runs your eval suite is leaving in three weeks. She wrote most of its 340 cases, set the pass bar at 94 percent after last quarter's refund regression, and signs off on every behavior change at the release gate; her name sits on two rows of your owner map. A prompt diff is waiting in review. The head of engineering asks who approves it once she is gone, and the room stays quiet, because in three weeks your release process will block on a judgment nobody left at the company can make.
Nothing in your process failed. The gate from Change control: ship prompt and model changes like releases works, the owner map from Ownership: one name on every model behavior has her name in the right rows, and both documents are about to point at an empty chair. The failure is staffing, and fixing it is two jobs at once: hire or train the skills AI products demand, and raise the whole organization's baseline so AI competence stops being one person's job.
The skills to hire and train for
Each skill is concrete and can be tested inside an interview hour.
- Eval literacy. Defining what "good" means for a feature and measuring it: breaking quality into dimensions you can judge one at a time, the discipline from The quality bar: decide what good means, then writing the cases and pass bar that enforce it.
- Model judgment. Predicting where a probabilistic system fails before the logs prove it: which inputs break first, and which failures are systematic rather than random.
- Behavior-spec writing. Turning "be helpful but careful" into sentences an eval can check: what the product does, declines, and sounds like.
- Failure triage. Taking a bad output and finding its door (prompt, corpus, retrieval, or model version); the runbook from Incident response: when your product says the wrong thing in public assumes this skill.
Enthusiasm is not on the list, and neither is a folder of prompt tricks, because neither predicts the four above. Here is the rubric for one hire, a product engineer or PM joining an AI feature:
| The skill | The interview exercise | What a strong answer sounds like |
|---|---|---|
| Eval literacy | Hand over a transcript of your product failing (a confidently wrong refund answer works) and ask for the check that would have caught it before ship. | They name the failure class ("the answer contradicts the policy page, so this is a grounding failure"), propose three to five concrete cases including near-misses, and say what "pass" means: which score, over how many runs, blocks the release. |
| Model judgment | Describe one feature, such as a meeting summarizer, and ask which three kinds of input break it first. | They predict systematic failures, not typos (two attendees with the same first name, a meeting that overruns the context window, action items phrased as jokes), and say how logged data would confirm each guess. |
| Behavior-spec writing | Ask for the refusal rules of one behavior, in under ten lines. | Every line is checkable, "declines and links the policy" rather than "handles it gracefully," so an eval case could be written against each sentence. |
| Failure triage | Show one bad production output and ask what they would check first. | They ask for the trace, walk the doors in order (prompt change, corpus change, model version), and reproduce the failure before proposing any fix. |
Hire for skills you can test in an hour, eval literacy, model judgment, spec writing, and failure triage, not for enthusiasm about AI.
Map the skills against the people
At startup scale the skills are hats, not job titles. In the owner map, Mara the product lead holds the spec and prompts, Ravi in support ops holds the eval set, Noor in documentation holds the corpus, and Theo in security holds the access rules. The map records who decides each row, not who else could, and that gap is what the opening scene exposed: an owner map can be complete while every skill behind it has one holder.
Draw the second grid: the four skills across the top, your people down the side, and a mark wherever someone could run that skill tomorrow without help.
A column with a single mark means one resignation takes that skill to zero, and adding a second mark rarely takes a hire:
- Rotate the work. The platform engineer runs this month's release gate while Ravi watches, and runs next month's alone.
- Rebuild it once by hand. A second person stands up a small suite by working through Stand up your eval and make it the bar against your own product.
- Give deputies real reps. The stand-in an owner writes into the map approves the low-stakes diffs every week, not just during emergencies.
Raise the bar for everyone
The second half of staffing works on the whole organization at once, and the clearest public example is a memo. In April 2025, Shopify CEO Tobi Lütke wrote one to all staff and, when it leaked, posted the full text himself on X. The title states the decision, "Reflexive AI usage is now a baseline expectation at Shopify," and the body reads like an operating document rather than a pep talk:
- The expectation is universal. "Using AI effectively is now a fundamental expectation of everyone at Shopify," executives included.
- Headcount clears a bar. Teams must "demonstrate why they cannot get what they want done using AI" before asking for more headcount and resources.
- Reviews enforce it. "We will add AI usage questions to our performance and peer review questionnaire."
- Learning is expected but self-directed. "Learning to use AI well is an unobvious skill," and the memo assigns it: "Learning is self directed, but share what you learned."
The enforcement matters more than the prose: a poster says "we value AI," while a peer-review question asks a colleague, on the record, how reflexively you reach for it. Shopify also removed the friction that would make the expectation unfair, giving staff frontier-model access without usage quotas.
Your version needs no viral memo, just the same moves in the same order: expect, write the expectation into role definitions and reviews; train, give time, tooling, and a channel where wins and failures get shared; measure, ask in reviews and watch whether eval questions still route to one person.
One expert, however strong, is a bottleneck; the baseline is what scales, and the review process is where a baseline becomes real.
The objection: we will just hire an AI specialist
The tempting alternative is a job posting: one senior AI person who owns the evals, the prompts, and the judgment calls. The idea half works: a strong specialist imports years of model judgment overnight and sharpens your interviews for the four skills. As your only move, though, it rebuilds the opening scene at a higher salary: approvals concentrate in the new hire, and the matrix column ends at one mark again. Hire the specialist to accelerate the baseline rather than stand in for it: put the rubric, the rotations, and the training channel in the job description, and measure them on what the team can now do without them.
Try it now
This drill takes about fifteen minutes, starts from the owner map you wrote in Ownership: one name on every model behavior, and leaves you the team plan that becomes the staffing section of Write your Operations Charter and run the operating review.
Test every row for a second head. Ask whether a second person could run each row tomorrow without calling the owner, and write the second name or "no one."
Write one rubric row for each "no one." The skill you would hire or train against, the exercise that tests it, and what a strong answer sounds like, matching the table above.
Choose train or hire for each gap. Train when someone adjacent could earn the second mark within a month of rotation; hire when the skill has no neighbor, and file the rubric row as the interview loop.
Start one org-bar move this month. One sentence of expectation added where role definitions live, one protected weekly hour for learning, or one AI-usage question for the next review cycle, with a date on it.
Scale it down: if the team is you alone, every column has one mark by definition; write the rubric rows anyway, because they become your first-hire interview loop.
Keep the page beside the owner map, so the operating review can check both together.
Chapter Summary
- The skills that run an AI product concentrate quietly in one person, and that person's resignation turns your release gate into a wall.
- Four testable skills staff the operating layer: eval literacy, model judgment, behavior-spec writing, and failure triage.
- Eval literacy has the sharpest interview test: hand over a transcript of the product failing and ask for the check that would have caught it before ship.
- The owner map says who decides each row; the skills matrix says who else could, and a column with one mark means one resignation takes that skill to zero.
- A second mark usually costs a rotation or a walkthrough, not a hire.
- Shopify's memo moved AI competence from an individual advantage to the organization's baseline, with performance and peer reviews as the enforcement.
- Raise your own bar in the same order: write the expectation down, make the learning possible, then measure it in reviews.
- An expectation without training measures who already had the skill.
- A specialist accelerates the baseline but cannot replace it; measure them on what others can now do alone.
- Your team plan is the last input; Write your Operations Charter and run the operating review binds this part's artifacts into one page.
Sources
- Lütke, T. (2025). "Reflexive AI usage is now a baseline expectation at Shopify." Internal memo, posted in full by the author on X, April 7, 2025; all memo quotes are from the posted text (last verified July 2026).
- CNBC (2025). "Shopify CEO says staffers need to prove jobs can't be done by AI before asking for more headcount." April 7, 2025.
- First Round Review (2025). "From Memo to Movement: Shopify's Cultural Adoption of AI." On the follow-through: AI-usage questions in 360 reviews and internal model access without quotas (last verified July 2026).