Skip to content
AI-Native PM

A real incident

Flip through what happened when an AI company shipped without human factors. Every gap you see is a chapter next door.

01 / 09
The Human Factors

In April 2025, an AI company’s own support bot invented a policy that never existed.

Users canceled in public, a cofounder apologized within hours, and the model was never the broken part.

Fortune, 2025 · the cofounder’s public account on Hacker News

02 / 09
What happened

Logging in on a second machine kicked users out. Support explained why.

FROM: SAM · CURSOR SUPPORT

“Sessions are limited to one device per subscription, by policy.”

No such policy existed.

The logouts were a session bug. Sam was a model answering support email under a human name, and paying customers read the reply as company policy.

03 / 09
What actually failed

The code generation Cursor sells had not failed. Everything around the reader had.

Nothing marked the reply as model output.

Nothing offered a way to check it.

No human saw it before it reached paying customers.

A better model would not have prevented any of it, because the gap was never in the model.

04 / 09
The evidence

Usage climbed while trust fell.

2 in 3

people now use AI regularly

< half

are willing to trust it, a smaller share than in 2022

The models got better across those three years. The trust went the other way, because the gap was never in the model. It is on the other side of the screen.

KPMG × University of Melbourne, 2025 · 48,000 people · 47 countries

05 / 09
The one idea

The user is a cognitive system with hard limits, and most AI products are designed as if they are not.

Your model has a context window with a published size and a team tuning it. The person on the other side has one too. Nobody publishes that spec, and nobody on most teams owns it.

06 / 09
The four lenses

The spec of the person on the other side.

Perception

The first glance commits in about a fifth of a second. What it misses may never be noticed at all.

Working memory

The active mind holds about seven items, and your session is competing for all of them.

Mental models

Nobody meets your product cold. A reply from a support address reads as company policy.

Metacognition

Knowing when to check is a skill, and most AI products give it nothing to work with.

Fitts & Jones, 1947 · Treisman, 1986 · Miller, 1956 · Norman, 2013 · Flavell, 1979

07 / 09
The objection worth answering

“Better models will make this unnecessary.”

The record points the other way. As accuracy rises, failures get rarer and land harder, because each one meets a user whose habit of checking has had months to fade.

The model behind Sam was state of the art for its job. The cost came from the missing label, source, and human, and no model upgrade supplies those.

08 / 09
What actually fixes it

One reply, four missing layers.

No label on the replyPerception

→ Give model output a visual signal of its own

A human name’s authorityMental models

→ Name the agent honestly, and show its limits

No source to openMetacognition

→ Ship the check with the answer

No human in the loopSupervision

→ A person approves before it sends

Any one of the four would have stopped it.

09 / 09
From story to part

Every gap here is a chapter.

Eight chapters turn these fixes into design rules you can ship, and end in a one-hour audit you can run on any AI product, yours or a competitor’s.

Start with chapter one →
↑ The Builder's StackThe Practice map →