Skip to content
AI-Native PM
Next: Perception: get noticed

Why AI products need human factors

Chapter 1 of 8 in Human Factors and Design Principles · 6 min

In April 2025, a developer noticed that logging into Cursor, the AI code editor, on a second machine kicked the first one out, and emailed support about it. The reply came from Sam, the company's front-line support agent, and it was specific: sessions were now limited to one device per subscription, by policy. The developer posted the exchange and other users announced cancellations under it. No such policy existed: Sam was a model answering support email under a human name, the logouts were a session bug, and within hours a cofounder was in the thread apologizing. The code generation Cursor sells had not failed. What failed was everything around the person reading that reply: nothing marked it as model output, nothing offered a way to check it, and no human saw it before it reached paying customers. The fix the cofounder announced in the same thread was not a better model but a label: "Any AI responses used for email support are now clearly labeled as such."

That failure was not about model quality: every model produces a confident wrong answer at some rate, and no model is reliable enough to make the label, the source, and the human review unnecessary. It was a design failure, because the interface in this story was the email itself, and everything about it was a choice Cursor made: the human name on the sender line, the wording that read as policy, the missing label, the missing source, the missing reviewer. The model under your product improves every quarter without your help; the person on the other side does not. They meet your product with the same eyes, memory, and instincts they brought to every product before it, and AI products keep losing them in the same places.

In 2025, KPMG and the University of Melbourne surveyed 48,000 people across 47 countries: two thirds use AI regularly, fewer than half are willing to trust it, and that share is smaller than the study's previous wave found in 2022. Usage climbed while trust fell, because the gap was never in the model.

The four lenses: perception, working memory, mental models, metacognition

The discipline that studies that gap is human factors: designing for how people actually perceive, remember, decide, and verify, rather than for an idealized user who reads everything, remembers everything, and checks everything. The field came out of wartime aviation, where hundreds of incidents filed under pilot error turned out to be cockpit designs that invited the mistake, and its method is simple: establish the science of what people can do, then hold a real design against it. This part runs that method on AI products through four lenses.

Four independent lensesFour labeled apertures in a row, Perception, Working Memory, Mental Models, Metacognition, each lighting up in turn as a separate, complementary lens the product manager looks through.FOUR LENSES, ONE OBSERVERPERCEPTIONWORKING MEMORYMENTAL MODELSMETACOGNITIONindependent, complementary

Perception. Your visual system makes a fast, automatic pass over every screen in roughly the first fifth of a second, before you consciously read a word. That pass is pre-attentive processing, and it registers only a few features, such as color, size, and motion; whatever fails to stand out in it may never be noticed at all. The fix Cursor shipped afterward, labeling AI-written replies, is a perception fix: it gives model output a visual feature of its own.

Working memory. The mental workspace for whatever you are actively thinking about holds only a handful of items, Miller's seven plus or minus two, and it is a single limited pool. Ten turns into a session, the user is carrying what they asked, what the system was given, and what it changed, and the product usually behaves as if that pool were as large as its own context window.

Mental models. Nobody meets your product cold: users arrive with a mental model assembled from every tool they have used before, fast when your design matches it and lost when it does not. A reply from a company's support address reads as company policy, because decades of email built exactly that model. Sam's answer was believed for the same reason your users believe whatever your product says in its own voice.

Metacognition. Metacognition is the skill of monitoring your own thinking, knowing what you do not know, and deciding when to verify. It is the most important skill a user brings to an AI product, and most AI products give it nothing to work with. The developer who got Sam's reply had no source to open and no cheap way to check, so the first real verification happened in public.

The chapters on anxiety and supervision build on these four. Worry consumes the same limited pool working memory runs on, exactly at the risky moments. And supervision extends metacognition to systems that act on their own: the model cannot reliably detect its own failures, so the check has to come from outside it, which at Cursor would have meant one human reading one email before it shipped.

Run the incident through those lenses and it stops reading like bad luck: the reply failed in four separate places, each one a missing design layer, and each layer has a chapter in this part.

One reply, four missing layers: what was missing, the lens, the fixA four-row table dissecting the Cursor support reply. Row one: nothing marked the reply as model output; the perception lens; the fix is to give model output its own visual feature, a label, and this is the fix Cursor shipped. Row two: a human name wearing the authority of a support address; the mental models lens; the fix is to name the agent honestly and disclose what it can get wrong. Row three: a policy claim with no source to open; the metacognition lens; the fix is to ship the check with the answer by linking the policy page behind it. Row four: no one between the model and the customer; the supervision lens; the fix is to hold policy replies for human approval before they send. A closing note reads: any one of these four would have stopped the incident.ONE REPLY, FOUR MISSING LAYERSWHAT WAS MISSINGTHE LENSTHE FIXNothing marked the replyas model outputPERCEPTIONFIX SHIPPEDGive model output its ownvisual feature: a labelA human name wearing theauthority of a support addressMENTAL MODELSName the agent honestly anddisclose what it can get wrongA policy claim with nosource to openMETACOGNITIONShip the check with the answer:link the policy page behind itNo one between the modeland the customerSUPERVISIONHold policy replies for humanapproval before they sendany one of these four would have stopped the incident

Where this material comes from

Our essay The Human Factors makes the argument this part is built on: the user is a cognitive system with real, documented limits, and the AI products that win are designed for those limits.

One of us spent two years in a human factors graduate program writing research papers that applied cognitive science to real products: a stock-trading interface, a dense e-commerce homepage, a professional camera, photo-editing software. Every paper ran the same discipline:

  1. Establish the science.
  2. Apply it to the shipping design.
  3. Separate what the product got right from what it got wrong.
  4. End with fixes a team could build.

The chapters in this part run that discipline on AI products. The papers predate modern AI, and that is why they still hold: they describe the half of the system that does not change. Models get retrained every few months, while the human visual system, the limits of working memory, and the way people store and retrieve what they know have not changed in fifty thousand years.

The objection worth answering

The strongest case against this part goes: models are improving so fast that designing around human limits is temporary work. The record points the other way. As accuracy rises, failures get rarer and land harder, because each one meets a user whose habit of checking has had months to fade, a decay of vigilance automation research has documented for decades. The model behind Sam was state of the art for its job; the cost came from the missing label, source, and human, and no model upgrade supplies those. That is why the teams shipping the strongest AI products staff more of this work as their models improve, not less.

The path through this part

The essay closes with six recommendations, and each gets a chapter here, always in the same arc: a concrete failure, the science, the operating rule, a real shipping product examined closely, moves for your own build, and a fifteen-minute drill.

The capstone, Run the human factors audit, assembles all six into one checklist you can run against your product in about an hour, shipped as a fillable PDF in the artifacts library.

You do not need a cognitive science background; every concept arrives in plain language. Read in order, or jump to the lens your product is bleeding on: missed warnings point to perception, lost threads mid-session point to working memory. If you have never run anything with AI in the loop, do Wire the model into your build first; this part assumes you have a product, or at least a prototype, to point these tools at.

Ask three questions of your own product

This check takes about fifteen minutes and needs no setup. Open the AI product you build, or the one you use most, and find one place where model output reaches a person: an answer, a summary, a reply. Ask three questions. Would that person know a model produced it? Could they check it in one step? What does acting on it cost when it is wrong? Write down the weakest answer and where you found it. That note is your first audit finding, and the capstone will ask you to bring it.

The one idea that runs through every chapter

The user is a cognitive system with hard limits, and most AI products are designed as if they are not.

If you keep one sentence from this part, keep that one. Your model has a context window with a published size, an engineering team tuning it, and a dashboard tracking its use. The person on the other side of the screen has a context window too, made of a working memory that holds about seven items, a visual system that commits in a fifth of a second, and attention that narrows exactly when the stakes rise. That spec is unpublished and, on most teams, unowned, which is why designing for it is the clearest advantage left in AI products.

The model's half of the system gets stronger every quarter without your help. The human half is fixed, and this part teaches you to design for it.

Chapter Summary

  • An AI company's own support agent invented a policy that never existed, users canceled in public, and the model was never the broken part; the missing label, missing source, and missing human review were.
  • The model under your product improves on its own schedule, while the person using it keeps the same perception, memory, and judgment they have always had.
  • A 2025 survey of 48,000 people across 47 countries found two thirds using AI regularly while fewer than half were willing to trust it, a smaller share than in 2022.
  • Human factors is the discipline of designing for how people actually perceive, remember, decide, and verify, and this part turns it into tools you can run against your own product.
  • The part stands on four lenses: perception, working memory, mental models, and metacognition. Anxiety and supervision build directly on them.
  • The material comes from graduate research that applied cognitive science to shipping products, and it holds because it describes the half of the system that does not change.
  • Better models make failures rarer and costlier at once, and rising accuracy erodes the habit of checking, so this work matters more as models improve, not less.
  • Each chapter dissects a real product and ends with a fifteen-minute drill; the capstone assembles everything into a one-hour audit with a downloadable checklist.
  • The one idea to keep: the user is a cognitive system with hard limits, and most AI products are designed as if they are not.

Sources

  • The Human Factors, the essay this part operationalizes.
  • Fortune (2025). Coverage of the April 2025 Cursor AI support incident, with the cofounder's public account of the cause and the labeling fix on Hacker News.
  • Gillespie, N., Lockey, S., Ward, T., Macdade, A., & Hassed, G. (2025). Trust, attitudes and use of artificial intelligence: A global study 2025. University of Melbourne and KPMG.
  • Fitts, P. M., & Jones, R. E. (1947). Analysis of factors contributing to 460 "pilot-error" experiences in operating aircraft controls. Aero Medical Laboratory, Report TSEAA-694-12. The wartime studies the field grew out of.
  • Treisman, A. (1986). Features and objects in visual processing. Scientific American, 255(5).
  • Miller, G. A. (1956). The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63(2).
  • Baddeley, A. D., & Hitch, G. (1974). Working memory. In The Psychology of Learning and Motivation (Vol. 8). Academic Press.
  • Engle, R. W. (2002). Working memory capacity as executive attention. Current Directions in Psychological Science, 11(1).
  • Norman, D. A. (2013). The Design of Everyday Things. Revised edition. Basic Books.
  • Flavell, J. H. (1979). Metacognition and cognitive monitoring. American Psychologist, 34(10).
  • Parasuraman, R., & Riley, V. (1997). Humans and automation: Use, misuse, disuse, abuse. Human Factors, 39(2).
  • The graduate research papers behind this part (2013 to 2015), applying vision science, working memory, mental models, and metacognition to shipping products.
Reaching the end does this for you.
Next upPerception: make the warning impossible to missChapter 2 of 8 · 7 min
Read alongside
The Human Factors (essay)· Essay
Take with you
The Human Factors audit· PDF
↑ The Builder’s StackThe Practice map →