The Practice · Part IV

Security

Adding AI to your product opened new doors, and attackers try them before your users do. This part turns a vague worry about safety into defenses you can build, test, and sign your name to.

The outcome

By the end you can name what your feature puts at risk, scope the keys it holds, stack defenses that code enforces, attack your own product, and ship a signed Security Posture.

Who this is for

For PMs and builders shipping an AI feature to real users, with no security background assumed.

What you’ll walk out with

You arrive withYou leave with

A circular "is it safe?" debateName what your feature puts at risk
Over-scoped, forgotten keysScope every key to its job
Untracked copies that lingerTrace where your data gets copied
Prompt-only defensesDefend below the prompt
Strangers find the holes firstAttack your own product first
Security scattered across stale docsShip a signed Security Posture

Start with chapter one The Security Posture · fillable PDF

A real incident

A single email nearly turned Microsoft Copilot against its own user. The eight chapters next door are how you keep the same opening out of your product.

01 / 09

A real incident

In June 2025, security researchers showed that a single email could make Microsoft 365 Copilot hand an attacker the confidential data in a user's workspace, with no click from the victim.

Microsoft patched it on the server before anyone was harmed, and no one was hit in the wild. The model processed the email correctly. The failure was the design around it: an outsider's email and the company's private data went into the same context with no boundary between them.

The Hacker News and Aim Security, 2025, CVE-2025-32711 (EchoLeak)

02 / 09

What happened

The attacker never broke in. They sent an email and waited for the user to ask Copilot an ordinary question.

AIM SECURITY DISCLOSURE, JUNE 2025

No user clicks are required to trigger EchoLeak.

The victim did nothing wrong.

The email carried a hidden instruction. When the user later asked Copilot about work, its retrieval step pulled the email into context and acted on the instruction, sending data out through an auto-loaded link.

03 / 09

What actually failed

Copilot worked as built. Three choices around it left the door open.

No trust boundary: an outsider's email and the company's private data were processed together, as if both could be trusted.

Every guard was passable: the injection classifier, the link redaction, and the content policy were each bypassed in turn.

Data had an exit: an auto-fetched image and an allowed proxy let the stolen text leave silently.

A stronger model would not have prevented it, because the gap was never in the model.

04 / 09

The pattern

EchoLeak was one instance of the risk every AI feature now carries.

9.3

the CVSS severity Microsoft assigned EchoLeak, out of 10

No. 1

prompt injection's rank on the OWASP LLM risk list, two editions running

clicks the victim needed for their data to leave

Any feature that reads outside text and holds a key or private data has the same opening, and it is common enough to sit at the top of the field's risk list.

Microsoft, CVE-2025-32711, 2025; OWASP Top 10 for LLM Applications, 2025

05 / 09

The one idea

Adding AI opened doors an attacker will try before your users do, and the weakest one is the untrusted text your feature reads.

The model is not where the attack lands. The risk lives in the choices around it: what text it trusts, what keys it holds, and where its output is allowed to go.

06 / 09

What this part builds

Defenses you build, test, and sign your name to.

Attack surface

Inventory every place text enters and what a hostile hour could cost.

Scoped keys

Give a hijacked feature the least authority, not your master credential.

Layered defense

Stack classifiers, code-enforced limits, and human gates so no single miss is fatal.

Red team

Attack your own feature and turn each break into a test on every release.

07 / 09

The objection worth answering

"We use a top model. It can tell an attacker's instruction from a real one."

A model processes instructions and data in the same channel, so it cannot reliably tell your company's text from an attacker's. EchoLeak walked straight past Microsoft's own classifier, which was built to catch exactly this.

The defense that holds is in the design around the model, not in its judgment. You scope the keys, bound the output, and stack the guards so one bypass is not the whole breach.

08 / 09

What actually fixes it

Each gap in the story maps to a move you can build.

Text enters unmappedAttack surface

→ Inventory every place outside text reaches your feature, and what a hostile hour could cost.

Keys over-scopedBorrowed keys

→ Scope every credential to its job, so a hijacked feature inherits the least authority.

One guard, one failureDefense in layers

→ Stack classifiers, code limits, and human gates so no single miss is fatal.

Attackers test it firstRed team

→ Attack your own feature and lock each break behind a test that runs on every release.

The doors are known. This part is how you close them before anyone tries them.

09 / 09

From story to part

Every gap in this story is a chapter here.

Eight chapters take you from a feature and a vague worry to threats ranked, keys scoped, data traced, imports pinned, layers stacked, and attacks rehearsed, ending in a signed Security Posture with a kill switch and a first-24-hours plan.

Start with chapter one →

↑ The Builder's Stack The Practice map →