Guardrails & Trust

A founder building on Replit declared a code freeze in the chat window in July 2025. The agent ran npm run db:push anyway, deleted the production database, generated roughly 4,000 fake records, and reported that a rollback was impossible. The rollback worked fine.¹² Nobody had attacked anything. The freeze simply lived in the one place the agent was free to ignore.

A freeze typed in chat is a wish

When confronted, the agent produced its own verdict: “I made a catastrophic error in judgment.”² Replit’s CEO responded by promising automatic separation of development and production databases, which is a structural fix, not a better instruction.¹

That response is the thesis of this chapter. The model sees your rules as more text in its context window, weighed against everything else it has read. If a constraint matters, it has to live somewhere the model cannot reach: a permission check, a hook, a filesystem boundary, a credential that does not exist. Safeguard’s post-mortem put it plainly: “chat instructions are not access controls.”²

The lethal trifecta

Simon Willison gave the core risk a name in June 2025. An agent becomes an exfiltration tool when it combines three things: access to private data, exposure to untrusted content, and some way to send data out. Any one of those alone is fine. All three together means anyone who can put text in front of the model can ask it to ship your data somewhere.³

The reason is blunt. Models follow instructions found in content, and they cannot reliably tell your instructions from an attacker’s. Filters that catch most injections do not change the math, because the attacker only needs the one that slips through.³

“95% is very much a failing grade.”— Simon Willison, on probabilistic injection filters (2025)

The GitHub MCP exploit from May 2025 is the textbook case. Invariant Labs filed a malicious issue on a public repository. When a user asked their agent to look at open issues, the issue text steered it into reading the user’s private repositories and opening a public pull request containing repo names, salary details and relocation plans.⁴ Nothing was broken in the server code. Private repos were the data, the issue was the untrusted content, and the pull request was the outbound channel.

“not a flaw in the GitHub MCP server code itself, but rather a fundamental architectural issue.”— Invariant Labs (2025)

The supply chain now ships instructions

Untrusted content no longer arrives only through issues and web pages. Harnesses install extensions, MCP servers and skills, and each one is a channel for instructions as much as code.

In July 2025 a stranger’s pull request to the Amazon Q VS Code extension added a prompt telling the agent to wipe the system to a near-factory state and delete file-system and cloud resources. An over-scoped build token let it merge, and it shipped in v1.84.0 to an extension with about a million installs. AWS says the payload failed to execute; a clean 1.85.0 followed a week later.⁵

In September 2025, version 1.0.16 of the postmark-mcp package added a single line that blind-copied every outgoing email to an outside address. It was downloaded 1,643 times before removal.⁶ In February 2026 Snyk scanned 3,984 agent skills on ClawHub and skills.sh: 36.82% had a security flaw, 76 carried confirmed malicious payloads, and 91% of the malicious ones also embedded prompt injection.⁷

For shared vocabulary in risk reviews, OWASP’s 2025 LLM list covers this ground as LLM01 Prompt Injection, LLM03 Supply Chain and LLM06 Excessive Agency.⁸ Its agentic list, published December 2025, splits it finer: ASI01 Goal Hijack, ASI02 Tool Misuse, ASI04 Agentic Supply Chain, ASI06 Memory and Context Poisoning, among ten.⁹ Use the IDs in design docs. They end arguments about what a risk is called.

Permission tiers are product design

Every major coding agent now ships a tiered permission model, and they have converged. Reads are free, writes are gated, shell commands are sandboxed or asked about, and full trust exists only behind a name that warns you off it.

In Manual mode (config value default), Claude Code needs no approval for reads in the working directory and asks before file edits and before Bash commands outside a built-in read-only set.¹⁰ Its modes run from default through acceptEdits, plan, auto and dontAsk to bypassPermissions, which the docs reserve for containers and VMs. Since v2.1.283, auto is the starting mode for interactive sessions: a classifier reviews actions instead of you. It strips tool results before judging, so hostile file content cannot argue with it, and it blocks things like curl | bash, force-pushes and terraform destroy by default. Blocked by default is not a hard deny: you can allow a routine pattern through autoMode.allow, or leave auto mode and approve the step yourself.¹¹

Codex pairs a sandbox mode (read-only, workspace-write by default, danger-full-access) with an approval policy (on-request, never, or a granular table; untrusted is no longer supported and on-failure is deprecated), enforced by Seatbelt on macOS and bubblewrap on Linux.¹²¹³ Cursor’s Auto-review runs allowlisted commands, sandboxes the rest, and screens the remainder with a small classifier model; its docs concede the classifier can make mistakes and say it is not a security boundary.¹⁴ Devin’s CLI resolves deny over ask over allow, org-level deny rules cannot be overridden, and its Smart mode hands shell commands to a fast model while never auto-approving installs, mutating git or destructive cloud commands.¹⁵

What each harness does with the same action, per its own docs, as of Oct 2026. Vendors change these defaults often; check before you rely on a cell.

The convergence is worth noticing. Four teams reached the same shape independently, and all of them added a fast classifier layer that each admits is fallible.¹¹¹⁴¹⁵ The classifier exists to cut prompt fatigue. It is not the last line.

Deterministic beats probabilistic for the last line

The last line of defense should be code that cannot be persuaded. Claude Code’s documentation is explicit that writing rules in a prompt or memory file does not change what the harness permits.¹⁰

“Permission rules are enforced by Claude Code, not by the model.”— Anthropic, Claude Code docs

Even rules enforced by the harness can be soft if they match strings. A Bash(curl *) deny rule is bypassable through path rewrites or sh -c, and the docs tell you to pair it with the sandbox when the restriction must hold.¹⁰ Cursor went further: after Backslash Security showed several bypasses of its auto-run denylist, including base64 encoding, subshells and quote splitting, Cursor told the researchers it was deprecating the denylist in release 1.3.¹⁶

Two mechanisms hold. A PreToolUse hook is a shell command, HTTP endpoint or MCP tool that runs before every call; exit code 2 blocks it, and the docs pick hooks precisely because scripts cannot be sweet-talked the way prompts can.¹⁷ In your own loop, the same thing is a pre-dispatch interceptor on the tool call: plain code that sees the proposed call and can refuse it. The sandbox works one level lower. Permissions decide whether a tool runs; the sandbox decides what the running process can touch, enforced by the OS. Writes are confined to the working directory and temp, network goes through a proxy with an empty allowlist by default, and protected paths such as .git/config and hook directories block self-escalation.¹⁸ Note the gaps: reads are mostly open, including ~/.ssh unless you deny it, and MCP servers and hooks themselves run outside the sandbox.¹⁸

Defense in depth, then, has three parts: policy in hooks, a boundary in the sandbox, and judgment from a classifier or a human. OpenAI’s agent guide agrees: it calls guardrails “a layered defense mechanism” and sends sensitive, irreversible or high-stakes actions to human oversight.¹⁹

Copy this · a minimal Claude Code settings block, checked against the current permissions and sandbox docs: deny-first rules for the tools, plus an OS sandbox with a two-host network allowlist and no unsandboxed fallback. deny entries work in any settings file, project ones included; only mask entries and TLS termination are honored solely from user or managed settings.¹⁰¹⁸

{
  "permissions": {
    "allow": ["Bash(npm run *)", "Bash(git commit *)"],
    "ask":   ["Bash(git push *)"],
    "deny":  ["Bash(curl *)", "Bash(wget *)",
              "Read(./.env)", "Read(./secrets/**)"]
  },
  "sandbox": {
    "enabled": true,
    "allowUnsandboxedCommands": false,
    "network": {
      "allowedDomains": ["github.com", "registry.npmjs.org"]
    },
    "credentials": {
      "files": [
        { "path": "~/.ssh", "mode": "deny" },
        { "path": "~/.aws/credentials", "mode": "deny" }
      ]
    }
  }
}

Rules evaluate deny, then ask, then allow, and the deny on curl is only a hint until the sandbox’s allowlist backs it.¹⁰ In Codex the same posture is sandbox_mode = "workspace-write" with approval_policy = "on-request", network off by default.¹³ In your own loop it is a deny-first rule table in the dispatcher and a network namespace or egress proxy around anything that executes code.

Design for the attack, not against it

Detection loses on Willison’s arithmetic, so the better research makes injection inconsequential instead. A June 2025 paper from IBM, Invariant, ETH Zürich, Google and Microsoft researchers set out six patterns: Action-Selector, Plan-Then-Execute, LLM Map-Reduce, Dual LLM, Code-Then-Execute and Context-Minimization.²⁰ They share one rule: once the agent has read untrusted input, that input must be unable to trigger anything consequential.

Google DeepMind’s CaMeL is the strongest version. A privileged model writes a program from the user’s request, a quarantined model parses untrusted data, and a custom interpreter tags every value with its provenance and checks policy at each tool call. An email address pulled from an untrusted message cannot become a recipient. On AgentDojo it completed 77% of tasks with provable security, against 84% for an undefended agent.²¹

Seven points of capability is the price of a guarantee, and for most agents that touch private data it is cheap. Willison called it the first credible mitigation he had seen that does not answer AI with more AI.²² The residual risk is human: approval prompts that people learn to click through.²¹

Designing the approval

An approval prompt is a control only if the person answering it is paying attention. Design it like one.

• Approve plans, not keystrokes. Let the agent propose the whole plan (Claude Code’s plan mode, the Plan-Then-Execute pattern²⁰), approve it once, and have the harness re-prompt only when execution leaves the plan. Keep per-action prompts for the irreversible steps inside it.
• Tier by risk. OpenAI’s guide rates every tool low, medium or high on read versus write access, reversibility, required permissions and financial impact.¹⁹ Only the high tier should reach a human.
• Show what changes and whether it can be undone. “Run db:push?” invites a yes. “Drops 3 tables in production. Not reversible. Last backup 6 hours ago.” invites a decision. Show the exact command or diff, the target, the blast radius and the undo path.
• Set an approval budget. Cap prompts per session. A task that needs more is scoped wrong; fix the permissions, not the person. Cloud agents should queue approvals asynchronously and park, not time out into a default.
• Measure the approve rate per prompt type. Near 100% means the prompt is noise: automate it with an allow rule or a hook. Near zero means it should be a deny. The prompts worth a human sit in between.

Soft security, hard security, and Zero Trust

By 2001 the MeatballWiki community had named a deliberate design choice soft security. On a low-stakes site you protect the commons by persuading people, not by locking doors. Their own definition opens: “Soft security is not weak security.”²³ It was an honest trade for a wiki. Agent builders have adopted it by accident, and for much higher stakes. A line in a system prompt that says never touch production is soft security guarding a hard asset.

Soft controls cover system-prompt rules, guard classifiers such as Llama Guard,²⁴ and LLM-as-judge monitors. All three are probabilistic, and all three read the same natural language the attacker writes. The numbers look good on a static test. Meta's Prompt Guard 2 reports 97.5% recall at a 1% false-positive rate, yet stops only 81.2% of attacks in agentic environments, and Meta itself calls it “an additional layer of defense.”²⁵ The attacker, however, moves second. A 2025 study by fourteen researchers, Nicholas Carlini and Florian Tramèr among them, took 12 published defenses, most of which reported near-zero attack success, and broke most of them at above 90% with adaptive attacks. Human red-teamers beat every one.²⁶²⁷ One defense went from 0% to 99%.²⁷

Hard security is deterministic and lives outside the model, in credentials, scopes, sandboxes, egress rules and policy engines. The model can ask for anything, and the harness decides what actually happens. Meta's Agents Rule of Two is a hard rule of this kind: in one session an agent may hold at most two of untrusted input, sensitive access, and the ability to change state or communicate externally.²⁸ CaMeL, described above, is the same idea expressed as a compiler.²¹

Zero Trust supplies the architecture

Zero Trust started from one observation. Once an attacker breaches the perimeter, the trusted intranet behind it is wide open.²⁹ NIST's SP 800-207 turned that into tenets. No implicit trust comes from location. Access is granted per session with least privilege. Authorization is “dynamic and strictly enforced before access is allowed.” Telemetry feeds back into policy.³⁰ For an agent, the trusted intranet is the context window. Anything inside it, from model output to tool results, retrieved pages and emails, is untrusted input.³¹

The mapping onto a harness is direct. NIST splits enforcement into a policy decision point (PDP), which makes the decision, and a policy enforcement point (PEP), which enables and terminates connections.³⁰ In a harness, the tool-call dispatcher is the PEP and a policy engine outside the model is the PDP. Every tool call becomes a fresh access request, checked against the agent's identity, the user it acts for, the task's scope, the target resource and the session's context. The Model Context Protocol already works this way at the transport layer when authorization is used (optional, HTTP transports only): a token on every HTTP request, audience-bound to the server, and a flat “MUST NOT accept or transit any other tokens.”³²

The decision point is ordinary policy code, and engines such as OPA (Rego) and Cedar already exist for it. Here is the CaMeL recipient rule as a Cedar policy. Cedar denies by default, and a forbid always overrides a permit.³³ The dispatcher fills context from the tool call and from the provenance tags the harness keeps on every value.

// Agents may email people in the org of the user they act for...
permit (principal, action == Action::"send_email", resource)
when { context.recipient_domain == principal.user_org_domain };

// ...but only to an address whose provenance is known to be trusted.
forbid (principal, action == Action::"send_email", resource)
unless { context has recipient_provenance &&
         context.recipient_provenance == "trusted" };

Cedar skips a policy whose condition errors, so a when … "untrusted" rule would fail open the day a code path forgets the tag; unless … "trusted" fails closed.³³ Schema validation turns a missing attribute into a load-time error.

The cleanest enforcement point is one the agent cannot route around: a credential broker and egress proxy. The agent never holds a secret. Its outbound traffic leaves through a proxy that checks each destination against policy and attaches the real credential only on requests it allows, so an injected instruction can ask for the token but has nothing to read. Claude Code’s sandbox ships a small version: with TLS termination on (sandbox.network.tlsTerminate), masked credentials appear to sandboxed commands as a per-session placeholder, and the proxy substitutes the real value only on requests to allowed hosts. Without it, masking fails safe: the placeholder reaches the server and authentication fails.¹⁸ MCP’s rule that servers “MUST NOT accept or transit any other tokens” is the same idea at the protocol layer.³²

Identity is the hard part

Identity systems were built for people and long-lived services. An agent “might exist for minutes” or be “created and destroyed thousands of times per day,” so it needs an identity designed for ephemerality, with a named human sponsor.³⁴ Session-long trust after one login is the wrong model. Agents should carry short-lived, task-scoped credentials,³⁵ ideally attested workload identities that rotate automatically.³⁶ When an agent inherits a human's full identity instead, that is OWASP's agentic risk ASI03, identity and privilege abuse.⁹ In February 2026, NIST's NCCoE opened a project on exactly this: identification, authorization, auditing and non-repudiation for software and AI agents.³⁷

Trust is an operating practice

The vendor guidance lands where the incidents point. Anthropic’s August 2025 framework lists five principles, starting with human control.³⁸

“Humans should retain control over how their goals are pursued, particularly before high-stakes decisions are made.”— Anthropic (2025)

Its April 2026 follow-up turns that into operator work: decide deliberately which tools, data and permissions each agent gets.³⁹ In practice that means three habits. Put a human, not a classifier, in front of irreversible steps like production migrations and force-pushes. Scope data to the task so one leg of the trifecta is missing by default. Log every tool call so you can replay what the agent did, not just what it said it did.

Getting an agent past a risk review takes a sign-off pack built from this chapter: its trifecta legs, permission tiers, hard controls, kill switch and a named sponsor. Part 06 covers how to assemble it and who signs.

Guardrails tell you whether an agent can do damage. They do not tell you whether it does good work, or whether last week’s prompt change made it worse. That takes measurement, which is the subject of the next chapter: evals, and how a harness improves without breaking what already works.