Operating the Harness

Of 1,340 teams LangChain surveyed at the end of 2025, 89% had observability on their agents. Only 52.4% ran offline evals, and 37.3% scored live traffic.¹ Most programs can see what their agents did; far fewer can say whether it was right. That is an operating problem, and it belongs to whoever owns the budget, the team and the sign-off.

Buy the inner harness, own the outer one

The number budget owners quote is MIT NANDA's: external partnerships reached deployment about 67% of the time, internally built tools about 33%.² Read it before you quote it: it rests on 52 organization interviews, 153 surveyed leaders and a review of 300+ public initiatives. The authors call the rates “self-reported” and warn that the correlation “does not necessarily prove causation”.² Teams that buy may simply pick narrower problems.

The counter-signal is real too. Per The Register, nearly a third of McKinsey's 2026 respondents had skipped buying at least one software product because agentic coding tools let them build it.³ And a16z found CIOs shifting toward buying, with 37% now using five or more models.⁴

Build vs buy is the wrong unit; ask it per subsystem. MIT's own diagnosis of the failures is a “learning gap”: tools that do not learn, integrate poorly, or do not match the workflow.² Those are outer-harness properties, in Böckeler's sense: the instruction files, tools and checks you add around the vendor's agent.⁵ The 33% are, plausibly, teams that tried to build both layers.

The rule in one line: buy the parts that are the same for every customer, own the parts that are your business. Ask every vendor of a bought part six questions:

1. Are our prompts, tool results and traces retained, for how long, where, and are they used for training?
2. Can we export full traces, in OpenTelemetry GenAI form, to our own store? (Ch 5)
3. Can our policy engine decide every tool call before it runs? (Ch 4)
4. Is sandbox egress default-deny, with an allowlist we control? (Ch 3)
5. Can we swap the model behind it, and prove it with our regression set? (Ch 5)
6. What does leaving cost, and which of our skills, tools and evals keep working without you?

Who owns what

Two failure shapes recur: every team builds its own sandbox, tracing and policy engine, badly and five times, or a central team builds a grand platform nobody adopts. Team Topologies names the middle path, the thinnest viable platform.⁶

“That might just be a wiki page - that might be your platform.”— Matthew Skelton, Team Topologies

For agents, the thin platform owns the inner-harness primitives: the sandbox image, the permission-policy engine, tracing, the eval runner, and a shared registry of skills and tools. Embedded engineers own each use case's outer harness: context, tool contracts, policy rules and eval tasks. This is the internal forward-deployed engineer, with the same test: does a field fix become a reusable artifact?

Staff evals first. The first of OpenAI's lessons from seven frontier companies is “Start with evals”; Morgan Stanley ran them for every use case before rollout.⁷ Thoughtworks' CTO makes the staffing case.⁸

“…a significant need for humans to proactively implement appropriate practices and technical harnesses…”— Rachel Laycock, Thoughtworks (2026)

When to form the platform team. My rule of thumb, not a published standard: start embedded, and form the platform team at the second or third use case, once a primitive has been built twice. Size it at roughly one platform engineer per three or four embedded ones. Its first deliverables, in order: tracing tagged with the harness version, the eval runner, the policy engine, the sandbox image. Nothing climbs the ladder without the first two.

Adoption is a harness job too. The reviewer is part of the harness. Redesign the workflow around the agent instead of bolting it onto the old one, train reviewers on what a bad output looks like, and watch their approve rate: one who approves everything has stopped reviewing (Ch 4). Give each team a named adoption owner, measured on the dashboard's adoption row.

The maturity ladder

Autonomy is a setting you choose, not a property the model has. The Knight Institute's levels define it by the user's role: Operator, Collaborator, Consultant, Approver, Observer.⁹ In their words, autonomy “can instead be a deliberate design decision made by agent developers.”⁹ The Cloud Security Alliance's Agentic Trust Framework adds what most ladders lack: promotion gates. An agent observes for at least two weeks; recommends with approval for at least four, at over 95% recommendation acceptance, before it may act with notification; and acts with notification for at least eight, with zero critical incidents and over 99% action accuracy, before it is autonomous within a domain. There, any significant incident triggers automatic demotion.¹⁰

METR's data explains why gates must use measured reliability. The task length models finish 50% of the time doubles about every seven months, and the 80%-reliability horizon is much shorter.¹¹ Demos show the 50% horizon. Production runs on the other one.

Full trust (bypass modes) is not a stage. It belongs only in disposable containers, at any stage.

Cost and routing

An agent's bill is a loop, so model it as one. Per task: cost ≈ Σ over turns of (prefix_t × cache-read price + new tokens_t × cache-write price + output_t × output price). The prefix is not constant: in an append-only loop it grows by every earlier turn's tool results and output, so read cost rises faster than linearly with turns. As of October 2026, Anthropic bills cache writes at 1.25× base input (five minutes) or 2× (an hour) and reads at 0.1×, newer models 0.025× to 0.05×.¹⁶ Ch 2 covers why the hit rate matters, Manus's numbers included.

A worked task, with illustrative inputs and Sonnet 5.5 list prices as of October 2026 ($2 per million input tokens, $2.50 cache write, $0.10 cache read, $10 output):¹⁷ a 20,000-token prefix, 30 turns, each adding 3,000 tokens of tool results and 500 of output. Reads total 2.1 million tokens ($0.21), cache writes 110,000 ($0.28), output 15,000 ($0.15): about $0.64 a task. Over 70% of those reads are the growing history, not the fixed prefix. Without caching, the same task costs about $4.60. Turns dominate, so fan-out, which multiplies them, buys quality with money (Ch 1). Decide that on purpose.

Three levers move the number. Route by job: small, fast models for classifiers, summarizers, compaction and well-specified subagents; the frontier model for planning and for the step whose error is expensive. Cursor screens commands with a small classifier model.¹⁸ Set the effort: Anthropic's effort setting (low, medium, high, xhigh, max) governs the tokens spent on thinking, text and tool calls (“fewer and terser tool calls” when lower). It is “a behavioral signal, not a strict token budget”; the docs advise an effort sweep on your own evals.¹⁹ Cap it: Anthropic recommends stopping conditions such as a maximum number of iterations;¹² make that a per-task hard budget in tokens and dollars that the harness enforces. A cap the model can talk past is a suggestion.

A route or effort change is a harness change: it passes the same paired regression gate as any other (Ch 5), and cost may fall only while pass^k holds.

Getting it past the risk review

Risk reviewers want a page in their vocabulary, not a demo. NIST's AI Risk Management Framework supplies it: four functions, Govern, Map, Measure and Manage, with a generative-AI profile, NIST AI 600-1, published in July 2024; the framework is being revised.¹⁴ NIST's agent standards initiative (February 2026) covers agent identity and authorization; no final standard exists yet.²⁰ ISO/IEC 42001 is the certifiable management-system standard, sharing the ISO management-system structure used by ISO 27001.²¹

EU AI Act dates moved this year. Per the Future of Life Institute's tracker, the Digital Omnibus, Regulation (EU) 2026/1744, in force since 27 July 2026, pushes Annex III high-risk obligations to 2 December 2027 and Annex I product obligations to 2 August 2028. Article 50 transparency duties stayed on 2 August 2026, with one transition: generative systems already on the market before that date have until 2 December 2026 to meet Article 50(2), the duty to mark synthetic output as machine-readable and detectable.²² For anyone shipping generated content in the EU, that is weeks away. Check the primary text before you plan against it.²³

The pack maps cleanly: inventory and trifecta are Map, eval evidence is Measure, kill switch and rollback are Manage, the sponsor is Govern. Threat-model it against the OWASP agentic Top 10, from goal hijack (ASI01) and tool misuse (ASI02) to rogue agents (ASI10).¹⁵

The data owner will ask about traces first. They hold prompts, tool results and customer data, and this book feeds them into evals and the dashboard. Decide before the first trace is stored: who may read raw traces (a named role), how long they are kept, what is redacted at capture rather than at query time, and whether production data may enter eval sets, scrubbed how. OpenTelemetry's GenAI conventions keep message content capture opt-in for this reason.²⁴

Copy thisA minimal one-page approval pack. Fill every angle-bracket field; an empty field is a finding.

# Agent approval pack · <agent name> · harness v<version>
Sponsor (accountable human): <name, role>          # NIST Govern
Use case / systems touched:  <one line>
Inventory ID / risk class:   <id> / <class>; EU AI Act check: <date>
Trace data: readers <roles>; retention <days>; redaction <method>;
            prod data in evals <y/n, how scrubbed>

## 1. Lethal trifecta (Ch 4)                          # NIST Map
Private data:      <which>
Untrusted content: <which sources>
External comms:    <which channels>
Leg cut by:        <control>

## 2. Permission tier: <ask-on-write | sandboxed auto | high-risk approval>
Tool               access   reversible  $ impact  rating
<name>             <r/w>    <y/n>       <none/$>  <low/med/high>

## 3. Hard controls (enforced outside the model)
- <rule> -> enforced by <hook/policy/sandbox> -> proven by <test id>

## 4. Eval evidence                                   # NIST Measure
Regression set: <N> failure-derived tasks; pass^<k> = <x%>; baseline run <id>
Security cases: <injection + incident replays>; attack success <x%>

## 5. Kill switch and rollback                        # NIST Manage
Kill switch: <how, who>, last tested <date>; breaker trips on <rule>
Revoke all sessions/credentials: <one action, owner>
Rollback to: harness v<previous>; comms owner: <name>

## 6. Rollout
Stage <n> -> <n+1>; gate evidence: <weeks, acceptance %, incidents, dated>
KPIs and alert thresholds: <list>; next review: <date>

What goes on the exec dashboard

Define the metric before you compare. Klarna's first month, February 2024: 2.3 million conversations, two-thirds of service chats, the work of 700 full-time agents, resolution down from 11 minutes to under 2, a projected $40 million profit gain.²⁵ In May 2025, Fortune reported from a Bloomberg interview that quality had suffered and Klarna was hiring humans again.²⁶

“As cost unfortunately seems to have been a too predominant evaluation factor when organizing this, what you end up having is lower quality.”— Sebastian Siemiatkowski, via Fortune (2025)

Intercom reports a 76% average resolution rate for Fin and is moving to counting outcomes, which include correct handoffs to humans.²⁷ Same agent, new definition, different number.

The gen_ai.* names come from the OpenTelemetry GenAI conventions.²⁴ Tag every span with the harness version, or no number on this board can be tied to a change.

ROI follows from the board. Value = tasks × success rate × value per task − (tokens + engineering + review) − escaped failures × cost per failure (rework, refunds, churn, incidents), where escaped failures = tasks × failure rate × (1 − review sampling rate). Reliability is what most business cases miss. τ-bench defines pass^k, the chance that all k tries at a task succeed.²⁸ My reasoning from there: unmeasured, every output needs a human; only measured and high can review drop to a sample.

An illustrative month (made-up inputs): 10,000 tasks at $15 of avoided work, $4,000 of tokens, $30,000 of engineering, $3 to review a task, $20 per escaped failure. At 80% success, full review leaves $56,000. Sampling 10% saves $27,000 of review but lets 1,800 failures through at $36,000: $47,000 (break-even, $15 a failure). At 97% success, full review leaves $81,500 and sampling about $103,000, since only 270 escape. The difference is the eval suite.

Tie the board to the P&L, not to anecdotes. McKinsey's 2026 survey, per The Register, found 80% of AI users reporting better personal productivity while the share attributing EBIT impact to AI stayed flat at 37%.³

When it fires anyway

An agent with write access will eventually do something nobody approved. This is the book's incident runbook (Ch 4 points here). The CSA framework lists the controls to have first: a manual kill switch that acts in under a second, a circuit breaker that halts on repeated failures, revocation of every agent session, and state rollback.¹⁰

The whole harness, on one page

Take this into the design review; every box links to its evidence.

The book started from Vivek Trivedy's equation, Agent = Model + Harness, the harness being “every piece of code, configuration, and execution logic that isn't the model itself.”²⁹ Five chapters took the harness apart; Part 06 added the people and paperwork that let it run on a Tuesday with nobody watching. You rent the model and own the harness, and operating it takes a thin platform, embedded owners, autonomy earned by measured gates, a pack reviewers can sign, and a board that ties reliability to money.

If you're taking an agent to production and want to compare notes on the harness, get in touch.