Context

Give Claude 3.5 Sonnet a short passage and a question whose answer shares no keywords with it, and it finds the answer 87.5% of the time. Pad the passage out to 32K tokens, a sixth of the 200K the model advertises, and the score falls to 29.8%.¹ The task did not change. Only the amount of text around it did. The gap between what a window can hold and what a model can actually use is the subject of this chapter, and managing that gap is most of what a harness does.

The job is context, not prompts

In mid-2025 several groups settled on the same name for this work within weeks of each other. Karpathy called it the art and science of filling the window with just the right information for the next step, and warned that too much, or the wrong kind, raises cost and lowers quality.^2,3 Cognition put context engineering at the core of agent reliability.⁴ Manus published the lessons of building its agent around it.⁵ Anthropic defined it as curating and maintaining the best set of tokens during inference, with prompt writing as one subset of the job.⁶

The rename matters because it moves the unit of work. A prompt is written once. Context is assembled on every turn of the loop from Chapter 1: instructions, tool definitions, files, earlier tool results, the model's own previous output. Anthropic's documentation describes the harness itself as the layer that supplies tools and manages what the model sees.⁷ A prompt is a sentence. Context engineering is a policy that runs fifty times per task.

The advertised window is not the usable window

Every serious long-context benchmark finds the same thing: accuracy falls well before the limit on the spec sheet, and often by 8K to 32K tokens.

RULER tested 17 models on 13 tasks. All claimed at least 32K of context; only about half held up at 32K.⁸ NoLiMa removed literal word overlap between question and answer, so the model had to make a one-step association instead of string matching. At 32K, 11 of the 13 models it tested scored below half their short-context baseline.¹ A Databricks study ran 20 models from 2K to 128K in a retrieval setting and found that only a handful of the newest stayed consistent past 64K; the rest failed in their own ways, from refusals to dropped instructions.⁹ Chroma's study of 18 models, GPT-4.1, Claude 4 and Gemini 2.5 among them, saw decline with length even on copying a run of repeated words. On LongMemEval, a focused prompt of about 300 tokens beat the full one of roughly 113K.¹⁰

Position matters as much as length. Lost in the Middle gave the curve its shape: with 20 documents, GPT-3.5-Turbo answered correctly 75.8% of the time when the relevant document came first, 53.8% when it sat tenth, and 63.2% when it came last. With no documents at all it scored 56.1%.¹¹ Burying the answer in the middle made the model slightly worse than giving it nothing.

Anthropic frames the cause as an attention budget. Every token attends to every other, so relationships grow with the square of length, and training data skews toward short sequences.⁶ Chroma named the symptom: context rot.¹⁰ The gauge below plots NoLiMa's numbers. Drag the scanline and watch where each model leaves its own 85% line.

These models are a generation or more old; NoLiMa is the cleanest controlled measurement, not the newest. Chroma's 2025 run on newer models, Claude 4 and Gemini 2.5 among them, found the same decline with length.¹⁰ Windows have grown since: as of October 2026, Claude Code's current models offer 1M tokens.¹² A larger window raises the point where the API refuses input. None of these studies shows it moving the point where accuracy starts to fall, which is why you measure that yourself.

Four ways context fails

Drew Breunig sorted long-context failures into four kinds.¹⁴ The taxonomy is useful because each one has a different fix, and applying the wrong one wastes a week.

Clash deserves attention because it is the one harnesses cause. In the sharded-prompt study Breunig cites, splitting the same task's information across several turns cost 39% on average.¹⁴ An agent that writes exploratory drafts into its main thread is manufacturing clash.

Budget it like RAM

Karpathy's framing holds up: the model is the CPU and the context window is RAM.³ MemGPT made the analogy literal, with a main context the model reads directly and an external store it pages data in from.¹⁵ The practical consequence is to split cost into two lines: fixed overhead and per-step spend.

Fixed overhead is everything loaded before the user types. In Claude Code that means the system prompt, CLAUDE.md, the first 200 lines or 25KB of auto memory, skill descriptions and MCP tool names.^12,7 MCP stays that small because tool search is on by default: only tool names and server instructions load at startup, and a full definition is fetched when a task needs that tool. On models older than the Claude 4 family, and on deployments that reject tool search, every definition loads upfront.¹⁶ Chapter 3 shows what eager loading costs. In your own loop the same pattern is a catalog of names plus a search tool that returns full schemas on demand. Each of those tokens is paid on every turn of every session. Per-step spend is file reads, tool output and the model's own reasoning.

Progressive disclosure keeps overhead small. Agent Skills load in three levels: name and description up front, the body only when relevant, bundled files only when read.¹⁷ The same discipline applies to instruction files. Anthropic's test for each line of CLAUDE.md is whether deleting it would cause mistakes; bloated files make the model ignore the instructions that matter.¹⁸

When spend does pile up, cut in order of cost. Claude Code clears older tool outputs first and summarizes only if that is not enough.⁷ At the API level, tool-result clearing triggers at 100,000 input tokens by default and keeps the last three tool uses.¹⁹ Anthropic measured context editing alone at +29% on an agentic search eval, +39% with a memory tool added, and 84% fewer tokens over a 100-turn web search run.²⁰ As of October 2026, Claude Code compacts by default when the conversation reaches the model's limit, or at about 967K tokens on models with a native 1M window, and /autocompact sets an earlier trigger.¹³

Compaction is lossy, so decide what survives it. After compacting, Claude Code re-reads up to five recently modified files (larger ones come back as paths), re-injects skill bodies capped at 5,000 tokens each and 25,000 in total, and reloads CLAUDE.md and memory from disk. Nested instruction files and path-scoped rules are summarized away until Claude next reads a file that triggers them, and SessionStart hooks matching the compact source run again and add their output.¹² Anthropic's advice for writing a summarizer is to tune for recall first, then precision, keeping decisions, open bugs and implementation details.⁶

Copy this · a minimal compaction contract: a handoff file the agent rewrites at each milestone, and a Claude Code hook that re-injects it after every compaction.²¹

.agent/handoff.md · the agent rewrites it at each milestone; keep it under ~40 lines

## Goal
One sentence on what "done" means and the check that proves it
(e.g. `npm test` green and the repro in issue #123 passes).

## Decisions (do not relitigate)
- Chose X over Y because Z.

## Open bugs and unknowns
- [ ] auth/session.test.ts fails about 1 run in 5; cause unknown.

## Files touched
- src/auth/session.ts: refresh logic rewritten

## Next step
Exactly one action the next context takes first.

.claude/settings.json · re-injects the contract after compaction; the absolute path survives the agent changing directory

{
  "hooks": {
    "SessionStart": [
      { "matcher": "compact",
        "hooks": [{ "type": "command", "command": "cat \"${CLAUDE_PROJECT_DIR}/.agent/handoff.md\"" }] }
    ]
  }
}

In another harness or your own loop, prepend the file to the first message after any summarization step. The contract only works if the loop, not the model, decides that it is re-read.

Move it out, pull it in

The filesystem is the cheapest memory tier because it is restorable. The lightest version is a notes file the agent writes and rereads.⁶ Manus has its agent rewrite a todo list every step, which pushes the goals to the recent end of context across tasks that average about 50 tool calls. It treats files as unlimited, persistent and directly operable, and compresses only in ways it can undo: keep the URL, drop the page.⁵

Heavier systems formalize the tiers. Letta splits memory into core blocks the agent edits in context, searchable recall history, and an archival store, and on overflow evicts about 70% of messages behind a recursive summary.²² Mem0 extracts and consolidates facts; on the LOCOMO benchmark it reported a 26% judge-score gain over OpenAI's memory, 91% lower p95 latency and over 90% fewer tokens than replaying full history.²³

Retrieval has the same push-or-pull choice.

“Claude Code doesn't use RAG currently. In our testing we found that agentic search out-performed RAG for the kinds of things people use Code for.”— Boris Cherny, Hacker News (2025)²⁴

Anthropic calls the pattern just-in-time context: hold lightweight identifiers such as paths, URLs and queries, and load the content only when a step needs it, with upfront retrieval kept for cases where speed matters.⁶

Subagents push the same idea further: they spend tokens in a separate window and return only a short summary, and read-only investigators reporting to one decision-making thread are the safe form. Chapter 1 covers what fan-out costs and where it breaks.

Pay attention to the cache

The cheapest token is one already sitting in the KV cache.

“the single most important metric for a production-stage AI agent”— Yichao “Peak” Ji, Manus, on KV-cache hit rate (2025)

Agent loops read far more than they write. Manus reports input-to-output ratios around 100:1, so the input price dominates the bill.⁵ At list prices as of October 2026, Sonnet 4.5 input costs $3 per million tokens, a cache read $0.30, a five-minute cache write $3.75, a one-hour write $6, and output $15.²⁵ The instrument below uses Sonnet 4.5 because its multipliers, 1.25× base for a write and 0.1× for a read, are the ones most models carry. Newer models discount reads further: Sonnet 5.5 lists input at $2 and bills reads at 5% of it.²⁵ The cache matches on prefix in the order tools, then system, then messages, and any change invalidates everything after it. Sonnet 4.5, now deprecated, needs at least 1,024 tokens to cache at all.²⁵

That makes cache discipline an architecture rule, not a style preference. Keep timestamps out of the system prompt, append instead of editing history, serialize deterministically, and mask tools rather than removing their definitions mid-session.⁵ It also explains a tension from Section 4: clearing old tool results breaks the cached prefix, which is why the API lets you require a minimum amount cleared per edit so the rewrite pays for itself.¹⁹

Run the numbers and the shape is clear. A cache miss is billed as a write at 1.25× the base input price,²⁵ so if every miss rewrote the whole prefix, hit rates below about 22% would lose money. Past that point the 10× read discount dominates, and with output a hundredth of input, the session cost follows the hit rate almost exactly. Treat the 22% as a worst-case bound. In a healthy loop each turn reads the cached prefix and writes only the new suffix, so the hit rate is high by construction.

Two things break that in practice. The first is expiry. The default cache lives five minutes, measured from the start of the request, and each hit refreshes it for free. A test suite, a build or a human approval that runs past the window means the next turn rewrites the whole prefix at 1.25×. A one-hour cache costs 2× base to write and pays off after two reads; Anthropic suggests it for side agents that run longer than five minutes.²⁵ The second is invalidation. Changing a tool definition invalidates the whole cache; changing the system prompt, or switches such as web search, invalidates the system and message layers.²⁵ A mid-session edit to either is a full rewrite. On the newest models you can append a system-role message instead of editing the top-level system field, and the cached prefix stays intact.²⁵ In your own loop, log cache-read and cache-write tokens per turn: a turn whose write is close to the whole prefix is the alarm.

Part 06 covers cost per task and model routing across a whole program.

What fills the window next

The pattern across this chapter is that a long context is a liability you manage, not a resource you fill. Keep fixed overhead small, cut stale results before summarizing, put durable state in files, pull content in on demand, and protect the cached prefix.

Most of the per-step spend, though, is tool output: file reads, search hits, command logs. How much of it lands in the window depends on what tools the agent has, how they are described and what they return. That is Chapter 3.