Context windows and token budgets

Why a bigger window is not the answer

Current flagship models offer a 1M-token context window at standard pricing, and Haiku 4.5 offers 200K. It is tempting to read that as “we no longer need to manage context”. Anthropic’s guidance says the opposite, for three reasons an architect should be able to state.

• Cost scales with volume. The whole context is input on every request. A 300K-token history re-sent across forty agent turns is twelve million input tokens, even if each turn asks a one-line question. Caching lowers the rate but not the volume (caching is covered in 2.5).
• Attention degrades. The context-engineering post describes “context rot”: as the token count grows, the model’s ability to recall information from context falls. It treats context as a finite resource with diminishing marginal returns — an attention budget, not just a size limit.
• Latency rises. More input means more to process before the first output token, on every call.

The post’s guiding principle is to find the smallest set of high-signal tokens that maximise the chance of the outcome you want. That turns token management into a design discipline with four moves: measure what is in context, curate what goes in, trim what has gone stale, and offload what must be kept but need not be read every turn.

Measure before you optimise

Every Messages API response reports a usage object with input and output tokens, so you can log exactly what each turn cost. Before sending, the token counting endpoint (client.messages.count_tokens) accepts the same system prompt, messages, tools, images and PDFs as a real request and returns an estimate. It is free, though rate-limited separately from message creation. Use it to reject or trim an oversized request before it fails, and to decide when to summarise.

Know the two overflow behaviours. If the input alone exceeds the window, the API returns a 400 “prompt is too long” error. If the input fits but input plus max_tokens does not, Claude 4.5 and later models accept the request and stop generation with stop_reason: "model_context_window_exceeded" — a truncated answer your code must not treat as complete.

Curate what goes in

The cheapest token is one you never send. Curation happens at design time. Keep the system prompt at the “right altitude” (covered in 2.2). Send only the tools a workflow needs: the manage-tool-context page suggests tool search once a toolset passes about twenty tools, so definitions load on demand. Prefer just-in-time retrieval — keeping lightweight identifiers such as file paths, record IDs or URLs in context and loading content only when needed — over pasting every possibly relevant document up front (the full trade-off is covered in 3.8). And preprocess noisy inputs in code: the Claude Code cost guide’s example is a hook that greps a 10,000-line log for errors, cutting tens of thousands of tokens to hundreds.

Trim and summarise: context editing and compaction

Long-running agents accumulate material that was useful once. Anthropic’s engineering post calls clearing old tool results the “lightest touch” form of compaction: once a tool call is deep in history, the agent rarely needs to see the raw result again. The API offers this server-side as context editing, and a heavier option, compaction, that summarises instead of deleting.

count = client.messages.count_tokens(
    model="claude-opus-5", system=SYSTEM, tools=TOOLS, messages=messages,
)
if count.input_tokens > 900_000:        # near the 1M window
    raise ContextTooLarge(count.input_tokens)   # trim or summarise first

response = client.beta.messages.create(
    model="claude-opus-5",
    max_tokens=8000,
    system=SYSTEM,
    tools=TOOLS + [{"type": "memory_20250818", "name": "memory"}],
    messages=messages,
    betas=["context-management-2025-06-27"],
    context_management={"edits": [{
        "type": "clear_tool_uses_20250919",
        "trigger": {"type": "input_tokens", "value": 60000},
        "keep": {"type": "tool_uses", "value": 4},     # recent results stay
        "clear_at_least": {"type": "input_tokens", "value": 20000},
        "exclude_tools": ["memory"],                   # never clear notes
    }]},
)
# Log what was cleared, for cost and debugging
print(response.context_management)

Three design details carry the marks. First, context editing is applied on the server before the prompt reaches Claude, so your client keeps the full history and does not have to sync an edited copy. Second, clearing invalidates the prompt cache from the point of the edit, which is why clear_at_least exists: clear a meaningful amount at once rather than a little every turn. Third, pair clearing with the memory tool, so Claude can record what matters before old results disappear and look it up later.

Compaction is for continuity. When input tokens reach the trigger, the API generates a summary, returns it as a compaction block, and on the next request drops everything before that block. Custom instructions replace — not supplement — the default summarisation prompt, so write them to preserve what your domain needs: decisions, identifiers, open questions. pause_after_compaction stops the response with stop_reason: "compaction" so you can inject context or enforce an overall token budget. The summary itself is an extra, billed sampling step, reported in usage.iterations; top-level usage covers only the non-compaction work, so cost dashboards must sum the iterations.

Traps the wrong answers are built from

You should now be able to

• Identify what consumes the context window and measure it with usage and token counting.
• Explain why larger contexts raise cost and latency and can lower accuracy (context rot).
• Choose between tool-result clearing, thinking clearing, compaction, memory and subagents.
• Configure server-side context editing and compaction with appropriate triggers and instructions.
• Handle context overflow correctly, including model_context_window_exceeded.
• Curate inputs by trimming tools, preprocessing results and retrieving just in time.