Observability at scale

Why Claude systems are hard to observe

Anthropic’s multi-agent research post is candid about the problem: agents behave differently across runs with identical prompts, small failures compound over long runs, and debugging without full production tracing is guesswork. At scale, six challenges recur, and each one points to a strategy.

Two of these deserve emphasis. Silent quality failure is what makes LLM monitoring different in kind: the transport layer succeeds, so every classic health check stays green. And cost volatility is real — the same post reports that multi-agent research used about fifteen times the tokens of a chat interaction, and that token usage alone explained most of the variance in quality. A cost spike may be a bug, or it may be the system doing more work because the task demanded it; only per-run data can tell you which.

Signal by signal: what to capture

Metrics answer “is something wrong?” cheaply, for every request. Track latency percentiles (time to first token and total), error rates by type — 429 rate limiting, 529 overloaded, 5xx, and errors that arrive mid-stream after a 200 — and tokens split into uncached input, cache reads, cache writes and output. Track the distribution of stop_reason: a rise in max_tokens or model_context_window_exceeded means truncated answers, and a rise in refusal means something changed in inputs or prompts. Rate-limit headers such as anthropic-ratelimit-input-tokens-remaining show headroom before you hit a wall. Keep metric labels low-cardinality — model, feature, tenant tier — so storage stays affordable.

Traces answer “what happened in this run?”. For agents built on the Claude Agent SDK, the CLI it runs can export OpenTelemetry spans: claude_code.interaction for a turn, claude_code.llm_request for each API call with model, latency and tokens, and claude_code.tool for each tool call, with subagent spans nested under the parent’s tool span so a whole delegation chain is one trace. The SDK propagates W3C trace context (TRACEPARENT), so the agent’s spans appear inside your application’s own trace rather than as a disconnected root. Tracing is enabled with CLAUDE_CODE_ENHANCED_TELEMETRY_BETA=1 and is in beta.

Events and logs answer “exactly which request?”. Every API response carries a request-id header (the Python SDK exposes it as _request_id), which support needs and which joins your logs to Anthropic’s. Log it with the model, prompt and module versions, effort, stop_reason and usage. In Claude Code and the Agent SDK, events such as tool_result, api_request, api_error and tool_decision share a prompt.id that links everything one prompt caused.

from opentelemetry import trace
tracer = trace.get_tracer("claims-assistant")

def ask(messages, tenant, feature):
    with tracer.start_as_current_span("claude.request") as span:
        span.set_attribute("app.tenant_tier", tenant.tier)   # low cardinality
        span.set_attribute("app.feature", feature)
        span.set_attribute("app.prompt_version", PROMPT_VERSION)
        resp = client.messages.create(model=MODEL, max_tokens=2000,
                                      system=SYSTEM, messages=messages)
        u = resp.usage
        span.set_attribute("claude.request_id", resp._request_id)
        span.set_attribute("claude.stop_reason", resp.stop_reason)
        for key in ("cache_read_input_tokens", "cache_creation_input_tokens",
                    "input_tokens", "output_tokens"):
            span.set_attribute(f"claude.usage.{key}", getattr(u, key) or 0)
        # No prompt or answer text here: content goes only to an
        # approved, access-controlled store, if anywhere.
        return resp

Scale: sample, attribute, and keep content under control

At millions of requests, keeping every trace in full is expensive and rarely useful. A common pattern is to keep metrics for everything, keep full traces for every error, truncation, refusal, unusually slow or unusually expensive run, and sample the healthy remainder. Quality needs its own sample: route a slice of production traffic to an automated grader — code checks where possible, a model-based rubric where necessary — and a smaller slice to human reviewers, as the evals post describes for offline testing. Read transcripts of failures; the evals post notes that reading them is what shows what the agent got wrong and why.

Attribution turns cost from a surprise into a signal. Tag every request with the tenant, feature and prompt version that caused it. In the Agent SDK, end-user and tenant identity can be added as OTEL_RESOURCE_ATTRIBUTES, because by default identity attributes describe your service’s credential, not the person the agent acted for — which also turns tool and permission events into a per-user audit trail for a SIEM. For the organisation’s authoritative numbers, the Usage and Cost Admin API reports usage grouped by model, workspace, API key and more, typically within about five minutes, and daily cost in USD. Use near-real-time telemetry for alerts and the Admin API to reconcile against the bill.

Content is the hardest call. The Agent SDK’s telemetry is structural by default: durations, model names, tool names and token counts, but not prompts, tool arguments, tool output or API bodies — each of those needs its own opt-in variable, and the docs advise leaving them off unless your pipeline is approved to store that data. Anthropic’s own research system monitors agent decision patterns and interaction structures without monitoring conversation contents. Follow that default: prove you need content for a specific diagnosis, capture it for a sample, redact it, restrict access and set retention (governance is covered in domain 5).

Traps the wrong answers are built from

You should now be able to

• Explain why non-determinism, silent failures and multi-step runs make LLM systems hard to observe.
• Select metrics, traces, events and sampled grading to match each observability question.
• Propagate trace context so agent model and tool calls appear inside application traces.
• Design cost attribution with request tags, workspaces and the Usage and Cost Admin API.
• Control sensitive content in telemetry with structural defaults and opt-in, approved capture.
• Keep observability affordable at scale with sampling and low-cardinality metrics.