Monitoring with logs and usage data

This lesson is about the practice of monitoring a Claude system so you can evaluate and improve it: what to record, which Anthropic tools report what, and how signals turn into fixes. Designing observability architecture for large, distributed deployments — tracing across services, sampling strategy at scale — is 3.4.

What every log line should carry

Anthropic's engineering team lists production monitoring as the layer that reveals real behaviour at scale and catches what synthetic evals missed — reactive and noisy, and only useful if the system is instrumented. The instrumentation is your application's log. The API gives you most of what you need in each response; your code adds the context the API can't know.

import json, time, logging
log = logging.getLogger("claude")

def call_claude(prompt_version: str, variant: str, **request):
    start = time.monotonic()
    r = client.messages.create(**request)
    u = r.usage
    log.info(json.dumps({
        "request_id": r._request_id,           # also in the request-id header
        "model": request["model"],
        "prompt_version": prompt_version,      # e.g. "support-v14"
        "variant": variant,                    # A/B arm, if any
        "stop_reason": r.stop_reason,
        "input_tokens": u.input_tokens,        # after the last cache breakpoint
        "cache_write": u.cache_creation_input_tokens,
        "cache_read": u.cache_read_input_tokens,
        "output_tokens": u.output_tokens,
        "latency_ms": round((time.monotonic() - start) * 1000),
    }))
    return r

From log lines to the live scorecard

The metrics you defined for release (4.1) are the metrics you watch in production. Aggregate them by model, prompt version and variant, as percentiles where they're times, and as rates where they're events. A few derived signals earn a place on every dashboard:

• Stop-reason mix. Track stop_reason values: a jump in max_tokens means truncated answers; a jump in refusal means users or a change are hitting safeguards; more tool_use turns per task can mean an agent is looping.
• Cost per successful task. Spend from usage divided by tasks your outcome signal marks as resolved — not cost per request.
• Cache hit rate. Cache-read tokens as a share of all input tokens. A sudden fall usually means something now changes at the top of the prompt (4.5).
• Errors by type. 429 rate limits, 529 overloads and 500s need different responses; the SDKs already retry transient failures twice by default, so count what still fails after retries.
• p50 and p95 latency, and TTFT for streamed responses.

Watching quality, not just health

Error rates and latency tell you the system is up. They don't tell you it's right. To monitor accuracy and safety in production, sample real transcripts every day or week and run the same graders you use offline — the code checks, and the calibrated model judge from 4.2 — on them. Add a human review queue for flagged and low-scoring cases, and treat explicit user feedback as a signal rather than a census: Anthropic's engineers note it is sparse, self-selected and skewed towards severe problems.

Security needs its own watch. Anthropic's jailbreak guidance recommends continuously analysing outputs for signs of successful injection, and noticing users who repeatedly trigger the same refusals — then adjusting responses, throttling or banning as policy allows. That only works if refusals and screen results are logged with a user or session identifier.

Anthropic's monitoring tools, and what each is for

Usage & Cost Admin API. Two endpoints give organisation-level history: /v1/organizations/usage_report/messages for tokens (uncached input, cache writes, cache reads, output, and server-tool use), bucketed by 1m, 1h or 1d and filterable or groupable by API key, workspace, model, service tier and more; and /v1/organizations/cost_report for USD costs, daily only, groupable by workspace or description. It needs an Admin API credential, not a normal workspace key. Data typically appears within about five minutes, and polling once a minute is supported for sustained use. Priority Tier costs aren't in the cost endpoint. Similar data appears on the Console's Usage and Cost pages, and Anthropic lists ready-made integrations with platforms such as Datadog, Grafana Cloud and Honeycomb.

Claude Code telemetry. For teams using Claude Code, setting CLAUDE_CODE_ENABLE_TELEMETRY=1 with OpenTelemetry exporters sends metrics such as claude_code.token.usage, claude_code.cost.usage and claude_code.session.count, and events such as claude_code.api_request, claude_code.api_error and claude_code.tool_decision, to any OTLP-compatible backend. Administrators can push the configuration through managed settings so developers can't redirect it. For per-user cost breakdowns, the docs point to the separate Claude Code Analytics API.

Traps the wrong answers are built from

You should now be able to

• Specify the fields a per-request log needs, including request ID, versions, usage, stop_reason and timing.
• Derive production metrics — stop_reason mix, cost per successful task, cache hit rate, latency percentiles — from logs.
• Set up online evaluation by sampling production transcripts through the same graders used offline.
• Choose between application logs, the Usage & Cost Admin API, Console pages and Claude Code OpenTelemetry for a monitoring question.
• Close the loop from alert to diagnosis to a new eval case.