Accuracy versus latency

“Latency” is at least two numbers

Anthropic’s reducing-latency guide separates baseline latency — the time to process the prompt and generate the whole response — from time to first token (TTFT), the wait before anything appears. They respond to different levers. A chat user staring at a spinner cares about TTFT; a back-end service that must return a JSON decision before an HTTP timeout cares about total time. A voice agent cares about both. Always ask which one the requirement is really about, and state it as a percentile (p95 or p99), because the slow tail is what breaks an SLA.

Total time is roughly input processing, plus thinking, plus output tokens divided by output speed, plus any tool round trips. That decomposition tells you where each lever acts. Shorter or cached input helps TTFT. Less thinking and shorter output help total time. Fewer tool round trips help both. Output speed is fixed per model unless you pay for fast mode.

Free speed first: levers that cost no accuracy

Some levers change what users experience without touching what the model does. Streaming sends tokens as they are generated, so a two-second answer starts appearing almost at once. Prompt caching skips reprocessing of a stable prefix, which shortens TTFT on long system prompts and documents (covered in 2.5). Parallelism in your own code — running independent retrievals or tool calls at the same time — removes waits the model never needed. And trimming input that does not affect the answer shortens processing and can even help accuracy (covered in 2.4). An architect who proposes a smaller model before trying these is paying for speed with quality when it could have been had for free.

Paid-for speed: levers that can cost accuracy

Effort is usually the first of these to try. It governs every output token — text, tool calls and thinking — and defaults to high. The effort docs describe lower effort as combining operations into fewer tool calls and skipping preamble, and higher effort as more tool calls, more planning and more thinking. Sonnet 5 at medium is described as comparable to Sonnet 4.6 at high, and low is recommended for high-volume or latency-sensitive work. Two cautions: effort is behavioural, not a hard budget, and on Opus 5 the docs note that lowering effort does not reliably shorten responses — ask for length in the prompt as well.

Model tier is the bigger step. The models overview lists comparative latency from Fable 5.1 (slower) through Opus 5 (moderate) and Sonnet 5 (fast) to Haiku 4.5 (fastest). Moving down a tier is justified when an eval shows the smaller model still meets the bar on the traffic it will serve (selection is covered in 2.1). Routing combines both: send routine requests to a fast configuration and escalate hard ones to a slower, stronger one, so the average is fast and the hard tail is still right.

Some additions do the opposite: buy accuracy with time. Retrieving more chunks, adding a verification pass, or fanning work out to several agents usually raise quality and always add latency and cost. Each should earn its place with a measured improvement, just as each speed-up must show it did not break quality.

import time, statistics

def run(config, case):
    start = time.perf_counter(); ttft = None
    with client.messages.stream(max_tokens=2000, messages=case.messages,
                                **config) as stream:
        for _ in stream.text_stream:
            ttft = ttft or time.perf_counter() - start   # first visible token
        msg = stream.get_final_message()
    return grade(case, msg), ttft, time.perf_counter() - start, msg.usage

CONFIGS = {
    "sonnet-high": {"model": "claude-sonnet-5"},
    "sonnet-low":  {"model": "claude-sonnet-5", "output_config": {"effort": "low"}},
    "haiku":       {"model": "claude-haiku-4-5"},
}
for name, cfg in CONFIGS.items():
    rows = [run(cfg, c) for c in EVAL_CASES for _ in range(3)]   # 3 trials each
    totals = sorted(r[2] for r in rows)
    print(name,
          "pass", statistics.mean(r[0] for r in rows),
          "p95 total", totals[int(0.95 * len(totals)) - 1],
          "median ttft", statistics.median(r[1] for r in rows))

Justifying the decision

A configuration is justified when a reviewer can see the targets, the alternatives, the evidence and the cost of the choice. Anthropic’s evals post supplies the vocabulary: run each task for several trials, because outputs vary; grade with code where you can and with a model or human where you must; and track latency, tokens and cost per task alongside quality. It also distinguishes pass@k (at least one of k tries succeeds) from pass^k (all k succeed) — for a customer-facing path, consistency is what matters, so pass^k is the honest measure. And it recommends reading transcripts: a configuration that is fast because it skips a tool call it needed will look fine on latency and wrong in the transcript.

Traps the wrong answers are built from

You should now be able to

• Distinguish time to first token from total latency and state targets as percentiles.
• Map each configuration lever to its effect on latency, accuracy and cost.
• Apply accuracy-neutral levers (streaming, caching, trimming, parallelism) before accuracy-reducing ones.
• Use effort, model tier and routing to trade accuracy for speed only where evals allow.
• Run a configuration sweep with multiple trials and report quality, latency and cost together.
• Write a decision record that states targets, alternatives, evidence and re-evaluation triggers.