Defining evaluation metrics

What makes a metric usable

Anthropic's guidance on success criteria asks for four properties. Specific: name the task, not a mood — “accurate sentiment classification”, not “good performance”. Measurable: a number, or a qualitative scale applied consistently alongside numbers. Achievable: grounded in benchmarks, prior experiments or expert knowledge, and not beyond what current frontier models can do. Relevant: tied to what users need — citation accuracy matters enormously in a medical tool and much less in a casual chatbot.

The docs also make a point architects often miss: even “hazy” qualities such as safety can be quantified. “Safe outputs” is not a criterion. “Fewer than 0.1% of outputs across 10,000 trials flagged for toxicity by the content filter” is.

The model should classify
sentiment well.

It should be fast,
safe, and affordable.

On a held-out set of 10,000
diverse posts:
- F1 at least 0.85
- 99.5% of outputs non-toxic
- 90% of errors cause
  inconvenience, not harm
- 95% of responses < 200 ms
- cost per 1,000 posts
  under an agreed budget

The first four lines of the measurable version follow Anthropic's own sentiment-analysis example; the cost line is the one most teams forget to add. Note the footnote the docs attach to it: you would still need to define what “inconvenience” and “egregious” mean. A threshold on an undefined category is not yet a metric.

The five dimensions, made concrete

The docs' own list of common criteria is broader — task fidelity, consistency, relevance and coherence, tone and style, privacy preservation, context utilisation, latency and price. Most of those roll up into the five headings this task statement names; consistency and tone are usually part of accuracy, privacy sits under safety. Anthropic's summary is blunt: most use cases need multidimensional evaluation.

Accuracy: outcome, not appearance

For a classifier, accuracy is simple: compare the label to the answer key. For an agent it is harder, and Anthropic's engineering team is precise about the vocabulary. A task is one test with inputs and success criteria; a trial is one attempt at it; the transcript is everything that happened; the outcome is the final state of the world. A travel agent that says “your flight is booked” has produced a transcript. Whether a reservation exists in the database is the outcome — and that is what accuracy should check.

Because model output varies, agents are run several times per task, and two metrics answer different questions. pass@k is the chance that at least one of k trials succeeds; it rises with k and suits tools where a person picks the best of several drafts. pass^k is the chance that all k succeed; it falls with k and suits customer-facing agents that must work every time. A 75% per-trial success rate gives pass^3 of about 42%.

p = 0.75          # per-trial success rate on a task
k = 3

pass_at_k  = 1 - (1 - p) ** k   # at least one of k succeeds
pass_hat_k = p ** k             # all k succeed

print(f"pass@3 = {pass_at_k:.0%}")   # 98% — "it nearly always works"
print(f"pass^3 = {pass_hat_k:.0%}")  # 42% — "three customers in a row?"

Pick the one that matches how the system is used, and say which one you picked. Reporting pass@k for a refund agent that must get every customer right is the metric equivalent of rounding up.

Latency and cost: tails and denominators

Anthropic's latency guide distinguishes baseline latency (overall time to process the prompt and generate the answer) from time to first token (TTFT), which is what a user watching a streamed reply actually feels. Define both, and define them at a percentile. An average of 1.2 seconds can hide one request in twenty taking nine seconds — and the user who got the nine-second answer does not care about the average. The docs' own example uses “95% of responses under 200 ms”, which is a p95 target.

Cost needs the right denominator. Price per million tokens is an input, not a metric: models differ in price, in how many tokens they spend on the same answer, and even in tokenizer — the pricing page notes that Claude 4.7 and later models use a tokenizer producing roughly 30% more tokens for the same text. What the business pays for is finished work, so measure cost per successful task: total spend divided by tasks that passed your accuracy check. A cheaper model that fails more often, or needs a retry, can cost more per success.

# Claude Sonnet 5, USD per million tokens (pricing page, Sep 2026)
PRICE = {"input": 2.00, "cache_write_5m": 2.50, "cache_read": 0.20, "output": 10.00}

def request_cost(u) -> float:
    return (u.input_tokens * PRICE["input"]
          + u.cache_creation_input_tokens * PRICE["cache_write_5m"]
          + u.cache_read_input_tokens * PRICE["cache_read"]
          + u.output_tokens * PRICE["output"]) / 1_000_000

def cost_per_success(runs) -> float:
    spend = sum(request_cost(r.usage) for r in runs)   # failed runs still cost money
    wins = sum(1 for r in runs if r.passed)
    return spend / wins if wins else float("inf")

Safety and security: measure both directions

Safety metrics count bad outputs reaching ordinary users: toxic content, unsafe advice, personal or health information that should not appear. The docs' medical example is a binary model-graded check — does this response contain PHI, yes or no — run over a test set that includes explicit, hypothetical and implicit PHI. Measure the opposite error too: a system that refuses legitimate requests is failing its users, and a safety metric that only counts harm will reward a system that refuses everything.

Security metrics assume an adversary. Anthropic separates two threat models: jailbreaks and direct injection, where the user is the attacker, and indirect injection, where instructions hide in a web page, email or tool result the model reads. Each needs its own adversarial test set, and the metric is the attack success rate — ideally judged by what the system did (did it call a tool the user never asked for?) rather than only by what it wrote. The docs recommend red-teaming your own agent with documents and tool outputs that deliberately contain injection attempts before deploying.

Building the datasets and harness that produce these numbers is covered in 4.2; comparing two versions against them in 4.3. Production monitoring of the same metrics is 4.6.

Traps the wrong answers are built from

You should now be able to

• Rewrite a vague goal into specific, measurable, achievable and relevant criteria with thresholds.
• Choose an appropriate metric and grader for each of accuracy, latency, cost, safety and security.
• Explain when to report pass@k versus pass^k for an agent.
• Compute cost per successful task from the API's usage fields and the price list.
• Distinguish safety metrics (harmful output to ordinary users) from security metrics (success of deliberate attacks).