Rubric
Contents — domains, guide and mocks

Evaluation datasets and test frameworks

CCAR-P 4.216 min read · checked 21 September 2026

Task statementDesign evaluation datasets and test frameworks using mixed methodologies

How an evaluation suite is built and kept alive

  1. Collect casesreal failures, logs, experts, edge cases
  2. Label and splitreference answers; dev vs held-out
  3. Grade each trialcode, model and human graders
  4. Read transcriptsare the failures fair?

New failures from review or production become new test cases

The loop matters as much as the steps: every failure found in review or production becomes a new test case.

Where the test cases come from

Anthropic's guidance on evals starts with one principle: be task-specific. The dataset should mirror the real distribution of inputs the system will see, edge cases included. A set of tidy examples someone wrote on day one measures how well the system handles tidy examples.

Anthropic's engineering team recommends starting early and small: 20–50 simple tasks drawn from real failures beats waiting until you have hundreds. Good sources are the checks people already do by hand before a release, and the bug tracker and support queue — each reported failure is a ready-made test case with a known wrong answer.

Slice of the datasetExample (support assistant)Why it's there
Representative trafficA sample of last month's real questions, anonymisedKeeps the score honest about typical use
Known failuresTickets where the old bot gave a wrong refund policyProves the fix, then guards against regression
Edge casesEmpty input, very long input, typos, ambiguous asksThe docs list these as cases to include deliberately
Should-not casesQuestions that must be declined or escalatedBalances the set so over-triggering is caught too
AdversarialEmails and documents with embedded instructionsMeasures security, not just quality

Two quality rules apply to every case. First, it should be unambiguous: Anthropic's test is that two domain experts would independently reach the same pass or fail verdict. Second, it should be solvable — writing a reference answer or solution proves that. If experts disagree about a case, the case is measuring your specification, not your system.

Balance matters as much as coverage. Anthropic describes building web-search evals for Claude.ai in both directions: questions that need a search (a weather forecast) and questions that don't (who founded a well-known company). An eval that only tests “should search” rewards a system that searches for everything.

Splits and suites

Keep a held-out set that nobody tunes prompts against. If you iterate on the same fifty cases for a month, the prompt learns those fifty cases, and the score stops telling you how it will do on the fifty-first. Use a development set for iteration, and check the held-out set when deciding to ship.

Anthropic also separates two kinds of suite. A capability eval asks what the system can do well and starts with a low pass rate — it is a target. A regression eval asks whether it still does everything it used to, and should sit near 100%; any drop is a break. Capability tasks graduate into the regression suite once they pass reliably. Watch for saturation: when the system passes every solvable task, the suite can no longer show improvement and needs harder cases.

Two suites, two questions

Capability suite

  • Asks “what can it do well?”
  • Starts with a low pass rate
  • Used to steer improvement work
  • Needs harder cases once saturated

Regression suite

  • Asks “does it still do what it did?”
  • Should stay close to 100%
  • Runs on every prompt or model change
  • Grows as capability tasks graduate

Mixed methodologies: which grader for which check

“Mixed methodologies” means using each kind of grader where it is strongest, often several on the same task. Anthropic's comparison is consistent across its docs and engineering writing:

GraderStrengthsWeaknessesUse it for
Code-basedFast, cheap, objective, reproducible, easy to debugBrittle to valid variations; no nuanceLabels, values, formats, database state, which tools were called
Model-based (LLM-as-judge)Flexible, scalable, handles open-ended outputNon-deterministic, costs calls, must be calibratedTone, relevance, groundedness, rubric criteria
HumanGold standard; matches expert judgmentSlow, expensive, hard to scaleHigh-stakes calls, disputed cases, calibrating the judge

A single task can carry several graders. Anthropic's own example for a coding-agent task pairs unit tests, a model-graded code-quality rubric, static analysis, a check that the security log recorded the right event, and a check that certain files were read. The same idea, applied to a support agent:

One task, four graders (illustrative format)yaml
task:
  id: refund-late-delivery-017
  input: "My order arrived 9 days late. Can I get the shipping fee back?"
  graders:
    - type: state_check            # code: did the outcome happen?
      expect: { refunds: { order: "A-1042", amount: 4.99 } }
    - type: tool_calls             # code: policy looked up before acting
      required: [ { tool: get_refund_policy } ]
    - type: llm_rubric             # model: tone and clarity
      rubric: rubrics/support_tone.md
    - type: regex_absent           # code: never echo a card number
      pattern: "\\b\\d{13,16}\\b"

Notice what isn't there: no check that the agent took exactly steps one, two and three in order. Anthropic's advice is to grade what the agent produced, not the path it took — agents regularly find valid routes the eval designer didn't anticipate. Check the path only where the path itself is the requirement, such as “looked up the policy before issuing money”.

Checking an LLM-as-judge before trusting it

  • Passes: Detailed rubric with anchored scale points“1 = contradictory … 5 = fully logical”
  • Passes: Reasons before it scores
  • Passes: One judge per dimensionnot one prompt grading everything
  • Passes: Can answer “Unknown”a way out instead of a guess
  • Check: Agreement with human labels measuredon a sample, and re-checked
  • Fails: Same model and prompt as the system under testdocs suggest a different model
A judge is itself a model output. These are the checks that make its scores worth reporting.
A calibrated-style judge: rubric, reasoning first, a way outpython
JUDGE = """Grade the answer against the rubric.
<rubric>
PASS: every factual claim is supported by the <source>.
FAIL: any claim is missing from, or contradicts, the <source>.
UNKNOWN: the source is too incomplete to decide.
</rubric>
<source>{source}</source>
<answer>{answer}</answer>
Think in <reasoning> tags, then give <verdict>PASS|FAIL|UNKNOWN</verdict>."""

def judge(source: str, answer: str) -> str:
    r = client.messages.create(
        model=JUDGE_MODEL,            # ideally not the model being graded
        max_tokens=800,
        messages=[{"role": "user",
                   "content": JUDGE.format(source=source, answer=answer)}],
    )
    text = next(b.text for b in r.content if b.type == "text")
    return text.split("<verdict>")[-1].split("</verdict>")[0].strip()

Before the judge's numbers go on a dashboard, have domain experts label a sample and measure how often the judge agrees. Anthropic's engineering guidance is that model graders should be closely calibrated against human experts; if agreement is poor, fix the rubric before trusting the score.

Keep trials clean, and read what happened

Agent evals need an isolated environment per trial. If one trial leaves a file, a database row or a cached result behind, the next trial is no longer independent, and your pass rate measures leftovers. Start every trial from a clean state. Because model output varies, run several trials per task and report the variation, not just the best run.

Then read transcripts. Anthropic's engineers repeat this more than any other advice, and give an example: one model scored 42% on a public benchmark until the team found grading bugs — including a grader that rejected “96.12” when it expected “96.124991…”. With those fixed, the score was 95%. A failing test should look fair when a person reads it. If it doesn't, the grader is broken.

Offline evals are one layer, not the whole framework

Layers that catch different failures

Before release → after release

  1. Automated evalsfast, repeatable, no user impact
  2. Manual transcript reviewbuilds intuition; doesn't scale
  3. Systematic human studiesgold standard; slow and costly
  4. A/B testingreal user outcomes; takes days or weeks
  5. Production monitoringreal behaviour at scale; reactive
  6. User feedbacksurprises; sparse and self-selected
Anthropic compares this to a Swiss-cheese model: every layer has holes, and the layers together catch what any one misses.

A mixed-methodology framework uses them together. Automated evals gate every change; transcript review keeps the graders honest; human studies calibrate the judges; A/B tests (4.3) confirm that an offline win is a real-world win; production monitoring (4.6) feeds new failures back into the dataset.

Traps the wrong answers are built from

Tempting but wrongDo this instead
A small, hand-written happy-path test setSample real traffic and known failures, then add edge, should-not and adversarial cases.
One uncalibrated LLM judge grading every dimensionUse code where possible; one rubric-driven judge per dimension, checked against human labels.
Tuning prompts against the same set used for release decisionsIterate on a development set; decide on a held-out set.
Grading the exact sequence of steps an agent tookGrade the outcome and only the steps that are real requirements.
Trials that share stateStart every trial from a clean, isolated environment.

You should now be able to

  • Assemble an evaluation dataset from real traffic, known failures, edge cases, should-not cases and adversarial inputs.
  • Choose code-based, model-based or human grading for each check, and combine them on one task.
  • Design and calibrate an LLM-as-judge with a rubric, reasoning, an “Unknown” option and human agreement checks.
  • Separate capability and regression suites, and development and held-out splits.
  • Place offline evals within a wider framework of transcript review, A/B tests, monitoring and user feedback.

Practice questions

Original questions written for this lesson, in the exam’s style. Answer first, then open the reasoning — every option is explained, including why the wrong ones are tempting.

  1. Question 1

    A team's summarisation eval uses a single LLM judge to score accuracy, tone and completeness in one prompt. Scores look good, but clinicians reviewing outputs keep finding omissions.

    What is the best next step?

    1. AReplace the judge with a larger model and keep the single prompt.
    2. BSwitch entirely to clinician review for every output.
    3. CSplit into one rubric judge per dimension and measure each against clinician labels.
    4. DRaise the pass threshold on the existing combined score.
    Show answer and reasoning
    1. AIncorrect. A stronger model doesn't fix a judge that is uncalibrated and grading several dimensions at once.
    2. BIncorrect. Accurate but unscalable; human review is best used to calibrate and spot-check, not to grade everything.
    3. CCorrect. Isolated judges with clear rubrics, calibrated against expert labels, address both causes of the mismatch.
    4. DIncorrect. Moving a threshold on a miscalibrated score doesn't make it measure omissions.

  2. Question 2

    An e-commerce team is building the first eval set for a returns assistant. They have two weeks and a backlog of support tickets.

    Which two actions give the most useful starting dataset? (Select 2.)

    1. ATurn 20–50 tickets where the old assistant failed into test cases.
    2. BWait until 1,000 cases are ready before running anything.
    3. CInclude cases where the assistant should decline or escalate.
    4. DUse only clean examples written by the product manager.
    5. EGrade each case by checking the exact order of tool calls.
    Show answer and reasoning
    1. ACorrect. Real failures are Anthropic's recommended starting point: known inputs with a known wrong answer.
    2. BIncorrect. Waiting delays learning; starting small from real failures is the recommended approach.
    3. CCorrect. Balanced sets catch over-triggering as well as under-triggering.
    4. DIncorrect. Hand-written clean cases don't reflect real traffic, typos or edge cases.
    5. EIncorrect. Grading the path penalises valid alternative approaches; grade the outcome.

  3. Question 3

    An agent's pass rate on a benchmark-style suite is 40%. Reading transcripts, an engineer sees many “failures” where the answer was correct but formatted slightly differently.

    What does this most likely indicate?

    1. AThe agent needs a stronger model.
    2. BThe grader is too rigid and must be fixed first.
    3. CThe suite has saturated and needs harder tasks.
    4. DThe results are non-deterministic, so more trials are needed.
    Show answer and reasoning
    1. AIncorrect. The answers were correct; changing the model won't fix the measurement.
    2. BCorrect. A failure that isn't fair to a human reader signals a grading bug, as in Anthropic's CORE-Bench example.
    3. CIncorrect. Saturation means everything passes; here the score is being understated.
    4. DIncorrect. More trials would repeat the same grading error at larger scale.

  4. Question 4

    Why should a team keep a held-out evaluation set that is not used during prompt iteration?

    1. AIt is required for the API to report token usage correctly.
    2. BHeld-out cases can be graded without any reference answers.
    3. CIt lets the team skip regression testing for model upgrades.
    4. DPrompts tuned on a set overfit to it; unseen cases give an honest estimate.
    Show answer and reasoning
    1. AIncorrect. Token reporting has nothing to do with how you split evaluation data.
    2. BIncorrect. Held-out cases still need reference answers; the difference is when they are used.
    3. CIncorrect. Regression suites are still needed; a held-out set doesn't replace them.
    4. DCorrect. Iterating on the same cases inflates their score; a held-out set shows how the system generalises.

Practise every question in Domain 4 →

Sources

Drafted with AI assistance and checked against the sources above; expert review is in progress. Spotted an error? Tell us and it gets fixed, dated and listed on how this is written.