CCAR-P 4.3 — A/B testing and iterative improvement

The improvement loop

Hypothesisone change, one expected effect
Offline evalscapability + regression suites
Live A/B testrandom split, pre-set metrics
Decide and recordship, roll back, or iterate

Failures found live become new eval cases; the next hypothesis starts from the new baseline

Every change passes the cheap, fast gate before it reaches the slow, expensive one. Anything learned in production goes back into the eval set.

Offline first, online second

Anthropic's prompt-engineering guide assumes three things before you touch a prompt: clear success criteria, a way to test empirically against them, and a first draft. Iteration without those is guessing. The offline eval suite (4.2) is where most iteration happens, because it is fast, repeatable and exposes no customer to a bad version.

Offline evals can't tell you everything. They measure what you thought to test, on inputs you collected. Anthropic's engineering team describes A/B testing as the layer that validates significant changes once you have enough traffic, measuring real user outcomes such as retention and task completion — at the cost of taking days or weeks, and only testing changes you have already deployed.

	Offline eval	Live A/B test
Answers	Does it do the task on our test cases?	Do real users get better outcomes?
Speed	Minutes to hours; runs on every change	Days to weeks; needs enough traffic
Risk to users	None	Some users see the worse version
Measures	Accuracy, safety, security, cost per task on known cases	Resolution, retention, escalations, satisfaction, real latency
Blind spot	Inputs you didn't think to include	Rare failures too infrequent to show up in the split

Designing a fair A/B test

Change one thing. A new prompt, or a new model, or a new retrieval setting — not all three. If you must bundle, accept that you are testing the bundle.
Write down the decision rule first. One primary metric (for example, tickets resolved without escalation), the minimum improvement worth shipping, and guardrail metrics that must not get worse (p95 latency, cost per resolved ticket, safety flags).
Randomise the right unit. Usually the user or conversation, not the individual request, so one person doesn't bounce between versions mid-conversation.
Log the variant on every request. Version of prompt and model, alongside the outcome, so results can be sliced later.
Size and time it in advance. Decide how many conversations you need and run the full period. Stopping the moment the graph looks good is how noise gets shipped.

Sample size matters more than most teams expect. A lift from 62.0% to 64.5% resolution looks meaningful. With 200 conversations per arm, it is well within noise. With 4,000 per arm, it is unlikely to be chance. The same arithmetic applies offline: an agent's per-task success varies between runs, so compare versions over several trials per task, not one lucky pass.

Is the difference bigger than noise? (two-proportion z-test)python

from math import sqrt, erf

def ab_test(success_a, n_a, success_b, n_b):
    p_a, p_b = success_a / n_a, success_b / n_b
    pooled = (success_a + success_b) / (n_a + n_b)
    se = sqrt(pooled * (1 - pooled) * (1 / n_a + 1 / n_b))
    z = (p_b - p_a) / se
    p_value = 2 * (1 - 0.5 * (1 + erf(abs(z) / sqrt(2))))  # two-sided
    return round(p_b - p_a, 3), round(p_value, 3)

print(ab_test(124, 200, 129, 200))      # (0.025, 0.604) — can't tell
print(ab_test(2480, 4000, 2580, 4000))  # (0.025, 0.02)  — likely real

You don't need to be a statistician for the exam, but you do need the instinct: the same observed lift can be noise or signal depending on how much data sits behind it, and a decision rule set in advance protects you from reading what you hoped for into the result.

Which test does this change need?

What kind of change is it?

Small prompt fix
Offline regression suitethen ship and monitor
Major prompt or model
Offline evals, then A/Bprimary + guardrail metrics
Low traffic, high stakes
Offline + expert reviewA/B would take too long
Security or safety fix
Red-team set, then shipdon't wait weeks on a split

Match the evidence to the size of the change and the traffic available. Whatever the branch, every change runs the regression suite first.

Iterating without going in circles

Iterative improvement is a loop with memory. Each round starts from a hypothesis (“the bot is quoting the old refund window because the policy excerpt is buried below the examples”), makes one change, runs the capability suite to see whether the target moved and the regression suite to see whether anything else broke, and records the result against a prompt version. Without the record, teams re-try ideas that already failed and cannot say which change caused a regression three weeks later.

Iterating by feel vs iterating by evidence

By feel

Edit the prompt, try three inputs, looks better
Several changes bundled into one release
No stored baseline to compare against
Production complaints handled one by one

By evidence

One change per version, scored on the full suite
Regression suite must stay near 100%
Each version's scores stored with its prompt and model
Each production failure becomes a test case

Model upgrades are the iteration teams most often under-test. Anthropic's engineering article makes the business case: when a stronger model ships, teams without evals face weeks of testing while teams with evals can find the model's strengths, tune prompts and upgrade in days. The migration guides show why it can't be a blind swap. Moving to Claude Sonnet 5, for example, brings a new tokenizer producing roughly 30% more tokens for the same text, adaptive thinking on by default, and rejected non-default sampling parameters — and the checklist ends by telling you to re-baseline cost on your typical workload before production deployment.

Worked example · A retailer's support bot gets a new prompt

An online retailer's support assistant escalates 38% of conversations to a human. The team rewrites the system prompt to put the returns policy first and add three worked examples.

Offline. The new prompt scores 91% vs 86% on the 600-case capability set; the regression suite stays at 99.5%. Cost per resolved case rises 4% from the longer prompt — inside the agreed guardrail.
Decision rule, written before launch. Primary metric: conversations resolved without escalation. Guardrails: CSAT not lower, p95 latency not above 4 s, no rise in safety flags. Run for two full weeks, randomised by customer.
Mid-test temptation. After three days the new prompt is ahead by five points. The team keeps going; by day eight the gap has narrowed to 2.5 points as weekday and weekend traffic even out.
Result. 4,000 conversations per arm, a 2.5-point lift, p ≈ 0.02, guardrails held. Ship the new prompt, and store its scores as the new baseline.
Feed back. Twenty escalations from the test where the new prompt misread gift-card returns become new eval cases.

Takeaway: The offline suite made the change safe to test; the A/B test showed it helped real customers; the written rule stopped the team shipping on day three's inflated number.

Worked example · A product team upgrades the model

A SaaS company's report-writing feature runs on Claude Sonnet 4.6. Claude Sonnet 5 is cheaper per token and the team wants to switch.

Keep everything else fixed. Same prompts, same retrieval — the model is the only variable.
Run the suites. Report quality improves on the capability set. Two regression tests fail: the parser read content[0].text, and with thinking now on, the first block isn't text.
Fix, then re-run. Select blocks by type; the regression suite passes.
Re-baseline cost. Per-token price is lower, but the new tokenizer and thinking tokens mean cost per report falls less than the price list suggests. The team measures it on 1,000 real reports.
Live check. 10% of accounts move first, with report-regeneration rate as the primary metric and latency as a guardrail, before full rollout.

Takeaway: An upgrade is a change like any other: one variable, both suites, a cost re-baseline, then a controlled rollout.

What to log so that each variant's results can be compared in production is covered in 4.6; diagnosing why a variant lost is 4.4.

Traps the wrong answers are built from

Tempting but wrong	Do this instead
Shipping a prompt change because a few outputs look better	Score it on the capability and regression suites, and A/B test significant changes.
Changing the prompt and the model in the same release	Change one variable per version so the effect can be attributed.
Stopping an A/B test as soon as the new version is ahead	Fix the sample size, duration and decision rule before starting, and run the full period.
Judging a test only on the primary metric	Set guardrail metrics — latency, cost per task, safety — that must not get worse.
Treating a model upgrade as a config change	Re-run evals, fix breaking changes, re-baseline cost, then roll out gradually.

You should now be able to

Sequence a change through offline evals before a live A/B test.
Design an A/B test with one variable, a pre-registered primary metric, guardrails, user-level randomisation and a fixed duration.
Judge whether an observed difference is likely real given the sample size.
Choose between regression-only, A/B, expert review or red-team testing for a given change.
Run a model upgrade as a controlled experiment, including a cost re-baseline.

A/B testing and iterative improvement