CCAR-P 5.3 — Human-in-the-loop validation

Three places to put a human

Before the outcome

Approve before act — the agent pauses; a person approves the tool call
Review before release — the model drafts; a person edits and sends
Slower, but errors never reach the world

After the outcome

Sample and audit — people check a share of completed work
Escalate by exception — low confidence or flagged cases go to a person
Faster, but some errors land before anyone sees them

The same agent can use all three: a gate on irreversible actions, review on outward-facing drafts, and sampling on routine output.

Why and where humans belong

Human review is a control like any other, and it has a cost: people are slow, expensive and get tired. So the question is never “should there be a human?” but “at which points does a human change the outcome enough to be worth it?” Three signals push toward a person in the loop: the action is hard to reverse (a payment, a deletion, a letter to a customer), the decision significantly affects someone (credit, a diagnosis, a benefits claim), or the input is new or unusual enough that the model is likely to be out of its depth.

Sometimes the answer is set for you. Anthropic’s Usage Policy lists high-risk use cases — including legal, healthcare, insurance, financial, employment, housing and academic decisions — and for these requires human-in-the-loop review, with a qualified professional in the field reviewing content or decisions before they are disseminated or finalised, and disclosure that AI was used. In the EU, GDPR Article 22 gives people the right not to be subject to a decision based solely on automated processing that has legal or similarly significant effects, and where such decisions are allowed, requires safeguards including at least the right to obtain human intervention, to express a point of view and to contest the decision (the regulation itself is covered in 5.4).

Which pattern fits?

What happens if the model is wrong here?

Irreversible action or money moves
Approve before actagent pauses at the tool call
Significant decision about a person
Qualified human decidesmodel drafts and cites evidence
Outward-facing but correctable
Review before releaseor exception routing at scale
Internal and low impact
Sample and auditreview a share, track error rate

Start from the consequence of an error, not from how good the model looked in testing.

Building the gate into an agent

Anthropic’s “Building effective agents” describes agents that pause for human feedback at checkpoints or when they hit blockers, and notes that even when automated tests pass, human review of a coding agent’s work still matters for fit with the wider system. In the Claude Agent SDK, the approval gate is the canUseTool callback: any tool call not settled earlier in the permission flow reaches your callback, where your application can show it to a person and return a decision. ask rules send matching calls to the callback even in bypassPermissions mode, and plan mode sends file edits to it regardless of allow rules, so an agent can propose changes that a person approves before they are made.

An approval gate on a refund tool

Agent

Your app

Reviewer

Payments

Request issue_refund(£240)

Matches ask rule

Show order, reason, evidence

Approve (or deny with note)

Execute refund

Result or denial reason

The agent does the research; the person makes the one decision that moves money. The reviewer sees the evidence, not just the proposed action.

Worked example · A law firm’s contract-review assistant

A law firm uses Claude to review supplier contracts against the firm’s playbook and propose redlines. Partners worry about both risk and cost: they do not want junior lawyers re-reading every clause.

Classify outputs. Clauses that match the playbook exactly are low risk; deviations on liability, indemnity and termination are high risk.
Route by risk. High-risk deviations go to a qualified lawyer, who decides; matching clauses are accepted and 1 in 20 is sampled.
Show evidence. Each proposed redline cites the playbook rule and quotes the contract text, so the lawyer can verify in seconds instead of re-reading.
Capture decisions. Every accept, edit or reject is logged with the reason and fed into the evaluation set.
Watch the numbers. If the sampled error rate rises, the sampling rate rises too; if it stays low for months, the firm can widen what counts as low risk.

Takeaway: Human attention is spent where legal judgement is needed, and every review makes the system measurably better.

Making review real: automation bias and review design

The most common way human-in-the-loop fails is not the absence of a human — it is a human who stops looking. When a model is right 98% of the time, reviewers learn to click “approve.” That is automation bias, and it quietly turns a control into theatre. Review design is how you resist it.

Rubber stamp versus meaningful review

Rubber stamp

Reviewer sees only the final answer
One-click approve, rejection needs a form
Every item reviewed, no matter how routine
Decisions never fed back

Meaningful review

Claims shown with cited sources
Uncertain or changed items highlighted
Review effort targeted by risk
Overrides tracked and fed into evals

Same people, same model. The right-hand design makes checking cheaper than guessing, and makes “reject” a normal outcome.

Useful techniques follow directly from the guardrail docs. Asking Claude to cite a supporting quote for each claim — and to retract claims it cannot support — gives the reviewer something concrete to check. Allowing the model to say it lacks enough information creates a natural escalation path: “insufficient evidence” goes to a person rather than becoming a guess. Some teams also seed known-bad items into the queue to measure whether reviewers catch them; if they do not, the review is not a control.

Worked example · A health insurer’s prior-authorisation drafts

A health insurer wants Claude to speed up prior-authorisation requests. Leadership proposes letting the model approve straightforward requests automatically and send denials to nurses.

Check the policy frame. Insurance and healthcare decisions are high-risk use cases in the Usage Policy, requiring qualified professional review and disclosure of AI use.
Spot the asymmetry. Auto-approval with human-reviewed denials still leaves approvals finalised without a professional — the policy’s requirement is about the decision, not only adverse ones.
Redesign. Claude drafts a recommendation for every request, citing the clinical criteria it matched and the chart evidence; a nurse reviewer finalises each one.
Target effort. Requests where every criterion is cited and matched show as “ready for sign-off”; those with gaps are flagged for fuller review.
Measure. Track reviewer agreement, time per case and overturned decisions to show both speed and safety.

Takeaway: The model can remove most of the reading; it cannot remove the qualified human from a high-risk decision.

Traps the wrong answers are built from

Tempting but wrong	Do this instead
Asking the model in the prompt to “check with a human first” as the approval mechanism.	Enforce approval in orchestration — an `ask` rule, `canUseTool` callback, hook, or a workflow that stops at a draft.
Routing every output to human review regardless of risk.	Target review by consequence and confidence; sample the routine and gate the irreversible.
Showing reviewers only the final answer.	Show cited evidence, highlight uncertain items, and make rejecting as easy as approving.
Removing human review from a high-risk decision because testing looked good.	Keep a qualified reviewer where the Usage Policy or regulation requires one; optimise the review instead.
Discarding reviewer decisions after the fact.	Log overrides with reasons and feed them into evaluations and prompt improvements.

You should now be able to

Choose between approve-before-act, review-before-release, exception routing and sampling based on the consequence of an error.
Identify when a human reviewer is required by policy or regulation, such as Usage Policy high-risk use cases or GDPR Article 22.
Implement approval gates outside the model, using Agent SDK permission rules, callbacks or workflow design.
Design reviews that resist automation bias with evidence, highlighting and measured reviewer accuracy.
Set measurable criteria for adjusting the level of human review over time.

Human-in-the-loop validation