Human review and confidence calibration

Human attention is the budget

A team can usually review a fraction of what an automated system produces. The design question is which fraction. Reviewing a random 10% catches 10% of errors. Reviewing the 10% most likely to be wrong can catch far more — but only if “most likely to be wrong” is measured, not assumed. Anthropic’s evaluation guidance treats human grading as the gold standard for quality but slow and expensive, and recommends automating grading wherever it has been shown to be reliable. That is the same trade in production: spend people where automation has not earned trust.

Confidence is only useful once calibrated

A confidence score is calibrated when it matches reality: of all the fields scored around 0.9, about 90% turn out to be correct. Asking the model to output a confidence number does not make it calibrated — it is another thing the model generated, and it can be high on a wrong answer. It may still be useful, but you find out only by testing it against labelled data. The same is true of any signal you route on.

1. Assemble a labelled validation set that looks like production — including edge cases and every document type you process.
2. Run the system and record each field’s confidence signal alongside whether the field was actually correct.
3. Group fields into confidence bands and measure real accuracy in each band.
4. Set the auto-approve threshold where measured accuracy meets your target, per field type if they differ.
5. Recalibrate whenever the model, prompt or input mix changes — and on a schedule, using reviewer labels.

Stated confidence is not the only signal, and often not the best. Stronger ones come from checks you can run: the output failed schema validation or a cross-field check (line items do not sum to the total); two independent extraction passes disagree (the hallucination guide’s best-of-N check); the model could not quote supporting text from the source; or the input is a document type the system rarely sees. Combine them, then calibrate the combination the same way.

import random

# Thresholds come from the calibration table, per document type and field.
THRESHOLDS = {("invoice", "total"): 0.95, ("invoice", "vendor"): 0.90,
              ("handwritten", "total"): None}      # None = always review
AUDIT_RATE = 0.03                                 # 3% of auto-approved items

def route(doc_type, field, value):
    t = THRESHOLDS.get((doc_type, field))
    if t is None or not value.passed_validation or value.passes_disagree:
        return "human_review"
    if value.calibrated_score < t:
        return "human_review"
    if random.random() < AUDIT_RATE:              # stratify by doc_type in practice
        return "audit_sample"
    return "auto_approve"

One accuracy number can hide a failing segment

Anthropic’s guidance on success criteria says most use cases need evaluation along several dimensions, and that evals should mirror the real-world task distribution, edge cases included. For review design this means never deciding from a single aggregate. An extraction system that is 97% accurate overall may be 99.5% accurate on typed invoices — which dominate the volume — and 80% accurate on handwritten forms. Cutting review because “97% is good enough” silently removes the safety net from exactly the documents that need it.

The audit sample should be segmented too. A purely random 3% sample of a stream that is 95% typed invoices will contain very few handwritten forms, so a new failure there could take months to show up. Stratified sampling draws a fixed share from each segment — document type, source, field — so every segment gets enough reviewed items to measure its error rate.

Make the human’s job small and specific

Review is faster and more accurate when the reviewer sees what was flagged and why: the specific field, the value extracted, the source passage or region it came from, and the reason it was routed (low score, failed sum check, passes disagree). Asking reviewers to re-read whole documents wastes the budget the routing saved. The reduce-hallucinations guide’s citation technique helps here: when the model must quote its supporting text, the reviewer can confirm or reject in seconds, and a missing quote is itself a routing signal.

Traps the wrong answers are built from

You should now be able to

• Explain what calibrated confidence means and why self-reported scores need testing.
• Set routing thresholds from measured accuracy by confidence band, field and document type.
• Combine validation failures, pass disagreement and missing evidence into routing signals.
• Detect segment-level failures that an aggregate accuracy figure hides.
• Design stratified audit sampling of auto-approved output and a label feedback loop.
• Present review items so a human can decide quickly from the evidence.