Validation and retry loops

Two kinds of error, two kinds of check

Extraction errors come in two families. Syntax and shape errors — unparseable JSON, a missing key, a string where a number belongs — are what schema enforcement removes (see 4.3). Anthropic’s tool-use docs say that strict tool use guarantees inputs match the schema, eliminating missing parameters and type mismatches. Semantic errors survive that guarantee: a total that does not match its line items, a due date copied into the invoice-date field, a start date after the end date, a currency that contradicts the supplier’s country. Only your own validation catches those.

The last row is a design choice worth copying. If the schema asks for a short quote from the document alongside each important value, your validator can check that the quote actually appears in the source and that the value appears in the quote. Anthropic’s hallucination guidance recommends grounding answers in direct quotes and retracting claims without supporting evidence; a source_text field turns that advice into something code can test.

Retry with the errors, not just again

When validation fails, the useful retry sends three things back: the original document, the extraction that failed, and a precise description of what was wrong. With tool-based extraction this fits the tool-use protocol naturally. Append Claude’s turn, then a tool_result with is_error: true whose content lists the failures. The docs describe exactly this behaviour for invalid tool calls: given an error, Claude retries with corrections — and they advise writing instructive errors that say what went wrong and what to try next, rather than a bare “failed”.

def validate(inv: dict) -> list[str]:
    errors = []
    items = sum(line["amount"] for line in inv["line_items"])
    if abs(items - inv["total"]) > 0.01:
        errors.append(f"line_items sum to {items:.2f} but total is "
                      f"{inv['total']:.2f}; check for missed lines or discounts.")
    if date.fromisoformat(inv["invoice_date"]) > date.today():
        errors.append("invoice_date is in the future; you may have used the due date.")
    return errors

def extract(document: str) -> dict:
    messages = [{"role": "user", "content": document}]
    for attempt in range(3):                                  # bounded
        resp = client.messages.create(model=MODEL, max_tokens=2048, messages=messages,
            tools=[record_invoice], tool_choice={"type": "tool", "name": "record_invoice"})
        call = next(b for b in resp.content if b.type == "tool_use")
        errors = validate(call.input)
        if not errors:
            return call.input
        messages += [{"role": "assistant", "content": resp.content},
                     {"role": "user", "content": [{"type": "tool_result",
                         "tool_use_id": call.id, "is_error": True,
                         "content": "\n".join(errors)}]}]
    return send_to_review(document, call.input, errors)       # never loop forever

If you build on the Agent SDK, part of this loop comes built in: when you set output_format (Python) or outputFormat (TypeScript), the SDK validates the final output against your schema and re-prompts on a mismatch. If no valid output emerges within its retry limit, the result’s subtype is error_max_structured_output_retries. The docs also note that a result can say success yet carry no structured_output, and that you should treat that as a failure too. Your semantic checks still sit on top.

When retrying cannot help

Retries fix errors of reading: a value put in the wrong field, a line missed, a format slip. They cannot fix errors of absence. If a CV never states a graduation year, or a scanned page is illegible, asking again only invites the model to guess — and a guess that passes validation is worse than a visible gap. Before retrying, ask whether the correct answer is actually in the input.

Feedback loops beyond a single document

The per-document retry is the inner loop. The outer loop is what you learn across thousands of documents. Log every validation failure with its rule, field and document type. When one rule fails far more than the others — say, dates on invoices from one country — that is not a retry problem; it is a prompt, example or schema problem. Fix it at the source with a clearer criterion (4.1), a targeted example (4.2) or a schema change (4.3), then re-run your test set.

The same logs tell you where to point human review. Anthropic’s prompting guide describes the common self-correction chain — draft, review against criteria, refine — as separate calls so each step can be logged and evaluated. Whether a reviewer is a person or a second model instance, it should see the validator’s findings, not start from scratch. How to calibrate confidence and sample outputs for human review is covered in 5.5; multi-pass review architectures are covered in 4.6.

Traps the wrong answers are built from

You should now be able to

• Separate schema-level errors from semantic errors and put each check in the right layer.
• Implement a bounded retry that returns specific validation errors via tool_result with is_error.
• Decide between retrying, accepting null and escalating for a given validation failure.
• Add self-checking fields such as stated versus calculated totals, conflict flags and source quotes.
• Use aggregated validation failures to improve the prompt, examples or schema.