Efficient batch processing

What the Message Batches API gives you

The Message Batches API accepts a large set of ordinary Messages API requests in one submission and processes them asynchronously. The documentation lists the terms: a batch costs 50% of standard API prices; most batches complete within an hour, but a batch that has not finished within 24 hours expires; one batch can hold up to 100,000 requests or 256 MB, whichever comes first; and results stay available for 29 days after creation. Almost everything a normal request can do works inside a batch — system prompts, vision, tool use, multi-turn message histories, extended thinking and prompt caching — but streaming does not, because results come back as a single file.

Good fits are the ones the docs name: large-scale evaluations, content moderation, data analysis and bulk generation. In architecture terms, that means backfilling an extraction over an archive, re-scoring last month’s support tickets, generating product descriptions for a catalogue, or running a nightly compliance review. Poor fits are anything a user or a pipeline is blocked on — the answer might take an hour, and on a bad day most of a day.

Running a batch well

Pilot first. The docs recommend dry-running a single request shape with the Messages API to avoid validation errors. Go further: run your prompt synchronously over a representative sample, check the outputs against your criteria (4.1) and validators (4.4), and only then submit ten thousand copies of it. A bad prompt in a batch fails ten thousand times at once — cheaply, but after a long wait.

Give every request a meaningful `custom_id`. Results can come back in any order and may not match the order you submitted, so the custom_id is the only reliable link between a result and its source. It must be 1–64 characters of letters, digits, hyphens and underscores. Use your own identifier — inv-2026-09-000417 — so a result can be joined straight back to the record it belongs to.

batch = client.messages.batches.create(requests=[
    {"custom_id": doc.id,                       # e.g. "inv-2026-09-000417"
     "params": {"model": MODEL, "max_tokens": 2048,
                "system": SYSTEM_BLOCKS,        # shared prefix, cached
                "tools": [record_invoice],
                "tool_choice": {"type": "tool", "name": "record_invoice"},
                "messages": [{"role": "user", "content": doc.text}]}}
    for doc in documents
])

# ...later, once processing_status == "ended"
retry, fix = [], []
for r in client.messages.batches.results(batch.id):
    match r.result.type:
        case "succeeded":
            save(r.custom_id, r.result.message)          # then validate (4.4)
        case "errored" if r.result.error.error.type == "invalid_request_error":
            fix.append(r.custom_id)                      # request must change first
        case "errored" | "expired":
            retry.append(r.custom_id)                    # safe to resubmit as-is

Resubmit only what failed. A request that succeeded is done; do not rerun the batch. The docs distinguish errors: an invalid_request_error means the request body must be fixed before re-sending, while other errors can be retried directly. expired requests never reached the model and can go into the next batch. Requests that succeeded but failed your own validation go through the retry-with-feedback process from 4.4 — in a follow-up batch if time allows. A common fix for a failed request is structural: a document too long for the context window can be split into sections and resubmitted as several requests.

Designing around the 24-hour window

Because a batch can take up to 24 hours, deadlines have to be designed backwards from the worst case, not the typical one. If a business promises results within 30 hours of a document arriving, the latest a document can join a batch is 6 hours after it arrives — so submitting a batch every 6 hours (or more often) keeps the promise even when a batch takes its full window. Submitting once a day would not: a document arriving just after the daily cut-off could wait almost 24 hours to be submitted and 24 more to finish.

Stacking batch with prompt caching

Batch requests in an extraction job usually share a long prefix — the same tool definitions, the same system instructions, the same few-shot examples — and differ only in the document at the end. The docs say prompt caching and batch discounts stack. To give the cache a chance, put the shared content first and mark it with identical cache_control blocks in every request. Cache prefixes are built in a fixed order: tools, then system, then messages.

Two caveats from the documentation keep expectations honest. Because batch requests run asynchronously and concurrently, cache hits are best-effort — the docs cite hit rates anywhere from 30% to 98% — and they suggest the 1-hour cache duration to improve the odds, since the default lifetime is five minutes. And anything that changes the prefix (a different tool list, a tweaked system prompt per request) breaks sharing, so keep per-document variation out of the cached blocks.

Traps the wrong answers are built from

You should now be able to

• Decide between the Message Batches API and synchronous calls from latency and blocking requirements.
• Use custom_id to correlate results and handle each result type correctly.
• Resubmit only failed items, distinguishing requests that need fixing from those that can be retried.
• Calculate a submission schedule that meets a deadline under the 24-hour processing window.
• Structure batch requests so a shared, cached prefix precedes per-document content.