Designing a RAG pipeline

First decision: do you need RAG at all?

Retrieval adds moving parts: a chunker, an embedding model, a vector store, a keyword index, a reranker, and evals for each one. Anthropic’s Contextual Retrieval post says plainly that if a knowledge base is small enough (it suggests under about 200,000 tokens), you can put the whole thing in the prompt and skip RAG. Prompt caching makes that repeated prefix much faster and cheaper.

A design review should ask three questions before anything else. How big is the corpus, and how quickly is it growing? How often does it change? Do users ask for exact identifiers such as part numbers, clause numbers or error codes, or do they ask about concepts? The answers decide the architecture before chunk size comes up.

Chunking: size, boundaries and self-containment

A chunk is the unit you retrieve, embed and eventually cite. Anthropic describes typical chunks as no more than a few hundred tokens, and lists chunk size, boundaries and overlap as choices that affect retrieval performance. There is no single right size. Small chunks give precise matches but lose surrounding meaning. Large chunks keep meaning, but their embedding blurs several topics together and each one uses more of the prompt.

Size matters less than whether a chunk still makes sense on its own. A clause that says “the Supplier shall pay the fee within 30 days” does not say which supplier, which contract or which fee. The embedding cannot contain information the text lacks. And BM25 cannot match a customer’s name that never appears in the chunk.

Contextual Retrieval: putting the context back

Anthropic’s fix is to prepend a short piece of context (usually 50–100 tokens) to each chunk before indexing it. Claude writes it, given the whole document and the chunk. The post’s example turns “The company’s revenue grew by 3% over the previous quarter” into a chunk that opens by naming ACME Corp’s Q2 2023 SEC filing and the previous quarter’s revenue. The same contextualised text goes into both the embedding index (Contextual Embeddings) and the keyword index (Contextual BM25).

def contextualize(doc_text: str, chunk: str) -> str:
    response = client.messages.create(
        model=MODEL,
        max_tokens=200,
        # The whole document is the cached prefix, so every chunk
        # after the first reuses it instead of paying for it again.
        system=[{
            "type": "text",
            "text": f"<document>\n{doc_text}\n</document>",
            "cache_control": {"type": "ephemeral"},
        }],
        messages=[{"role": "user", "content": (
            f"<chunk>\n{chunk}\n</chunk>\n"
            "Give a short, succinct context that situates this chunk within "
            "the document, to improve search retrieval. Reply with the context only."
        )}],
    )
    context = response.content[0].text
    return f"{context}\n\n{chunk}"   # index THIS text in both indexes

The failure rate there is 1 minus recall@20: the share of queries where a relevant chunk was not among the top 20 retrieved. Because the document is cached, the post puts the one-time cost of contextualising at about $1.02 per million document tokens. That is a one-off ingest cost, not a per-query cost, which is often the argument that wins a budget discussion.

Indexing: meaning and exact words

Embeddings capture meaning: “cancel my plan” lands near “terminate subscription”. They are weak on exact strings. The Contextual Retrieval post uses a support query for “Error code TS-999” as the case where BM25, which scores exact term matches, finds what embeddings miss. A hybrid index runs both and combines the ranked lists with a rank-fusion method (reciprocal rank fusion is a common choice), removing duplicates before the top-K go forward.

Anthropic does not offer its own embedding model. Its docs point to Voyage AI, which offers general models (the voyage-4 family), domain models for code, finance and law, and rerankers. Two details matter at design time. Embed documents with input_type="document" and queries with input_type="query". And weigh domain fit, latency at your scale, and whether the model can be customised to your vocabulary.

import voyageai
vo = voyageai.Client()                 # reads VOYAGE_API_KEY

doc_vecs = vo.embed(contextualized_chunks, model="voyage-4",
                    input_type="document").embeddings
query_vec = vo.embed([user_query], model="voyage-4",
                     input_type="query").embeddings[0]
# Voyage vectors are normalised, so a dot product is cosine similarity.

Store metadata alongside every chunk: document ID, title, section, version date, and the access-control attributes your permissions model needs. Metadata lets you filter before ranking (“only the current policy version”, “only documents this user may see”) and gives the generation step something to cite. Permission design itself is covered in 3.2.

Rerank, then hand chunks to Claude so they can be cited

Anthropic’s best configuration retrieved 150 candidates, scored each against the query with a reranking model, and passed the top 20 to Claude. Reranking adds some latency and cost per query. That is a trade-off to justify with numbers, and 3.3 covers it. Retrieving more chunks raises the chance the answer is present but costs more prompt, so tune K with an eval.

How you pass chunks to Claude is part of the design too. The Messages API has a search_result content block with a source, a title and text content. It can come back from a retrieval tool or be placed in a user turn. With citations enabled, Claude’s answer carries search_result_location citations pointing back to the exact result and block, parsed by the API rather than written as free text. Citations must be enabled on every search result in a request or on none.

{
    "type": "tool_result",
    "tool_use_id": block.id,
    "content": [{
        "type": "search_result",
        "source": "kb://contracts/harlow-2024#clause-14.2",
        "title": "Harlow Logistics MSA, clause 14.2 (Termination)",
        "content": [{"type": "text", "text": chunk_text}],
        "citations": {"enabled": True},
    }],
}

Traps the wrong answers are built from

You should now be able to

• Decide whether a knowledge base needs retrieval at all, or can be placed in a cached prompt.
• Choose a chunking strategy that fits the structure of each document type.
• Explain Contextual Embeddings and Contextual BM25, and what each adds over a standard pipeline.
• Design a hybrid index with rank fusion, reranking and filterable metadata.
• Pass retrieved chunks as search_result blocks so answers carry verifiable citations.
• Separate retrieval metrics from generation metrics when diagnosing a RAG failure.