Say you've built a billing assistant on top of your company's policy PDF. A customer asks, "What is the refund window for annual renewals?" The policy answers it in one sentence: "Customers who cancel after the renewal date are eligible for a prorated refund only if the request is submitted within 14 days."
Your assistant answers with the monthly-plan refund rule instead. Nothing crashed. The right sentence is in the index. It just wasn't the one that came back.
The cause is usually the chunk. When the policy was split into pieces, that sentence was separated from its heading, "Section 8: Refunds for annual plans." On its own, the sentence never mentions annual plans, so the search had no reason to prefer it over the monthly-plan paragraph.
Contextual Retrieval fixes this at indexing time. It adds a short, chunk-specific note to each chunk before embedding and before it goes into the keyword index. In this guide, we'll cover why chunks lose context, what the technique does, how it combines with keyword search, rank fusion and reranking, what it costs, and how to check that it's worth it for your data.
Key Takeaways
- Chunking can strip out the subject, scope, date, or definition a sentence needs to be found. Contextual Retrieval puts that information back by prepending a short LLM-written note (usually 50 to 100 tokens) to every chunk before indexing.
- In Anthropic's tests, contextual embeddings cut failed top-20 retrievals by 35% (5.7% to 3.7%). Adding contextual BM25 raised the reduction to 49% (2.9%), and adding a reranker took it to 67% (1.9%).
- The context is generated once, at ingestion time. Your users' questions don't trigger any extra LLM calls for this step.
- It adds ingestion cost, and some knowledge bases can skip it. If yours is under about 200,000 tokens, you can skip RAG and put it all in the prompt.
Where Does Contextual Retrieval Fit in a RAG Pipeline?
It sits on the indexing side, between chunking and embedding. A RAG system has two halves. The first half prepares your documents ahead of time. The second half runs every time someone asks a question.
INDEXING (runs when documents are added or updated)
Documents → Ingestion → Chunking → [ Add context ] → Embeddings + keyword index
▲ you are here
ANSWERING (runs on every question)
User question → Query transformation → Hybrid retrieval → Rank fusion → Reranking → LLM answerContextual Retrieval improves what chunking hands to the embedding model.
If you're new to the vocabulary, this is all you need for now. An embedding is a list of numbers that represents a text's meaning. A vector database stores those lists and finds the ones closest to your question. RAG (retrieval-augmented generation) means you fetch relevant text first, then give it to an LLM along with the question so it answers from your data.
Why Do Chunks Lose Context?
Because a chunk is cut out of a document, and the things that explained it stay behind. A heading, a product name, a date range, or an earlier definition can all end up in a different chunk from the sentence that depends on them.
Large knowledge bases need chunking, so the split itself is fine. The trouble is that a chunk boundary can separate a statement from the information needed to interpret it. Our guide to chunking strategies for RAG covers how to choose those boundaries. Here's what tends to go missing:
| What went missing | Example | Why retrieval suffers |
| Subject | "It increased by 3%." | Nothing says what "it" is. |
| Scope | A rule that applies only to annual plans | The chunk looks like every other refund rule. |
| Time period | A figure from one quarter or one policy version | The search can't tell Q1 from Q2. |
| Distinct wording | Generic phrases such as "within 14 days" | Many chunks look alike when viewed alone. |
Anthropic's own example is a collection of SEC filings. A user asks, "What was the revenue growth for ACME Corp in Q2 2023?" The chunk that holds the answer says, "The company's revenue grew by 3% over the previous quarter." It names no company and no quarter, so it's easy to miss.
What Is Contextual Retrieval?
Contextual Retrieval is a preprocessing technique Anthropic described in September 2024. For each chunk, an LLM reads the whole document plus that chunk and writes one or two sentences saying where the chunk fits. That note is placed in front of the original chunk, and the combined text is what gets indexed.
| Stage | Text |
| Original chunk | "The company's revenue grew by 3% over the previous quarter." |
| Generated context | "This chunk is from ACME Corp's Q2 2023 performance filing; it discusses quarter-over-quarter revenue growth." |
| Text that gets indexed | The generated context, followed by the original chunk |
The indexed text now carries ACME Corp, Q2 2023, performance filing, and revenue growth. A question using those words has something to match.
The original chunk stays untouched. The context is extra text that helps search find the chunk, and the source text stays exactly as it was. When the chunk goes to the LLM to answer a question, you still have the original wording and its source location to cite.
What does the prompt look like?
It's a plain instruction. Here is the prompt from Anthropic's article:
<document>
{WHOLE_DOCUMENT}
</document>
Here is the chunk we want to situate within the whole document:
<chunk>
{CHUNK_CONTENT}
</chunk>
Please give a short, succinct context to situate this chunk within the overall
document for the purposes of improving search retrieval of the chunk.
Answer only with the succinct context and nothing else.Generated context usually runs 50 to 100 tokens. Treat that as a starting point. A very long note adds cost and can drown out the chunk's own words.
What does good context look like?
- Specific to the chunk. Each note describes its own chunk. Putting the same document title on every chunk makes chunks look more alike, which works against the goal.
- Written per chunk. Anthropic reports that adding generic document summaries to chunks gave very limited gains.
- An addition to your source. Keep the original text and its provenance (document, page, section).
- A retrieval improvement. It improves which chunks come back as candidates. You still have to test whether the final answer is right.
How Do Contextual Embeddings and Contextual BM25 Work Together?
You use the contextualized text in two places: the embedding model and the keyword index. Anthropic calls these contextual embeddings and contextual BM25.
Contextual embeddings. Send the context plus the chunk to the embedding model as one text, and store the vector it returns. The vector now reflects the chunk's subject and scope as well as its sentence. (Use the same embedding model for questions later, as covered in our embeddings guide.)
Contextual BM25. The BM25 algorithm is a keyword-ranking function. It scores a chunk by how many query words it contains, how rare those words are, how often they appear, and how long the chunk is. Building the BM25 index over the contextualized text means words added by the context, like "ACME" or "annual plans," can now be matched by keyword search too.
Why bother with both? They fail in different ways. Here's semantic search vs keyword search, side by side:
| Contextual embeddings | Contextual BM25 | |
| Good at | Paraphrases, related concepts, natural-language questions | Exact IDs, error codes, product names, function names |
| Weak at | Rare strings and literal identifiers | Wording that shares few words with the query |
| Example win | "Change a forgotten password" finds a page titled "Resetting your credentials" | "TS-999" finds the one page that contains TS-999 |
Take a support database with an article about the error code TS-999. A semantic search for "What does TS-999 mean?" tends to return articles about error codes in general. BM25 looks for the literal string and finds the article. A search for "How do I fix a login failure?" goes the other way: semantic search finds the authentication-error articles that never use the word "login," and BM25 alone would miss them. Running both and merging the lists covers both cases. This is the same hybrid search covered in our guide to embeddings and vector databases.
How Does Reciprocal Rank Fusion Merge Hybrid Search Results?
Run both searches, then combine the two ranked lists with Reciprocal Rank Fusion (RRF). Semantic similarity scores and BM25 scores come from different scales, so adding them directly makes little sense. RRF skips the scores and uses only each chunk's position in each list.
RRF score = sum over all lists of 1 / (k + rank)rank is the chunk's position in a list (1 is best). k is a smoothing constant, commonly around 60. A chunk that ranks well in several lists collects several contributions and rises to the top. Here is an example with k = 60:
| Chunk | Semantic rank | BM25 rank | RRF score |
| A | 2 | 5 | 1/62 + 1/65 ≈ 0.0315 |
| B | not returned | 1 | 1/61 ≈ 0.0164 |
| C | 4 | not returned | 1/64 ≈ 0.0156 |
Chunk A wins because both searches liked it. Chunk B still makes the list on the strength of one very good BM25 rank. These numbers are only an illustration.
Many search engines and frameworks ship RRF built in, so check yours before writing your own.
What Does Reranking Add?
Reranking re-scores a broad candidate list more carefully, so the best chunks sit at the top before they reach the LLM. Fast retrieval is built to avoid missing things. Ordering them well is a separate job.
A reranker is usually a cross-encoder-style model. It reads the question and one candidate chunk together and scores how well the chunk answers the question. That's slower than comparing vectors, so you only run it on a bounded set of candidates.
In Anthropic's setup, initial retrieval returned the top 150 chunks, a reranker (Cohere's) scored them, and the top 20 went to the model. Passing 20 chunks beat passing 10 or 5 in their tests. Treat those numbers as starting points to test against.
RRF and reranking work together. RRF is a cheap merge of two lists. Reranking is a closer reading of the merged list. In Anthropic's results, adding the reranker took the reduction in failed retrievals from 49% to 67% (a failure rate of 2.9% down to 1.9%), at the cost of some latency and per-query cost.
What Does the Complete Pipeline Look Like?
Context generation happens during ingestion. After the user's question, everything is search, merge, rerank, and answer. Before the search runs, many apps also rewrite the user's question so it matches the documents better, which is query transformation.
INDEXING
Source documents
→ extract text + metadata
→ split into chunks (keep chunk IDs and source locations)
→ for each chunk: LLM writes context using the full document
→ indexed text = context + original chunk
├─→ embedding model → vector index
└─→ tokenizer → BM25 index
QUERY TIME
User question
├─→ semantic search (vector index)
└─→ keyword search (BM25 index)
→ RRF merge + deduplicate
→ reranker scores question + candidate
→ keep top K
→ LLM answers from the original chunk text
→ return the answer with source referencesHow Much Does It Cost, and How Does Prompt Caching Help?
The cost is paid mostly during document ingestion, because the full document goes into the prompt once per chunk. A document of 8,000 tokens split into ten chunks would be read ten times if you sent it fresh each time.
Prompt caching reduces this. The provider keeps the repeated part of the prompt (the full document) so later calls pay only for the new part (the chunk). In Anthropic's estimate, with 800-token chunks, 8,000-token documents, 50 tokens of instructions and 100 tokens of context per chunk, one-time context generation costs about $1.02 per million document tokens. That was a 2024 figure under stated assumptions.
Cached calls still cost money. A rough ingestion cost model:
Ingestion cost ≈ context-generation input (cache writes + reads + new chunk tokens)
+ context-generation output
+ embeddings
+ index storage
+ retries and failed callsQuery-time cost is separate: the two searches, the reranker, and the answer LLM's input and output.
Ways to keep it down:
- Generate context once per document version. Never regenerate it per question.
- Skip unchanged documents by comparing content hashes.
- Cache contexts and embeddings keyed by source version, chunker version, prompt version, and model version.
- Try a small, cheap model for context generation, and check its output quality on samples.
- Bound concurrency, and retry transient failures with backoff.
- Treat caching as a discount. Correctness should never depend on it, so a cache miss should cost more and still finish the job.
- Track cost per thousand chunks and cost per correct answer alongside cost per API call.
When Should You Skip It?
If your knowledge base is small, you may not need RAG at all. Anthropic notes that under about 200,000 tokens (roughly 500 pages), you can place the whole knowledge base in the prompt, and prompt caching makes that faster and cheaper. Start there if it fits.
Contextual Retrieval pays off when your corpus is large, your chunks are ambiguous on their own (policies, filings, manuals, specifications with many similar sections), and your users search with names, codes, and specific wording.
Where Does It Help, and Where Does It Bite?
Where it helps:
- Chunks that said "it" or "the company" get a named subject.
- Sections that look alike get distinguishing details such as plan, version, region or quarter.
- Keyword search can match terms that only appeared in a heading or an earlier section.
- Every gain stacks with hybrid search and reranking.
Where it bites:
- Invented details. The LLM can add a fact the document never states. Use a conservative prompt, read samples, and keep the original text as the evidence.
- Noise. Long or vague context makes chunks look alike. Cap its length and compare against a no-context baseline.
- Stale indexes. If a document changes, regenerate its contexts and replace every old chunk in both indexes. Make re-indexing safe to repeat.
- Ingestion cost and time. One LLM call per chunk adds up on a large corpus.
- Provider limits. Rate limits and outages can stall a batch. Checkpoint progress so a job can resume.
- Access leaks. Apply permission and tenant filters before reranking and before building the prompt.
- Prompt injection. Text inside source documents can contain instructions aimed at the model. Treat retrieved content as untrusted evidence. Never let it act as commands.
- Unproven gains. The improvements above are averages from Anthropic's datasets. Yours may differ.
Common Mistakes
- Writing one generic context for every chunk in a document.
- Indexing the context but discarding the original chunk, or the reverse.
- Adding raw vector scores to raw BM25 scores. Merge by rank.
- Reranking the whole corpus. Rerank a bounded candidate set.
- Forgetting to re-run context generation when the document, chunker, prompt, or model changes.
- Passing the same candidate chunks to the LLM several times because duplicates and near-duplicates weren't removed.
Conclusion
Contextual Retrieval gives each chunk back the information that chunking took away: which document it came from, what it's about, and what it applies to. Contextual embeddings help semantic search, and contextual BM25 helps keyword search. RRF merges their results, and a reranker puts the strongest evidence first. Caching keeps the ingestion bill manageable.



