Chunking Strategies for RAG: 9 Methods Compared

In our latest guide on embeddings and vector databases, we saw how embeddings turn text into vectors and how a vector database stores and searches them. Here we look at the step that comes before that: how to split documents into chunks.

Retrieval-Augmented Generation (RAG) helps Large Language Models (LLMs) answer questions using external data, such as company documents, product manuals or research papers. But a good RAG application isn't just an LLM wired to a vector database. How you prepare the data before storing it decides how accurately the system retrieves information.

That's where chunking comes in.

Picture a 200-page document uploaded to an AI chatbot. When someone asks a question, the chatbot doesn't reread all 200 pages. It searches for the relevant sections and answers from those. Chunking makes this possible by breaking large documents into smaller pieces that can be stored, searched and retrieved.

Split the document the wrong way, though, and quality drops. A chunk that misses context leaves the model guessing. A chunk that's too large buries the exact answer in noise. In this article, we'll walk through nine chunking strategies for RAG, how each works, where each falls short, and which one to start with.

Key Takeaways

• Chunking strategy alone can change retrieval recall by up to 9% on the same data (
• Recursive chunking at around 200 tokens with no overlap is a strong, cheap baseline.
• Semantic chunking's extra cost isn't backed by consistent gains (

What Is Chunking in RAG?

Chunking is the process of dividing a large document or dataset into smaller, manageable sections called chunks. Each chunk must fit the embedding model's input limit, which is 8,191 tokens for OpenAI's text-embedding-3 models, though most RAG pipelines use chunks far smaller than that ceiling.

These chunks are converted into embeddings and stored in a vector database. When a user asks a question, the system searches for the most relevant chunks and passes them to the LLM as context for its answer.

How Does Chunking Fit Into the RAG Pipeline?

Chunking sits third in a typical eight-stage RAG pipeline, and every later stage depends on it. NVIDIA's 2025 benchmark found end-to-end answer accuracy varied from 0.603 to 0.648 across chunking methods on identical documents and models (NVIDIA Developer Blog, 2025).

A typical RAG pipeline follows this process:

1. Document ingestion: Load PDFs, Word files, web pages or other data sources.
2. Document processing: Extract text and keep useful structural information.
3. Chunking: Divide the extracted content into meaningful sections.
4. Embedding generation: Convert each chunk into a numerical representation of its meaning (the vector's dimensions depend on the embedding model you choose).
5. Vector storage: Store embeddings alongside their content and metadata.
6. Retrieval: Find chunks relevant to the user's question.
7. Context construction: Prepare the retrieved information for the LLM.
8. Response generation: Generate an answer grounded in the retrieved context.

If important information gets separated from its surrounding context, even a capable embedding model may struggle to find it. The goal isn't simply smaller chunks. It's chunks that keep useful meaning and improve retrieval.

Why Is Chunking Important in RAG?

Chunking affects retrieval accuracy, context quality, token cost and speed all at once. Anthropic showed that simply prepending context to each chunk before embedding cut failed retrievals by 35%, and by 49% when combined with BM25 keyword search (Anthropic, 2024).

1. Retrieval Accuracy

The retrieval system has to find the information that best matches a user's question. Meaningful chunks help it return relevant content instead of unrelated sections from the same document.

2. Context Preservation

Information often depends on the sentences around it. A good chunking strategy keeps enough context that each retrieved section makes sense on its own.

3. Response Quality

An LLM can only use what's in its context. If retrieval returns incomplete or irrelevant chunks, the answer will be incomplete or wrong too.

4. Token Usage and Cost

Sending unnecessary text to an LLM raises token usage and cost. Well-sized chunks keep the context window lean.

5. Processing Speed

Chunk size and count affect embedding time, vector search and context preparation. The right balance depends on your application and infrastructure.

A useful way to picture the relationship:

Chunking → Embeddings → Retrieval → Context → LLM Response

Good chunking can't guarantee a correct answer. Poor chunking, however, can break things before the LLM ever sees the data.

9 Chunking Strategies for RAG at a Glance

No single method wins on every document. Chroma's 2024 evaluation found the popular 800-token, 400-overlap default performed below simpler settings on recall-efficiency trade-offs (Chroma Research, 2024). Use this table to shortlist, then read the detail below.

1. What Is Fixed-Size Chunking?

Fixed-size chunking divides text into equal sections measured in tokens or characters. NVIDIA tested sizes from 128 to 2,048 tokens and found 128-token chunks scored worst on one dataset (0.421 accuracy) while page-sized chunks scored best (0.520), so size matters even within this simple method (NVIDIA, 2025).

Make sure the chunk size fits within your embedding model's maximum input length, or the extra text may be cut off. Once the limit is reached, the system starts a new chunk.

Advantages

• Easy to implement.
• Needs very little processing.
• Useful for simple documents and early RAG prototypes.
• Gives predictable chunk sizes.

Limitations

• May split a sentence or paragraph in the middle.
• Can separate key information from its explanation.
• Doesn't understand the meaning of the text.

When should you use it?

Fixed-size chunking is a reasonable starting point for straightforward text, especially when you want a baseline before testing more advanced methods.

2. What Is Sentence-Based Chunking?

Sentence-based chunking splits text at sentence boundaries instead of at an arbitrary character or token position. It avoids the mid-sentence cuts of fixed-size splitting, though a single sentence rarely holds enough context alone, so most pipelines group three to five sentences per chunk.

Advantages

• Keeps complete sentences.
• Produces natural-reading text sections.
• Works well for articles, FAQs and general prose.

Limitations

• Single sentences may be too small to carry enough context.
• Some sentences run very long.
• Sentence boundaries don't always mark complete ideas.

When should you use it?

Use it when the source is well-written prose and sentence boundaries give meaningful divisions. Group related sentences rather than treating every sentence as its own chunk.

3. What Is Semantic Chunking?

Semantic chunking splits text where its meaning changes, rather than at a fixed size or sentence boundary. It's popular, but a 2024 study across retrieval and answer-generation tasks concluded its computational cost was "not justified by consistent performance gains" (Qu, Tu and Bao, arXiv, 2024).

The idea is to keep related information together and start a new chunk when the subject shifts. That way, chunks hold related ideas instead of unrelated sections that just happen to sit next to each other.

One common implementation embeds sentences or groups of sentences, compares their cosine similarity (covered in our embeddings guide), and marks a boundary where similarity drops.

Advantages

• Can keep related ideas in the same chunk.
• Useful for documents covering several subjects.
• Can improve retrieval when meaning-based boundaries matter.

Limitations

• Usually needs extra embedding calculations.
• Picking the similarity threshold is tricky.
• Slower or more expensive than simple splitting.
• Results depend on the embedding model and boundary-detection method.

When should you use it?

Semantic chunking can help with long reports, knowledge bases and research documents where topics change throughout. Evaluate it against simpler methods first. Don't assume it's better.

4. What Is Recursive Chunking?

Recursive chunking splits a document at its largest natural boundaries first, then falls back to smaller ones only when a piece is still too big. In Chroma's benchmark, a recursive splitter at 200 tokens with no overlap scored "consistently high" across recall, precision and IoU, making it one of the best low-cost defaults (Chroma Research, 2024).

If a section exceeds the configured chunk size, the system splits it again while trying to keep meaningful boundaries intact.

Example of a typical separator hierarchy (the default in LangChain's RecursiveCharacterTextSplitter):

1. Split by paragraph   ("\n\n")
2. Still too big? Split by line   ("\n")
3. Still too big? Split by word   (" ")
4. Still too big? Split by character   ("")

This balance between chunk size and text structure is why recursive splitting is so common in RAG implementations.

Advantages

• Tries to keep natural text boundaries.
• Adapts to different document lengths.
• Works well with articles, documentation and reports.
• A practical starting point for most text-based RAG systems.

Limitations

• Results depend on the splitting hierarchy.
• One paragraph may hold several unrelated ideas.
• Doesn't understand the meaning of the content.

When should you use it?

Recursive chunking is a strong baseline for general-purpose RAG, especially when documents have reasonably consistent paragraph and section structure.

5. What Is Sliding Window Chunking?

Sliding window chunking creates overlapping chunks by moving a fixed-size window across the text. Overlap protects context at boundaries but isn't free: NVIDIA found 15% overlap beat 10% and 20% on FinanceBench, while Chroma found removing overlap improved IoU because it cuts duplicate tokens (NVIDIA, 2025).

Instead of starting each chunk where the last one ended, the system repeats part of the previous content. Imagine a concept introduced at the end of one chunk and explained at the start of the next. Overlap makes it more likely that at least one chunk holds both.

Advantages

• Keeps some context across boundaries.
• Helps when information spans adjacent sentences.
• Easy to combine with fixed-size or recursive splitting.

Limitations

• Repeated text raises storage needs. Every extra chunk is an extra vector to store and search.
• Near-duplicate chunks may compete during retrieval.
• Heavy overlap increases embedding and processing costs.

When should you use it?

Use sliding windows when adjacent text has important dependencies. Start with modest overlap, around 10-15%, and test whether the retrieval gain justifies the extra storage.

6. What Is Hierarchical Chunking?

Hierarchical chunking organizes a document into levels such as document, chapter, section, subsection and paragraph, and keeps the links between them. NVIDIA's results support structure-aware splitting: page-level chunks had the highest average accuracy (0.648) and the lowest variance (0.107) of every method tested (NVIDIA, 2025).

Instead of treating every chunk as independent text, the system can search at a small level and pull the parent section in for extra context.

Hierarchical chunking is related to parent-child retrieval, but the terms aren't identical. Parent-child retrieval specifically means indexing small child chunks and returning their larger parent for context.

Store the parent ID and section name as metadata with each chunk's vector, so the parent can be found at retrieval time.

Advantages

• Keeps document organization intact.
• Connects specific details to their broader section.
• Useful for long, structured documents.
• Supports more flexible retrieval workflows.

Limitations

• Requires maintaining document structure and relationships.
• Parent sections may be too large to send directly to an LLM.
• Retrieval logic gets more complex.

When should you use it?

Hierarchical chunking suits technical documentation, employee handbooks, research papers, legal documents and other content with clear sections and subsections.

7. What Is Topic-Based Chunking?

Topic-based chunking groups content by subject or theme, often using explicit topic labels or a classification model. Unlike semantic chunking, which infers boundaries from embedding similarity, it assigns named topics, which makes metadata filtering and namespace routing easier in multi-domain knowledge bases.

It focuses on what the text discusses rather than how much text has been processed.

Topic labels can also decide which namespace each chunk is stored in. Keep the two ideas separate, though: chunking decides how text is split and labeled, while namespaces decide where vectors are stored.

Advantages

• Keeps information about the same subject together.
• Useful for multi-topic knowledge bases.
• Makes subject-specific retrieval and filtering easier.
• Helps organize large document collections.

Limitations

• Needs reliable topic detection or classification.
• Some content belongs to more than one topic.
• Wrong labels send content into the wrong group.

When should you use it?

It helps when building domain-specific assistants, enterprise knowledge systems or support chatbots that cover several distinct subjects.

8. What Is Modality-Specific Chunking?

Modality-specific chunking applies different splitting rules to each content type, such as text, tables, images, audio, video and code. It matters because extracting only the text around a table can break the link between column headers and values, which is often the exact data a user asks about.

Text, images and audio often use different embedding models with different dimensions, so they usually need separate indexes.

Take a financial report with a table of quarterly revenue. Keeping the table structure, or tying each row to its headings, preserves what retrieval needs. Similarly, a video-based RAG system may need to link transcript segments with timestamps and relevant frames.

Advantages

• Keeps information specific to each content type.
• Supports multimodal search and question answering.
• Retains relationships between text and non-text content.

Limitations

• Often needs specialized parsing or extraction tools.
• Tables, diagrams and images may need extra interpretation.
• Pipelines get more complex.

When should you use it?

Use it when your RAG application handles PDFs with tables, scanned documents, images, audio, video or other non-text data. The principle is simple: different content types need different rules to keep their meaning.

9. What Is Agentic Chunking?

Agentic chunking uses an AI agent to choose or adjust the chunking approach for each document. In Chroma's tests, a GPT-4o-driven chunker reached the highest recall of any method (91.9%) but only average precision and IoU, so the extra model calls buy recall, not across-the-board gains (Chroma Research, 2024).

Rather than applying the same rules to every input, an agent inspects the content and picks a strategy. It can also check the resulting chunks, spot missing context, and try a different approach when the first result falls short.

Agentic chunking is an emerging design pattern, not one standardized algorithm. Implementations vary widely in how much autonomy they give the agent.

Advantages

• Adapts processing to different document types.
• Supports flexible document-ingestion workflows.
• Can build in validation and feedback.
• May help with complex, mixed datasets.

Limitations

• Adds model calls and processing costs.
• Can increase latency and system complexity.
• Needs clear rules and validation to avoid inconsistent output.
• Doesn't guarantee better retrieval than a well-tuned conventional strategy.

When should you use it?

Agentic chunking is worth exploring in advanced systems that process varied document types or choose between several workflows. For a small RAG application, a simpler strategy is easier to maintain and evaluate.

Which Chunking Strategy Should You Start With?

Start with recursive chunking at 200-500 tokens and little or no overlap, then measure. Across the benchmarks cited here, the gap between a sensible baseline and the best method was a few percentage points, smaller than the gap between a bad default and a sensible one.

Here's a simple test plan:

1. Build 30-50 real questions with known answer passages from your own documents.
2. Run your baseline (recursive, ~200-400 tokens) and record recall at top-5.
3. Change one variable at a time: chunk size, overlap, then strategy.
4. Keep the change only if it improves recall or answer accuracy enough to justify its cost.

If you're building on the JVM, our Spring AI with Spring Boot guide shows where document splitting plugs into a Java RAG pipeline.

Conclusion

There's no single chunking strategy that works best for every document. Fixed-size chunking is a fine way to set a baseline, and recursive chunking is a strong default for general-purpose RAG. More advanced methods, such as semantic, hierarchical, topic-based, modality-specific and agentic chunking, should earn their place against simpler ones on your own data.

The goal isn't smaller chunks. It's chunks that keep useful meaning and improve retrieval.