RAG Chunking: 7 Steps to Better Retrieval

RAG chunking is the step where most retrieval quality is won or lost, and it is the step most teams tune last. If your AI app returns answers that are almost right, cite the wrong section, or miss information you know is in the source documents, the chunker is the first place to look — before you change models, rewrite prompts, or add a reranker.

This is a practical tuning guide: seven steps, in the order we run them, with the decisions that actually move retrieval quality. Neomeric, a Melbourne-based AI product and consulting company — and the team behind NeoMind, Australia’s onshore AI teammates platform — ships retrieval systems for Australian businesses every week, and this is the sequence we follow.

Step 1: Build an evaluation set before you change anything

You cannot tune what you cannot measure, and retrieval quality is not something you can eyeball. Before touching the chunker, write 30 to 50 real questions with the answer location recorded as a character offset in the source document — not as a chunk number.

This detail matters more than it sounds. If you record “the answer is in chunk 47” and then change your chunking strategy, every gold label is invalid, because the chunk boundaries moved. Store the span in the original document and compute which chunk contains it at evaluation time. That one decision lets you compare strategies honestly instead of rebuilding your test set every iteration.

Measure two things separately: retrieval recall (did the correct span appear in the top-k results?) and end-to-end accuracy (was the final answer correct?). They diverge more often than people expect, and the gap between them tells you whether your problem is retrieval or generation. Our guide to AI evals covers building the harness.

Step 2: Fix your document parsing first

A surprising share of “chunking problems” are parsing problems. If your PDF extractor is interleaving two-column text, dropping table structure, or losing headings, no chunking strategy will save you — you are splitting corrupted text into tidy pieces.

Before tuning, dump 10 parsed documents to plain text and read them. Check that headings survive, tables are coherent, and reading order is correct. Fixing the parser is unglamorous and routinely produces a larger improvement than any chunking change. Our AI data pipeline for RAG guide covers the ingestion layer in more detail.

Step 3: Choose a splitting strategy — and start boring

The realistic options, in increasing order of complexity:

• Fixed-size: split every N tokens. Fast, predictable, structure-blind.
• Recursive: split on paragraph, then sentence, then character, until chunks fit the target size. Respects natural boundaries. This is the sensible default.
• Structural: split on the document’s own headings and sections. Excellent for well-formed documents like policies, manuals and specs.
• Semantic: split where embedding similarity between adjacent sentences drops. Produces topically pure chunks.
• LLM-based: have a model decide the boundaries. Highest quality ceiling, highest cost.

Start with recursive splitting on your document structure. Semantic chunking is intuitively appealing and often improves pure retrieval metrics, but the trade-off is real: topically pure chunks can retrieve well while omitting the surrounding context the model needs to compose a complete answer. Recent evaluation work on chunking methods for RAG examines exactly this tension between chunking effectiveness and computational cost. If you adopt semantic chunking, measure end-to-end accuracy, not just recall — otherwise you may be optimising the metric that does not pay.

Step 4: Tune chunk size and overlap against your query type

There is no universal chunk size, but there is a reliable heuristic: match chunk size to how much text a human would need to read to answer your typical question.

Short factual lookups (“what is the excess on this policy?”) work with small chunks, because the answer is one sentence and extra text is noise. Analytical or multi-hop questions (“how does the cancellation process differ between plans?”) need larger chunks, because the answer spans a section.

The practical method is to sweep. Run your eval set at several chunk sizes across a reasonable range and plot both metrics. It takes an afternoon and it settles the argument with data specific to your corpus. Add modest overlap — typically around 10–15% of chunk size — so that sentences spanning a boundary are not orphaned. Large overlap inflates your index and your bill without proportionate gain.

If you serve mixed query types, do not compromise on one size. Index the same corpus at two granularities and route the query to the appropriate index.

Step 5: Give each chunk its context back

This is the single highest-leverage technique in the list. A chunk taken from the middle of a document loses the information that made it interpretable — which product it describes, which version, which section it belongs to.

Anthropic’s contextual retrieval method prepends a short generated blurb to each chunk explaining its place in the document before embedding. The published results are specific: combining contextual embeddings with contextual BM25 reduced the top-20-chunk retrieval failure rate by 49% (from 5.7% to 2.9%), and adding reranking reduced it by 67% (to 1.9%).

There is a cheaper version that captures much of the benefit: prepend the document title and heading path to every chunk as plain text before embedding. It costs one string concatenation and no model calls. Do this first, measure, and only pay for generated context if the gap justifies it. Either way, keep an eye on the API cost implications — contextualising every chunk at ingest is a per-document bill that recurs each time you re-index.

Step 6: Add hybrid search and reranking

Dense vector search is bad at exact matches. Product codes, error numbers, policy references and proper nouns are precisely where embeddings blur and keyword search excels. Running BM25 alongside vector search and fusing the results fixes a category of failure that no amount of chunking tuning will.

Then rerank. Retrieve generously — more candidates than you intend to use — and pass them through a cross-encoder reranker that scores each against the query directly, keeping only the best handful for the prompt. The Anthropic figures above show reranking contributing a substantial share of the total improvement.

Reranking also lets you send fewer tokens to the model, which matters: research on long-context behaviour from Chroma on context rot found model performance degrades as input length grows, driven substantially by the volume of irrelevant surrounding content. Retrieving twenty chunks and passing all twenty is a way to make a good retrieval result into a worse answer.

Step 7: Measure, then lock it into CI

Re-run the eval set after every change and record retrieval recall, end-to-end accuracy, tokens per query and latency together. A configuration that improves accuracy by two points while tripling cost is a decision, not an upgrade — make it deliberately.

Then make it permanent. Retrieval quality regresses quietly when documents are added, an embedding model is upgraded, or the parser changes. Run the eval set in CI, fail the build on a recall drop, and monitor retrieval metrics in production alongside the rest of your observability. How the retrieved material is then assembled into a prompt is its own discipline — our guide to context engineering for AI agents and the companion piece on AI agent memory architecture cover what happens after retrieval returns.

The short version

Eval set with character-offset labels, fix the parser, recursive splitting on document structure, sweep chunk size against your real queries, prepend heading context to every chunk, add BM25 and a reranker, then lock the whole thing into CI. Most teams find the largest single gain in step 5, and the most wasted effort in jumping to step 3’s exotic options before doing step 1. For the broader picture, see our RAG architecture guide.

Frequently asked questions

What chunk size should I use for RAG?

There is no universal answer — it depends on your corpus and query type. The reliable method is to sweep several sizes against your own evaluation set and compare both retrieval recall and end-to-end accuracy. As a heuristic, match chunk size to how much text a person would need to read to answer your typical question.

How much overlap should chunks have?

Enough that a sentence spanning a boundary is not orphaned — commonly around 10–15% of chunk size. Larger overlap grows your index and your embedding bill without a proportionate quality gain, so treat it as a parameter to test rather than to maximise.

Is semantic chunking better than recursive splitting?

Not automatically. Semantic chunking produces topically pure chunks that can score well on retrieval metrics, but those same chunks may omit surrounding context the model needs to compose a complete answer. Measure end-to-end accuracy, not just recall, before adopting it.

What is contextual retrieval?

It is a technique that prepends a short description of each chunk’s place in its source document before embedding, so the chunk remains interpretable out of context. Anthropic reported that combining contextual embeddings with contextual BM25 cut top-20 retrieval failures by 49%, and that adding reranking cut them by 67%.

Do I still need a reranker if my chunking is good?

Usually yes. Reranking scores candidates against the query directly rather than relying on embedding proximity, and it lets you send fewer, better chunks to the model. Since long inputs padded with irrelevant content degrade answer quality, sending fewer good chunks is a gain in itself.

How often should I re-tune chunking?

Whenever the corpus composition changes materially, the embedding model is upgraded, or the parser changes. The practical approach is not periodic re-tuning but continuous measurement: run the retrieval eval set in CI so regressions surface as build failures rather than user complaints.

Sources

• Anthropic — Contextual Retrieval in AI Systems
• Chroma — Context Rot: How Increasing Input Tokens Impacts LLM Performance
• arXiv — Chunking Methods on Retrieval-Augmented Generation: Effectiveness Evaluation Against Computational Cost and Limitations
• arXiv — A Survey of Context Engineering for Large Language Models
• arXiv — Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions