An AI data pipeline for RAG is the machinery that gets your source content into the retrieval index and keeps it correct as the source changes. Most teams build the first half and skip the second. The result is familiar: a demo that impressed everyone in week two, and a system three months later that confidently quotes a price list you retired in March. Retrieval quality is a pipeline problem long before it is a model problem.<\/p>\n
This is a practical build order — seven stages, in the sequence we implement them, with the decision at each one and the failure it prevents. It assumes you have already chosen a retrieval design; if you have not, start with our RAG architecture guide<\/a>.<\/p>\n The single most consequential early decision is whether your pipeline can tell what changed. Rebuilding the entire index nightly works at a thousand documents and collapses at a hundred thousand — it is slow, it is expensive in embedding calls, and it makes freshness a function of your cron schedule rather than your business.<\/p>\n Build for incremental updates from day one. For each source, record a stable document ID and a content hash. On each run, compare hashes: unchanged documents are skipped entirely, changed documents are re-chunked and re-embedded, and deleted documents are removed. Where the source supports it — a database write-ahead log, a webhook, a filesystem watcher — drive the pipeline from the change event rather than polling. Everything downstream gets cheaper when only 2% of your corpus moves per day.<\/p>\n Extraction is where most retrieval quality is silently lost. A PDF flattened to plain text destroys the table that held the answer; an HTML page scraped without stripping navigation fills your index with menu items that match everything and mean nothing.<\/p>\n Extract to a structured intermediate — typically Markdown — that preserves headings, lists and tables. Keep the heading path for every block, because that path is the cheapest context signal you will ever get. Strip boilerplate deterministically before it reaches the index rather than hoping the reranker ignores it. If a document type resists parsing, it is usually cheaper to run a vision model over the pages than to accept degraded text for the life of the product.<\/p>\n Chunking has more folklore attached to it than almost anything else in RAG, and the research is a useful corrective. A 2026 controlled study of chunking in retrieval-augmented code completion crossed four chunking strategies with four retrievers, five generators and nine parameter configurations across 864 experimental settings, and found that the intuitively correct approach — chunking on function boundaries — underperformed every other strategy tested by 3.57 to 5.64 percentage points<\/a> on one benchmark. A separate evaluation of advanced strategies found that contextual retrieval preserved semantic coherence more effectively while late chunking was more efficient but sacrificed relevance<\/a>.<\/p>\n The lesson is not that one strategy wins. It is that the winner is corpus-specific and cannot be reasoned about from first principles. Start with recursive splitting on structural boundaries, keep chunks in the several-hundred-token range with modest overlap, carry the heading path and source metadata on every chunk — then measure alternatives against your own eval set rather than adopting whatever a blog post recommends. Our guide to running AI evals<\/a> covers how to build that set.<\/p>\n Honest cost benchmarks, the hidden costs vendors don’t quote, and a 10-line scoping worksheet.<\/p>\n Get the free guide<\/a><\/div>\n An isolated chunk often loses the thing that makes it findable. A paragraph reading “the fee increased by 12% in the second half” is nearly unretrievable because it never names the fee, the product or the year.<\/p>\n Contextual retrieval fixes this by prepending a short, generated description of where each chunk sits in its parent document before embedding it. Anthropic’s engineering write-up reports that contextual embeddings combined with contextual BM25 cut the top-20 retrieval failure rate from 5.7% to 2.9% — a 49% reduction — and that adding a reranking step brought it to 1.9%, a 67% reduction<\/a>. Those gains come from indexing-time work, which means you pay once per chunk rather than on every query.<\/p>\n Two practical notes. Generate the context with a small, cheap model — this is a summarisation task, not a reasoning one. And make it part of the pipeline, so a re-chunked document gets fresh context automatically rather than inheriting stale annotations.<\/p>\n Embedding is the expensive stage, so make it skippable. Key every embedding by a hash of the chunk text plus the model name and version. If the hash exists, reuse the vector. This turns a re-run into a no-op for unchanged content and makes an interrupted job safe to restart.<\/p>\n Two rules that save painful weeks:<\/p>\n Dense vector search is excellent at paraphrase and poor at exact tokens — part numbers, error codes, surnames, SKUs. Keyword search is the reverse. Running both and fusing the results covers each other’s blind spots, which is why the Anthropic results above pair contextual embeddings with BM25 rather than replacing it.<\/p>\n Then rerank. Retrieve generously — several dozen candidates — and use a cross-encoder to select the handful that actually go into the prompt. Reranking is the highest-leverage single addition to most retrieval stacks. It does add latency and cost at query time, which is the trade-off to measure rather than assume.<\/p>\n A pipeline you cannot see is a pipeline that has already broken. Four signals to emit from day one:<\/p>\n Wire these into the same tracing you use for the rest of the system — see our AI observability guide<\/a> for what to capture and how. For the tools at each stage, our rundown of AI development tools for 2026<\/a> covers the categories worth paying for.<\/p>\n Incremental design is a cost decision as much as a freshness one. If 2% of a corpus changes daily, an incremental pipeline does roughly 2% of the embedding work of a nightly rebuild, and embedding calls plus context generation are the dominant line item in most RAG pipelines. Cache aggressively, generate chunk context with the cheapest capable model, and keep reranking scoped to the candidates you actually retrieved. Our guide to cutting AI API costs<\/a> goes through the levers.<\/p>\n Neomeric, a Melbourne-based AI product and consulting company — and the team behind NeoMind, Australia’s onshore AI teammates platform — builds retrieval pipelines like this for Australian founders and businesses. If your data will include personal information, decide where the pipeline runs before you build it: our guide to data sovereignty for AI in Australia<\/a> covers the hosting question, and note that from 10 December 2026 entities using personal information in automated decision-making that can significantly affect a person’s rights or interests must disclose that in their privacy policy.<\/p>\n As often as the source changes, which is why change detection matters more than schedule. With document hashing and incremental updates, most sources can be reindexed continuously or hourly at modest cost, because only changed documents are re-embedded. Set your alert on index lag — the age of the oldest un-reindexed change — rather than on whether a nightly job ran.<\/p>\n There is no universal answer, and published research shows intuition is a poor guide. A 2026 controlled study of 864 configurations found that chunking on function boundaries underperformed other strategies by 3.57 to 5.64 percentage points on one code benchmark. Start with recursive splitting on structural boundaries at a few hundred tokens with modest overlap, then test alternatives against your own evaluation set.<\/p>\n Usually yes, because the cost is paid once at indexing time rather than on every query. Anthropic reports that contextual embeddings combined with contextual BM25 reduced top-20 retrieval failures from 5.7% to 2.9%, and that adding reranking reduced them to 1.9%. Generate the chunk context with a small, inexpensive model.<\/p>\n Yes for most corpora. Dense vector search handles paraphrase well but is weak on exact tokens such as part numbers, error codes and proper nouns, where keyword search is strong. Hybrid retrieval fuses both, which is why published contextual retrieval results pair contextual embeddings with BM25 rather than replacing it.<\/p>\n The existing index becomes unusable, because vectors produced by different models are not comparable. Plan a dual-write period where both indexes are populated, validate retrieval quality on the new one against your eval set, then cut over. Versioning the model name into your embedding cache key from the start makes this manageable.<\/p>\nStage 1: Ingest with change detection, not full re-crawls<\/h2>\n
Stage 2: Extract to structure, not to a wall of text<\/h2>\n
Stage 3: Chunk deliberately, then measure<\/h2>\n
Free: The Australian AI MVP Cost Guide 2026<\/h4>\n
Stage 4: Add context to chunks before you embed them<\/h2>\n
Stage 5: Embed and index idempotently<\/h2>\n
\n
Stage 6: Retrieve hybrid, then rerank<\/h2>\n
Stage 7: Instrument freshness and quality as first-class metrics<\/h2>\n
\n
What this costs to run<\/h2>\n
Frequently asked questions<\/h2>\n
How often should a RAG index be updated?<\/h3>\n
What is the best chunk size for RAG?<\/h3>\n
Is contextual retrieval worth the extra pipeline cost?<\/h3>\n
Do I still need keyword search if I have embeddings?<\/h3>\n
What happens when I change embedding models?<\/h3>\n