RAG architecture is the set of design choices that determine whether your AI app answers from your data accurately or confidently makes things up. Retrieval-augmented generation is simple to prototype and hard to get right, and the failures almost always sit in the retrieval half rather than the model. This is the seven-decision checklist we work through when we build a RAG system that has to survive real users.<\/p>\n
Neomeric is a Melbourne-based AI product and consulting company — and the team behind NeoMind, Australia’s onshore AI teammates platform. We ship retrieval systems most weeks; these are the choices that consistently matter.<\/p>\n
A RAG system takes a user’s question, finds the most relevant pieces of your own content, and hands those pieces to a language model along with the question so the answer is grounded in your material rather than the model’s training data. The architecture is everything between “question arrives” and “context is assembled”: how documents are split, how they are indexed, how candidates are found, how they are re-ordered, how much gets sent, and how you know any of it worked. If you are still deciding whether retrieval is the right approach at all, start with our comparison of RAG versus fine-tuning<\/a>.<\/p>\n Chunking is the highest-leverage decision and the one most teams make by accident. Fixed-size splitting at 500 tokens is fast and destroys meaning at the boundary; a clause gets separated from the condition that qualifies it, and your retrieval returns half an answer.<\/p>\n Better defaults: split on the document’s own structure first (headings, sections, list items), then subdivide only what is too long. Keep a small overlap so a sentence spanning a boundary survives. Attach the document title and section heading to every chunk as a prefix — retrieval quality improves noticeably because the chunk carries its own context. Research comparing chunking methods finds real differences in effectiveness, but also that the more elaborate approaches carry meaningful computational cost, so the trade-off between chunking sophistication and compute is worth measuring for your own corpus<\/a> rather than assumed.<\/p>\n Pure vector search finds things that mean the same thing but misses exact tokens — product codes, error numbers, surnames, legislation references. Pure keyword search does the opposite. Hybrid retrieval runs both and merges the results, and for most business corpora it is the single easiest quality win available.<\/p>\n Practical shape: run a BM25-style keyword query and a vector query in parallel, merge with reciprocal rank fusion, and take the top 20–50 candidates forward. Do not tune this by intuition — tune it against a retrieval metric (see decision 6).<\/p>\n Honest cost benchmarks, the hidden costs vendors don’t quote, and a 10-line scoping worksheet.<\/p>\n Get the free guide<\/a><\/div>\n Usually, yes. First-stage retrieval optimises for recall — get the right chunk somewhere in the top 50. A cross-encoder reranker then scores each candidate against the query properly and reorders them, so the top 5 you send are the top 5 that matter. It adds latency and cost, and it is often the difference between a demo and a product. The rule of thumb: retrieve wide, rerank hard, send narrow.<\/p>\n Long context windows tempt teams to skip retrieval quality and just send everything. The research says do not. The well-known Lost in the Middle<\/em> study found that model performance is highest when relevant information sits at the beginning or end of the input and degrades significantly when the model must use information buried in the middle of a long context<\/a> — including for models explicitly built for long contexts. Work comparing retrieval against long-context approaches has also found that retrieval remains a strong and efficient option even as context windows grow<\/a>.<\/p>\n So: send fewer, better chunks, and place the strongest candidates first and last. Every extra chunk also costs money on every request — see our guide to AI API cost optimisation<\/a> for how quickly that compounds.<\/p>\n Metadata is what turns a search index into a product. At minimum, store source document, section, last-updated date, and access scope on every chunk. That gives you permission-filtered retrieval, a preference for current documents over superseded ones, and citations users can verify. Permission filtering is not optional: if retrieval is not scoped per user, your AI feature will eventually surface a document to someone who should not see it — a failure mode we cover in our guide to AI app security<\/a>.<\/p>\n You cannot improve a RAG system you are not measuring, and measuring the final answer alone tells you nothing about where it went wrong. Split evaluation in two.<\/p>\n Retrieval metrics<\/strong> answer “did we find the right chunk?” Build a set of representative questions, label which chunk should be retrieved for each, and track hit rate at k and mean reciprocal rank. This is cheap, deterministic, and catches most regressions.<\/p>\n Generation metrics<\/strong> answer “given the right chunk, did we produce a grounded answer?” — faithfulness to the retrieved context, and whether the answer actually addresses the question. The academic literature on RAG evaluation has grown quickly and offers a useful map of the available approaches and their limitations; a comprehensive survey of RAG evaluation in the LLM era<\/a> is a good starting point before you pick a framework. Our practical walkthrough of setting this up lives in how to build evals for AI products<\/a>.<\/p>\n Once retrieval is good, the generation model matters less than people expect: a smaller, cheaper model on well-retrieved context routinely beats a frontier model on poorly-retrieved context. Choose for latency, cost and instruction-following on your actual task, not benchmark position — see how to choose an AI model for your app<\/a>.<\/p>\n No. Research on long contexts has found that model performance is highest when relevant information appears at the start or end of the input and degrades significantly when it sits in the middle, even for models designed for long contexts. Retrieval also stays cheaper and faster, because you pay for every token on every request.<\/p>\n Hybrid retrieval runs a keyword search and a vector search over the same content and merges the results. Vector search captures meaning but misses exact tokens like product codes and reference numbers; keyword search captures those but misses paraphrases. Merging the two covers both failure modes, which is why it is a common default for business corpora.<\/p>\n There is no universal number. Split on the document’s own structure first, subdivide only what is too long for your model and budget, and keep a small overlap so meaning is not cut at a boundary. Chunking strategy measurably affects retrieval quality and computational cost, so the honest answer is to test two or three configurations against a labelled retrieval set for your own content.<\/p>\n Improve retrieval first, because most hallucination in a RAG system is the model filling a gap left by a bad chunk. Then instruct the model to answer only from the provided context and to say explicitly when the context does not contain the answer, and require citations so users can verify. Measure faithfulness as an explicit metric rather than assuming it.<\/p>\nDecision 1: How should you chunk your documents?<\/h2>\n
Decision 2: Vector search, keyword search, or both?<\/h2>\n
Free: The Australian AI MVP Cost Guide 2026<\/h4>\n
Decision 3: Do you need a reranker?<\/h2>\n
Decision 4: How much context should you actually send?<\/h2>\n
Decision 5: What metadata do you index?<\/h2>\n
Decision 6: How will you measure retrieval quality?<\/h2>\n
Decision 7: Which model goes on top?<\/h2>\n
A reference architecture<\/h2>\n
\n
Frequently asked questions<\/h2>\n
Does a bigger context window make RAG unnecessary?<\/h3>\n
What is hybrid retrieval and why does it help?<\/h3>\n
How big should my chunks be?<\/h3>\n
How do I stop RAG from hallucinating?<\/h3>\n