AI agent memory architecture is the set of design decisions that determine what your agent remembers, where it stores it, how it retrieves it, and when it forgets. Get it right and an agent stays useful across weeks of interaction. Get it wrong and you have an expensive system that repeats questions, contradicts itself, and quietly gets worse as it accumulates history.<\/p>\n
This is the single most common architectural gap we see in agent projects that stall between demo and production. The demo works because the whole conversation fits in one context window. The production system fails because real users come back tomorrow, and the week after, and expect the agent to know things.<\/p>\n
Neomeric, a Melbourne-based AI product and consulting company — and the team behind NeoMind, Australia’s onshore AI teammates platform — builds these systems for founders and operators every week. This guide covers the memory tiers that matter, how to choose between them, what to measure, and the governance questions that decide whether your design survives contact with Australian privacy obligations.<\/p>\n
A bigger context window is working memory, not storage. It holds what the agent is looking at right now. It does not give you persistent state across sessions, structured organisation of what was learned, selective retrieval from months of history, or the ability to delete a specific fact on request. Those are separate problems and they need separate machinery.<\/p>\n
There is also a performance reason not to simply stuff everything into context. Research from Chroma on what it calls context rot<\/a> tested frontier models across increasing input lengths and found performance degrades as input grows — and that the degradation is driven substantially by the sheer volume of irrelevant content surrounding the answer, not by the difficulty of the question itself. The same research notes that degradation accelerates as the semantic similarity between the target information and the query decreases, which is exactly the situation in a long, meandering customer history.<\/p>\n The practical implication: a long context window makes a bad memory design cheaper to ignore, not better.<\/strong> You still need to decide what deserves to be in front of the model on this turn.<\/p>\n Most production designs settle into four tiers. You do not need all four on day one, but you should know which ones you are deliberately skipping.<\/p>\n The current conversation, the system prompt, tool definitions, and whatever you have retrieved for this turn. It is fast, exact, and expensive per token. It disappears when the session ends unless you write it somewhere. Treat working memory as a budget you allocate, not a bucket you fill.<\/p>\n A durable record of past interactions: the conversation that occurred on 3 September, the order that was placed, the complaint that was escalated. Episodic memory answers “what did we do last time?” Its natural storage is an ordinary database with timestamps, not a vector store. Most teams reach for embeddings here when a Distilled facts, independent of when they were learned: this customer prefers email, this account is on the enterprise plan, this property has three bedrooms. Semantic memory answers “what do we know?” It is small, it is high-value, and it is what most people actually mean when they say they want their agent to have memory.<\/p>\n Learned workflows, tool-use patterns, and corrections the agent has been given. This is the least mature tier in practice and the one most often better handled by simply editing your prompt or tool definitions, at least until you have real evidence the agent needs to adapt per-tenant.<\/p>\n Academic work is converging on the same decomposition. A 2026 survey of memory for autonomous LLM agents<\/a> catalogues these mechanisms and the evaluation gaps around them, and recent multi-layer frameworks decompose dialogue history into working, episodic and semantic layers with adaptive retrieval gating.<\/p>\n Honest cost benchmarks, the hidden costs vendors don’t quote, and a 10-line scoping worksheet.<\/p>\n Get the free guide<\/a><\/div>\n The write path is where most memory systems go wrong. Writing everything is the default and it is the worst option: it fills storage with transient noise, and it guarantees that retrieval will surface the wrong thing later.<\/p>\n We use four questions before anything is committed to durable memory:<\/p>\n Two architectural options exist for making the write decision. The deterministic option is a rules layer: your code decides what gets stored, based on event types you control. The model-driven option gives the agent memory operations as tools and lets it decide. Anthropic’s memory tool<\/a> takes the second approach, exposing a structured file interface — view, create, str_replace, insert, delete, rename — over a scoped memory directory, executed client-side by your application. Research frameworks are exploring the same shape: Agentic Memory<\/a> integrates long-term and short-term memory management into the agent’s own policy, exposing store, retrieve, update, summarise and discard as actions the agent chooses.<\/p>\n Our default for commercial systems: deterministic writes for anything that affects money, identity or compliance; model-driven writes for preferences and context.<\/strong> The agent can decide to remember that someone likes morning appointments. It should not be deciding, unsupervised, what to record about their account status.<\/p>\n Retrieval is a routing problem before it is a search problem. Before you embed anything, ask what kind of question is being answered:<\/p>\n A large share of “our RAG doesn’t work” diagnoses turn out to be structured questions routed through a vector store. If you are tuning the semantic layer, our RAG architecture guide<\/a> and our companion piece on RAG chunking and retrieval tuning<\/a> cover the retrieval mechanics in detail, and context engineering for AI agents<\/a> covers how to assemble what you retrieve into a prompt.<\/p>\n Whatever you retrieve, compress it before it enters context. A memory system that returns ten paragraphs to answer a one-line question has simply moved the context-rot problem one layer down.<\/p>\n Memory is the part of an agent most likely to regress silently, because failures look like slightly worse answers rather than errors. You need tests.<\/p>\n The research benchmark worth knowing is MemoryAgentBench<\/a>, which converts long-context tasks into incremental multi-turn streams and scores four competencies: accurate retrieval, test-time learning, long-range understanding, and selective forgetting. Its headline finding is a useful warning — no evaluated system mastered all four, and selective forgetting was the consistent weak point, with all evaluated methods reaching at most 28% accuracy on the multi-hop forgetting condition. The paper also found that retrieval-based methods outperformed long-context approaches on retrieval, while struggling with global summarisation.<\/p>\n Build the equivalent for your own domain. A workable starting suite:<\/p>\n Run these in CI on every prompt or model change, the same way you would run the rest of your AI evals<\/a>, and watch them in production through your observability layer<\/a>.<\/p>\n A memory system is a personal information store. That framing changes the engineering requirements, and in Australia it changes your legal ones.<\/p>\n Three design rules follow directly:<\/p>\n There is a concrete date on the horizon. Under amendments introduced by the Privacy and Other Legislation Amendment Act 2024, from 10 December 2026 entities using personal information in automated decision-making that could significantly affect a person’s rights or interests must describe, in their privacy policy, the kinds of personal information used and the kinds of decisions made. The OAIC has been consulting on guidance<\/a> for that obligation. If your agent’s memory feeds decisions of that kind, the architecture you choose now determines whether you can describe it accurately later.<\/p>\n For teams weighing where memory should physically live, our guide to data sovereignty for AI in Australia<\/a> covers onshore hosting and the residency questions that follow.<\/p>\n Do not build all four tiers. Build this:<\/p>\n This gets most products further than they expect. Add vector retrieval, summarisation and model-driven writes when you have evidence from real usage about what the simple version is failing to do. Building in this order is also how you keep the cost model<\/a> honest — every tier you add is tokens at every turn, forever.<\/p>\n Not to start. Identity-keyed and recency-keyed lookups — which cover most production memory needs — are better served by an ordinary relational database: exact, cheap and auditable. Add a vector store when you have genuine open-ended recall questions that a key lookup cannot answer.<\/p>\n RAG retrieves from a corpus of documents that exists independently of the conversation. Agent memory stores and retrieves what happened in and was learned from the interactions themselves. They use overlapping machinery and solve different problems; most real agents need both.<\/p>\n As little as will answer the question. Research on long-context performance indicates that adding irrelevant surrounding content degrades accuracy, so treating context as a budget to allocate rather than a space to fill is the safer default. Compress retrieved memory before it enters the prompt.<\/p>\n Yes, and tool-based memory interfaces are designed for exactly that. We recommend splitting the decision: let the model manage preferences and soft context, and keep deterministic code in charge of anything touching money, identity or compliance.<\/p>\n Only cleanly if you designed for it. Each stored fact needs a stable identifier and provenance, and you need to remove anything derived solely from it — including summaries that absorbed it. If facts live only as embedded text in an index, deletion is unreliable, which is a compliance problem as well as an engineering one.<\/p>\n The first version described above is typically days, not months, on top of an existing agent. The time goes into the eval suite and the governance decisions, not the storage. That is usually the right ratio.<\/p>\nWhat are the tiers in an AI agent memory architecture?<\/h2>\n
1. Working memory (in-context)<\/h3>\n
2. Episodic memory (what happened)<\/h3>\n
WHERE customer_id = ?<\/code> would have been faster, cheaper and exact.<\/p>\n3. Semantic memory (what is true)<\/h3>\n
4. Procedural memory (how to do things)<\/h3>\n
Free: The Australian AI MVP Cost Guide 2026<\/h4>\n
How do you decide what the agent should remember?<\/h2>\n
\n
How should retrieval work?<\/h2>\n
\n
How do you evaluate agent memory?<\/h2>\n
\n
What about privacy and governance?<\/h2>\n
\n
What does a sensible first version look like?<\/h2>\n
\n
Frequently asked questions<\/h2>\n
Do I need a vector database for agent memory?<\/h3>\n
What is the difference between RAG and agent memory?<\/h3>\n
How much memory should be loaded into each turn?<\/h3>\n
Can the agent decide for itself what to remember?<\/h3>\n
How do I delete something from agent memory?<\/h3>\n
How long does it take to build this?<\/h3>\n