AI Agent Memory Architecture: A 2026 Guide

AI agent memory architecture is the set of design decisions that determine what your agent remembers, where it stores it, how it retrieves it, and when it forgets. Get it right and an agent stays useful across weeks of interaction. Get it wrong and you have an expensive system that repeats questions, contradicts itself, and quietly gets worse as it accumulates history.

This is the single most common architectural gap we see in agent projects that stall between demo and production. The demo works because the whole conversation fits in one context window. The production system fails because real users come back tomorrow, and the week after, and expect the agent to know things.

Neomeric, a Melbourne-based AI product and consulting company — and the team behind NeoMind, Australia’s onshore AI teammates platform — builds these systems for founders and operators every week. This guide covers the memory tiers that matter, how to choose between them, what to measure, and the governance questions that decide whether your design survives contact with Australian privacy obligations.

Why doesn’t a bigger context window solve agent memory?

A bigger context window is working memory, not storage. It holds what the agent is looking at right now. It does not give you persistent state across sessions, structured organisation of what was learned, selective retrieval from months of history, or the ability to delete a specific fact on request. Those are separate problems and they need separate machinery.

There is also a performance reason not to simply stuff everything into context. Research from Chroma on what it calls context rot tested frontier models across increasing input lengths and found performance degrades as input grows — and that the degradation is driven substantially by the sheer volume of irrelevant content surrounding the answer, not by the difficulty of the question itself. The same research notes that degradation accelerates as the semantic similarity between the target information and the query decreases, which is exactly the situation in a long, meandering customer history.

The practical implication: a long context window makes a bad memory design cheaper to ignore, not better. You still need to decide what deserves to be in front of the model on this turn.

What are the tiers in an AI agent memory architecture?

Most production designs settle into four tiers. You do not need all four on day one, but you should know which ones you are deliberately skipping.

1. Working memory (in-context)

The current conversation, the system prompt, tool definitions, and whatever you have retrieved for this turn. It is fast, exact, and expensive per token. It disappears when the session ends unless you write it somewhere. Treat working memory as a budget you allocate, not a bucket you fill.

2. Episodic memory (what happened)

A durable record of past interactions: the conversation that occurred on 3 September, the order that was placed, the complaint that was escalated. Episodic memory answers “what did we do last time?” Its natural storage is an ordinary database with timestamps, not a vector store. Most teams reach for embeddings here when a WHERE customer_id = ? would have been faster, cheaper and exact.

3. Semantic memory (what is true)

Distilled facts, independent of when they were learned: this customer prefers email, this account is on the enterprise plan, this property has three bedrooms. Semantic memory answers “what do we know?” It is small, it is high-value, and it is what most people actually mean when they say they want their agent to have memory.

4. Procedural memory (how to do things)

Learned workflows, tool-use patterns, and corrections the agent has been given. This is the least mature tier in practice and the one most often better handled by simply editing your prompt or tool definitions, at least until you have real evidence the agent needs to adapt per-tenant.

Academic work is converging on the same decomposition. A 2026 survey of memory for autonomous LLM agents catalogues these mechanisms and the evaluation gaps around them, and recent multi-layer frameworks decompose dialogue history into working, episodic and semantic layers with adaptive retrieval gating.

How do you decide what the agent should remember?

The write path is where most memory systems go wrong. Writing everything is the default and it is the worst option: it fills storage with transient noise, and it guarantees that retrieval will surface the wrong thing later.

We use four questions before anything is committed to durable memory:

1. Is it durable? A fact that expires on its own — where the user is right now, what they are working on this afternoon — should not be written. Only write what will still be true next month.
2. Did the user actually state it? Inferences the agent drew about a person are the fastest route to an embarrassing and hard-to-correct system. Store what was said; derive the rest at read time.
3. Does it change a future answer? If recalling the fact would not alter what the agent does, it is trivia. Trivia costs tokens at every retrieval and buys nothing.
4. Can you delete it cleanly? If a fact is smeared across an embedding index with no stable identifier, you cannot honour a deletion request. Design the delete path before the write path.

Two architectural options exist for making the write decision. The deterministic option is a rules layer: your code decides what gets stored, based on event types you control. The model-driven option gives the agent memory operations as tools and lets it decide. Anthropic’s memory tool takes the second approach, exposing a structured file interface — view, create, str_replace, insert, delete, rename — over a scoped memory directory, executed client-side by your application. Research frameworks are exploring the same shape: Agentic Memory integrates long-term and short-term memory management into the agent’s own policy, exposing store, retrieve, update, summarise and discard as actions the agent chooses.

Our default for commercial systems: deterministic writes for anything that affects money, identity or compliance; model-driven writes for preferences and context. The agent can decide to remember that someone likes morning appointments. It should not be deciding, unsupervised, what to record about their account status.

How should retrieval work?

Retrieval is a routing problem before it is a search problem. Before you embed anything, ask what kind of question is being answered:

• Identity-keyed lookups (this customer’s plan, their last three orders) → direct database query. Exact, cheap, auditable. No embeddings.
• Recency-keyed lookups (what happened in the last session) → ordered query with a limit. Also no embeddings.
• Open-ended recall (has this person ever mentioned anything about accessibility?) → this is where semantic search earns its place.

A large share of “our RAG doesn’t work” diagnoses turn out to be structured questions routed through a vector store. If you are tuning the semantic layer, our RAG architecture guide and our companion piece on RAG chunking and retrieval tuning cover the retrieval mechanics in detail, and context engineering for AI agents covers how to assemble what you retrieve into a prompt.

Whatever you retrieve, compress it before it enters context. A memory system that returns ten paragraphs to answer a one-line question has simply moved the context-rot problem one layer down.

How do you evaluate agent memory?

Memory is the part of an agent most likely to regress silently, because failures look like slightly worse answers rather than errors. You need tests.

The research benchmark worth knowing is MemoryAgentBench, which converts long-context tasks into incremental multi-turn streams and scores four competencies: accurate retrieval, test-time learning, long-range understanding, and selective forgetting. Its headline finding is a useful warning — no evaluated system mastered all four, and selective forgetting was the consistent weak point, with all evaluated methods reaching at most 28% accuracy on the multi-hop forgetting condition. The paper also found that retrieval-based methods outperformed long-context approaches on retrieval, while struggling with global summarisation.

Build the equivalent for your own domain. A workable starting suite:

• Recall: state a fact in session one, ask for it in session five.
• Update: state a fact, contradict it, confirm the agent uses the newer one.
• Forgetting: ask for deletion, then probe for the fact and anything derived from it.
• Restraint: confirm the agent does not volunteer stored details when they are irrelevant to the question.
• Isolation: confirm tenant A’s memory never surfaces for tenant B.

Run these in CI on every prompt or model change, the same way you would run the rest of your AI evals, and watch them in production through your observability layer.

What about privacy and governance?

A memory system is a personal information store. That framing changes the engineering requirements, and in Australia it changes your legal ones.

Three design rules follow directly:

• Every stored fact needs an identifier and a provenance record — what was stored, when, from which interaction. Without this you cannot answer an access request or execute a deletion.
• Tenant isolation belongs in the storage layer, enforced by keys and row-level access, not by a filter applied after retrieval. Our note on multi-tenant AI SaaS architecture covers the pattern.
• Sensitive categories need an explicit decision, not a default. Health information, financial details and identifiers should be excluded from general-purpose memory unless you have a specific, consented reason and controls to match.

There is a concrete date on the horizon. Under amendments introduced by the Privacy and Other Legislation Amendment Act 2024, from 10 December 2026 entities using personal information in automated decision-making that could significantly affect a person’s rights or interests must describe, in their privacy policy, the kinds of personal information used and the kinds of decisions made. The OAIC has been consulting on guidance for that obligation. If your agent’s memory feeds decisions of that kind, the architecture you choose now determines whether you can describe it accurately later.

For teams weighing where memory should physically live, our guide to data sovereignty for AI in Australia covers onshore hosting and the residency questions that follow.

What does a sensible first version look like?

Do not build all four tiers. Build this:

1. A conversations table — every session stored, keyed by user and tenant, with timestamps. This is episodic memory and it costs you nothing but a schema.
2. A facts table — one row per durable fact, with the user it belongs to, the source interaction, and a written-at timestamp. This is semantic memory. Keep it deliberately small.
3. A deterministic write rule — your code decides what becomes a fact, from a short list of event types.
4. Retrieval by key, not by similarity — load this user’s facts and their last N sessions. Add semantic search only when you have a question that genuinely needs it.
5. An eval suite — the five tests above, running in CI.

This gets most products further than they expect. Add vector retrieval, summarisation and model-driven writes when you have evidence from real usage about what the simple version is failing to do. Building in this order is also how you keep the cost model honest — every tier you add is tokens at every turn, forever.

Frequently asked questions

Do I need a vector database for agent memory?

Not to start. Identity-keyed and recency-keyed lookups — which cover most production memory needs — are better served by an ordinary relational database: exact, cheap and auditable. Add a vector store when you have genuine open-ended recall questions that a key lookup cannot answer.

What is the difference between RAG and agent memory?

RAG retrieves from a corpus of documents that exists independently of the conversation. Agent memory stores and retrieves what happened in and was learned from the interactions themselves. They use overlapping machinery and solve different problems; most real agents need both.

How much memory should be loaded into each turn?

As little as will answer the question. Research on long-context performance indicates that adding irrelevant surrounding content degrades accuracy, so treating context as a budget to allocate rather than a space to fill is the safer default. Compress retrieved memory before it enters the prompt.

Can the agent decide for itself what to remember?

Yes, and tool-based memory interfaces are designed for exactly that. We recommend splitting the decision: let the model manage preferences and soft context, and keep deterministic code in charge of anything touching money, identity or compliance.

How do I delete something from agent memory?

Only cleanly if you designed for it. Each stored fact needs a stable identifier and provenance, and you need to remove anything derived solely from it — including summaries that absorbed it. If facts live only as embedded text in an index, deletion is unreliable, which is a compliance problem as well as an engineering one.

How long does it take to build this?

The first version described above is typically days, not months, on top of an existing agent. The time goes into the eval suite and the governance decisions, not the storage. That is usually the right ratio.

Sources

• Chroma — Context Rot: How Increasing Input Tokens Impacts LLM Performance
• arXiv — Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions (MemoryAgentBench)
• Claude Platform Docs — Memory tool
• arXiv — Agentic Memory: Learning Unified Long-Term and Short-Term Memory Management for LLM Agents
• arXiv — Memory for Autonomous LLM Agents: Mechanisms, Evaluation, and Emerging Frontiers
• OAIC — Consultation on Guidance for Transparency in Automated Decision Making