AI Product Architecture: A Founder’s Guide

An AI product is not a model with a nice front end. It is a small set of components — an interface, an orchestration layer, a model gateway, a knowledge layer, memory, guardrails and an evaluation loop — and the founders who ship fastest are the ones who decide early which of those they need on day one and which they can add later. This guide walks through each layer, the build-or-skip decision for each, and the order we would build them in.

What does the architecture of an AI product actually consist of?

Most production AI products are made of seven layers, and an MVP needs a thin version of only four of them. Naming the layers up front keeps a build honest, because every layer you skip is a decision you are making deliberately rather than by accident.

• Interface: the web app, chat window, phone line or API your user touches. It should know nothing about which model sits behind it.
• Orchestration: the code that decides what happens on each request — which prompt runs, which tools are called, what gets retried. This is where the workflow-versus-agent choice lives.
• Model gateway: one internal function that every model call goes through, so swapping or routing models is a configuration change rather than a rewrite.
• Knowledge layer: retrieval over your own documents and data. Our RAG architecture guide covers this layer in depth.
• Memory and state: what the product remembers between turns and between sessions. See AI agent memory architecture for the tiers and the write-path rules.
• Guardrails: input and output checks, permission boundaries on tools, and a safe refusal path.
• Evaluation and observability: a test set that tells you whether a change made the product better, and logging that shows what really happened in production.

For an MVP, build the interface, a simple orchestration path, the model gateway and a first evaluation set. Add retrieval when the product genuinely needs your private data, and add memory only when users come back and expect continuity.

Should your first version be a workflow or an agent?

Start with a workflow. Anthropic’s engineering guidance on building effective agents recommends finding “the simplest solution possible” and only adding complexity when needed, and it distinguishes workflows, where predefined code paths orchestrate the model and tools, from agents, where the model dynamically directs its own process.

The same guidance names five workflow patterns that cover a large share of real products: prompt chaining, routing, parallelization, orchestrator-workers and evaluator-optimizer. It also warns that agents bring higher costs and the potential for compounding errors, which is why they earn their place only when the path genuinely cannot be fixed in advance. Our posts on AI agent design patterns and multi-agent orchestration patterns go deeper on when each pattern is worth the cost.

A useful rule: if you can draw the steps on a whiteboard before the first user arrives, it is a workflow. If the number and order of steps depends on what the model finds along the way, it may be an agent — and you should budget for the extra testing that implies.

Which model should you build on, and how do you avoid lock-in?

Pick a capable model to prove the product works, then route cheaper models to the easy steps once an evaluation set tells you it is safe. The architectural point is not which vendor you choose; it is that the choice is cheap to change.

That is what the model gateway is for. Keep prompts, model names, temperature and token limits in configuration, send every call through one function, and log inputs, outputs, token counts and latency at that single point. When a better or cheaper model appears, you run your evaluation set against it and decide on evidence. Our guide on how to choose an AI model for your app sets out a selection process, and the AI product cost model shows how model choice flows through to your monthly bill.

Anthropic’s guidance also cautions that frameworks can obscure the underlying prompts and responses, making them harder to debug, and suggests starting with direct API calls. For a founder this is practical advice: if you cannot read exactly what was sent to the model, you cannot fix it when it misbehaves.

Where should your data live if your customers are Australian?

Know where prompts and responses are processed before you sign a customer, because under the Privacy Act an Australian business remains accountable for what an overseas recipient does with personal information. The OAIC’s guidelines on APP 8 on cross-border disclosure say an entity that discloses personal information overseas must take reasonable steps to ensure the recipient does not breach the Australian Privacy Principles, and is accountable for acts or practices of the recipient that would breach them.

Deployment options matter here. Microsoft’s documentation on Foundry deployment types explains that data at rest stays in the designated Azure geography, but that inferencing data is processed differently depending on type: Global deployments may process data in any Azure region, Data Zone deployments within a specified zone, and Standard deployments within the customer-specified geography. Read the current terms for the exact region and model you plan to use, because availability changes.

The OAIC has also published guidance on privacy and commercially available AI products, including a recommendation that organisations do not enter personal information, and particularly sensitive information, into publicly available generative AI tools. Treat that as a design constraint from the first sprint, not a clean-up task before launch. Our post on data sovereignty for AI in Australia covers the practical options.

What should you test before you ship?

Build a small evaluation set from real failures before you add features. Anthropic’s article on evals for AI agents argues that teams do not need hundreds of tasks to start and describes 20–50 simple tasks drawn from real failures as a great start, with code-based, model-based and human graders each playing a role. It also stresses reading the transcripts and grades from many trials, so you know your graders are doing their job.

For a founder this changes the order of work. Write the first ten test cases during scoping, before the build, and they double as your definition of done. Our guides on how to test AI products and the AI app production-ready checklist turn this into a pre-launch routine.

What order should a founder build it in?

1. Write the job to be done and ten test cases. One user, one task, and what a good answer looks like.
2. Build the thinnest workflow that passes them. One prompt chain, one model, no framework unless you can read what it sends.
3. Put the model gateway and logging in from day one. They cost almost nothing early and are painful to retrofit.
4. Add retrieval only when the answers need your private data. Then measure it against the same test cases.
5. Add guardrails and a safe refusal path before real users arrive.
6. Review cost per task and move easy steps to cheaper models once the tests confirm quality holds.
7. Add memory or agents last, and only for a problem the simple version demonstrably cannot solve.

Do you need a technical cofounder to build this?

Not necessarily. What you need is someone accountable for the architecture decisions above and for shipping a working version. If you are weighing options, our posts on technical cofounder alternatives in Australia and AI app development in Melbourne set out the trade-offs, and if you are choosing a partner, nine questions to ask an AI consultant in Melbourne gives you a checklist.

This is the work we do every week. Our free 15-minute scoping call is for founders who want a straight answer on what to build first, and our 2-week Build Sprint at A$6,900 fixed produces a working slice of the product, with the fee credited toward a pilot, which starts from A$40k. If you are an owner digitising a business rather than building a product, NeoMind may be the faster route.

Frequently asked questions

What is AI product architecture?

AI product architecture is the set of components that turn a model into a working product: an interface, an orchestration layer, a model gateway, a knowledge layer, memory, guardrails, and an evaluation and observability loop.

Should a startup build an AI workflow or an AI agent first?

Start with a workflow. Anthropic’s guidance recommends finding the simplest solution possible and adding complexity only when needed, and notes that agents bring higher costs and the potential for compounding errors.

How many test cases do I need before launching an AI product?

You can start small. Anthropic’s article on agent evals describes 20–50 simple tasks drawn from real failures as a great start, and recommends reading the transcripts and grades from many trials.

Does an Australian AI product have to keep data in Australia?

Not as a blanket rule, but under APP 8 an Australian entity that discloses personal information overseas must take reasonable steps to ensure the recipient does not breach the Australian Privacy Principles and is accountable for the recipient’s breaching acts. Check where your model provider processes prompts and responses.

What does Neomeric offer founders who are building an AI product?

A free 15-minute scoping call, then a 2-week Build Sprint at A$6,900 fixed, fully credited toward a pilot that starts from A$40k.

Neomeric, a Melbourne-based AI product and consulting company — and the team behind NeoMind, Australia’s onshore AI teammates platform — ships AI products from Melbourne every week, so this guide reflects what we build rather than what we read.

Sources

• Anthropic — Building Effective AI Agents
• Anthropic — Demystifying evals for AI agents
• OAIC — Chapter 8: APP 8 Cross-border disclosure of personal information
• OAIC — Guidance on privacy and the use of commercially available AI products
• Microsoft Learn — Understanding deployment types in Microsoft Foundry Models