AI Insights
AI observability is the practice of recording what your AI system actually did — every prompt, retrieval, tool call, token and final answer — so that you can explain, debug and improve it after it ships. It is not the same as uptime monitoring. A traditional web app fails loudly with a 500 error; an AI product fails quietly, returning a fluent, confident, wrong answer that no dashboard flags. If you cannot reconstruct why your model said what it said, you do not have a product yet — you have a demo.
This guide is written for founders and product teams who have something working and are now trying to keep it working in front of real users. Neomeric, a Melbourne-based AI product and consulting company — and the team behind NeoMind, Australia’s onshore AI teammates platform — builds and operates systems like this every week. What follows is the instrumentation we put in before an AI product goes anywhere near a customer.
Because the failure mode is different. Conventional software is deterministic: the same input produces the same output, and when it breaks it throws an exception you can alert on. A language model is probabilistic. The same prompt can produce a good answer on Monday and a subtly wrong one on Tuesday, and both return HTTP 200. Your error rate stays at zero while your product quietly degrades.
The industry data reflects how uncomfortable this is. In Stack Overflow’s 2025 Developer Survey, 84% of developers said they were using or planning to use AI tools — yet trust in the accuracy of those tools fell to 29%, with more developers actively distrusting AI accuracy (46%) than trusting it (33%). The single most-cited frustration, named by 45% of respondents, was AI output that is “almost right, but not quite”. That is precisely the class of failure that monitoring dashboards do not catch and observability does.
The commercial consequence shows up further downstream. Gartner has predicted that over 40% of agentic AI projects will be cancelled by the end of 2027, citing escalating costs, unclear business value and inadequate risk controls. All three are observability problems before they are strategy problems. You cannot control a cost you do not measure, demonstrate a value you cannot evidence, or manage a risk you cannot see.
Log the whole trace, not just the answer. A single user request in a modern AI product touches a retrieval step, one or more model calls, possibly several tool invocations, and a formatting layer. If you only store the final response, you have thrown away every piece of evidence you need when a user complains.
A production-grade trace captures:
That last category is the one teams skip and later wish they had. Explicit feedback is rare; implicit feedback is abundant. A user who rewords the same question three times has told you the answer was bad without ever clicking a rating.
Get the free Australian AI MVP Cost Guide 2026 — we’ll email it straight to you.
Honest cost benchmarks, the hidden costs vendors don’t quote, and a 10-line scoping worksheet.
Use OpenTelemetry’s GenAI semantic conventions. They are the closest thing the field has to a neutral standard, and instrumenting against them means your traces are portable between tools rather than locked to whichever vendor you picked in month one.
The conventions define a shared vocabulary for AI telemetry so that a span emitted by an agent framework looks structurally identical to a span from a raw model call. The GenAI attribute registry specifies fields including gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens and gen_ai.response.finish_reasons, with prompt and completion content captured through structured attributes such as gen_ai.input.messages and gen_ai.output.messages. OpenTelemetry’s own write-up on GenAI observability walks through how a single model call decomposes into spans.
The practical benefit is unglamorous but real: when you outgrow your first observability vendor — and most teams do — you migrate a collector configuration rather than re-instrumenting your entire codebase. Standardised attributes also make cost analysis trivial, because token counts arrive in the same field regardless of which provider served the request. If you are actively managing spend, pair this with our notes on AI API cost optimisation.
You define quality narrowly, per use case, and you measure it continuously rather than once. “Is this a good answer” is unanswerable in the abstract. “Is this answer supported by the retrieved documents”, “did it cite a source”, “did it refuse when it should have refused”, and “did it stay inside scope” are all answerable, and each can be scored automatically.
Three layers work together in production:
Deterministic checks. Cheap, fast, and run on every single response. Does the output parse as valid JSON where the schema demands it? Does every cited document ID actually exist in the retrieval set? Is the response inside length bounds? Did it leak a system instruction? These catch a surprising share of real incidents at effectively zero cost.
Model-graded evaluation on a sample. Route a slice of production traffic through an evaluator model that scores groundedness, relevance and tone against a rubric. Sampling matters: grading every response doubles your inference bill for diminishing insight. What you want is a trend line sensitive enough to alert on a statistically meaningful shift after a prompt or model change.
Human review of the tails. Automated scores tell you the distribution moved; they rarely tell you why. Pull the worst-scoring traces, the sessions with three retries, and a random control sample, and have a human read them weekly. Every serious AI team we work with does this, and every one of them finds failure classes their evaluators missed. Our guide to building evals for AI products goes deeper on turning those findings into a regression suite.
The critical discipline is that production observability and your offline eval set are one loop, not two systems. Every genuine failure found in production becomes a test case. Over a few months that suite becomes the most valuable asset your AI product has, because it is the only thing standing between you and a silent regression the next time you change a model.
Agents need decision-level visibility, not just call-level visibility. Once a system can choose its own sequence of actions, the interesting question stops being “what did it say” and becomes “why did it choose that, and what did it touch on the way”.
The Australian Signals Directorate’s guidance on the careful adoption of agentic AI services is direct about what this requires: monitoring that gives visibility into agent activities, decision-making and system interactions throughout task execution, including alerts for anomalous or unauthorised behaviour, logging of actions and outcomes, and mechanisms to interrupt or halt agent operations where necessary. That last item is frequently missing from otherwise well-built systems. Observability without a stop button is a very detailed record of something you could not prevent.
Concretely, an agent trace should let you replay the run: the plan it formed, each tool it selected and why, the intermediate results, any retries or loops, and where a human approved a high-impact action. Loop detection deserves its own alert — an agent that calls the same tool eleven times is usually both a quality failure and a cost incident. If you are still shaping the architecture, our post on AI agent design patterns covers which patterns stay debuggable at scale.
Observability data is often the most sensitive data your system holds, because it is a verbatim record of what users typed. Treat trace storage with the same care as your production database, not as debug output.
Three obligations matter for Australian builders. First, cross-border disclosure: Australian Privacy Principle 8 governs sending personal information overseas, and observability platforms are a commonly overlooked path for exactly that — prompts containing customer detail routed to an offshore analytics vendor. Second, breach exposure: the OAIC reported 1,205 data breach notifications in the 2025 calendar year, an 8% increase on 2024 and the highest annual figure since the scheme commenced in 2018. A trace store full of unredacted prompts is a genuine liability. Third, automated decision-making transparency: the reforms in the Privacy and Other Legislation Amendment Act 2024 commence on 10 December 2026, and organisations relying on automated decisions will need to explain them in their privacy policies. Explaining a decision you did not log is not possible.
For APRA-regulated entities, CPS 230 has been in force since 1 July 2025, and the transitional arrangements for pre-existing contractual arrangements ended on 1 July 2026. Service provider monitoring obligations under that standard apply to AI vendors like any other material provider.
Practically, this means redaction at the collection point rather than the query point, defined retention windows for trace data, access controls on who can read production prompts, and a clear answer to where the trace store physically lives. Our AI app security guide covers the adjacent controls, and the ASD’s guidance on deploying AI systems securely treats logging and monitoring as core post-deployment activities.
Observability is the mechanism that makes AI governance real rather than documentary. The NIST AI Risk Management Framework organises AI risk into four functions — GOVERN, MAP, MEASURE and MANAGE — where MEASURE applies quantitative and qualitative methods to analyse, benchmark and monitor AI risk, and MANAGE covers continuing to detect and respond to changes across the deployed lifecycle.
Most organisations write an AI policy and stop. The policy asserts that outputs are monitored for accuracy and bias. Observability is what turns that sentence into evidence — a number you can show an auditor, a board, or an enterprise customer’s procurement team. For Australian teams, that evidence is increasingly what unlocks the deal rather than what satisfies the regulator.
Start smaller than you think. In order:
Teams that do steps one through three before launch ship far more calmly than teams that add observability after their first bad week. The production-readiness checklist we use covers where this sits relative to everything else that has to be true before launch.
Monitoring tells you that something is wrong — latency spiked, errors increased, spend jumped. Observability lets you work out why, by preserving enough detail about each request to reconstruct it afterwards. For AI systems, observability matters more because the most common failures produce no error at all.
Not on day one. If you instrument against the OpenTelemetry GenAI semantic conventions, you can start with the tracing infrastructure you already run and adopt a specialised tool when your evaluation and prompt-management needs outgrow it. Instrumenting to the standard first is what keeps that migration cheap.
Log them, but redact at the point of collection and set a retention window. Prompt content is the most useful debugging artefact you have and often the most sensitive data you hold. Strip identifiers before storage, restrict who can read production traces, and be deliberate about where that data is hosted — Australian Privacy Principle 8 applies when personal information is disclosed overseas.
Deterministic checks are worth running from the first request because they cost almost nothing. Sampled model-graded evaluation only becomes informative once you have enough volume for a shift in the score distribution to be distinguishable from noise, so most teams start with human review of every trace at low volume and switch to sampling as traffic grows.
Not meaningfully, when traces are exported asynchronously. The cost that surprises teams is storage and evaluation inference rather than request latency, which is why sampling policy and retention windows should be decided deliberately rather than left at a default.
It supplies the evidence. Explaining an automated decision, demonstrating that outputs are monitored, showing an enterprise buyer how errors are detected, and meeting service provider monitoring expectations all require records of what the system actually did. From 10 December 2026, transparency obligations for automated decision-making commence under the Privacy and Other Legislation Amendment Act 2024, and organisations will need to describe those decisions in their privacy policies.
Neomeric is a Melbourne AI product studio — 7+ products shipped, including our own. Start with a free 15-minute scoping call, or a 2-week Build Sprint at A$6,900 fixed, fully credited toward your pilot.
What an AI MVP really costs in Australia in 2026 — line-item budgets, the traps that blow them out, and how to scope a build that pays for itself.
AI Insights
AI for veterinary clinics in Australia: what it should handle, what must never touch it, onshore hosting, and how fast setup really is. Read the guide.
AI Insights
The 10 AI development tool categories that matter in 2026, what each is genuinely for, and the four you should not launch without. A builder's shortlist.
AI Insights
AI for childcare centres in Australia: where it genuinely helps, what must stay out of the Brain, onshore hosting, and how fast setup actually is. Read more.