AI Insights
Online evals score your AI app’s real production traffic, so you find quality problems that a pre-launch test set never saw. They complement offline evals rather than replace them: offline tests protect you before a change ships, and online evals tell you what is happening after it does. Here is a practical way to set them up without drowning in data or cost.
Offline evals run a fixed set of test cases against your system during development. Online evals score live traces from real users as they arrive. Anthropic’s engineering team describes production monitoring as something that “kicks in post-launch to detect distribution drift and unanticipated real-world failures”, and notes that it reveals real user behaviour at scale but is reactive, since problems can reach users before anyone notices.
That last point is why you need both. Offline evals are your gate; online evals are your smoke alarm. If you have not built the offline layer yet, start with our guide to how to test AI products, then come back here. The pilot stage, where this all begins, is covered in our AI pilot to production playbook.
Score the few qualities that would hurt you if they slipped, not everything you can measure. A sensible starting set has four checks:
Write each check as a yes/no question with a clear definition. Vague scores like “quality 1 to 10” drift, and they are hard to act on.
Honest cost benchmarks, the hidden costs vendors don’t quote, and a 10-line scoping worksheet.
Get the free Australian AI MVP Cost Guide 2026 — we’ll email it straight to you.
Use sampling and filters. Langfuse’s documentation on online evaluation describes a rule that decides which incoming observations get scored, using filters, a sampling rate and one or more evaluators. For LLM-as-a-judge evaluators it advises checking the estimated cost and lowering the sampling rate if needed. The same idea works in any tooling: you do not need to judge every conversation to learn whether quality is moving.
A practical sampling plan looks like this:
Use an LLM judge for qualities that need language understanding, such as relevance, tone or whether an answer matches the source material. Langfuse describes core evaluation concepts that include model-based scoring alongside manual review and user feedback, and these methods work best together. Use plain code for anything you can check exactly, such as format, required fields or banned phrases, because code is cheaper, faster and does not disagree with itself.
Whatever judge you use, read the evidence. Anthropic’s guidance is blunt that you will not know if your graders are working well unless you read the transcripts and grades from many trials, and it recommends sampling transcripts to read weekly. Treat the judge as an instrument that needs calibrating, not as the truth.
Connect scores to decisions before you need them. Set an alert threshold for each check, name who gets the alert and write down what they do first. A falling grounding score might trigger a review of the knowledge base; a rising handover failure rate might trigger a prompt change and a rerun of your offline tests.
Then close the loop: every real failure you find should become a new offline test case. Over time your offline set stops being a guess and becomes a record of what has actually gone wrong. Our AI observability and monitoring guide covers the tracing layer that makes all of this possible.
Production traces often contain personal information, so treat your evaluation pipeline as part of your privacy design. The OAIC’s guidance on commercially available AI products recommends that organisations do not enter personal information, particularly sensitive information, into publicly available AI tools. If you send traces to an external judge model or a hosted tool, check where they are processed, what is retained and whether personal details can be removed first.
Online evals score real production traces from live users as they arrive, rather than a fixed test set run during development. They help detect drift and unexpected failures after launch.
No. Offline evals protect you before a change ships, and online evals show what happens after launch. Anthropic describes production monitoring as a post-launch complement to automated evals.
No. Langfuse’s documentation describes rules that use filters and a sampling rate to choose which observations are scored, and advises lowering the sampling rate if LLM-as-a-judge costs are too high.
Use an LLM judge for qualities that need language understanding, such as relevance or tone, and use plain code for anything you can check exactly. Read the transcripts and grades regularly to check the judge is working.
Traces can contain personal information, so check where they are processed and what is retained. The OAIC recommends not entering personal information, particularly sensitive information, into publicly available AI tools.
Neomeric, a Melbourne-based AI product and consulting company — and the team behind NeoMind, Australia’s onshore AI teammates platform — builds and monitors AI products for Australian founders, so this is the monitoring set-up we would start with.
Neomeric is a Melbourne AI product studio — 7+ products shipped, including our own. Start with a free 15-minute scoping call, or a 2-week Build Sprint at A$6,900 fixed, fully credited toward your pilot.
What an AI MVP really costs in Australia in 2026 — line-item budgets, the traps that blow them out, and how to scope a build that pays for itself.
AI Insights
Learn which AI API errors to retry, how to back off, when to fail over to a backup model and what breaks in practice. Book a free 15-minute scoping call.
AI Insights
Planning custom AI development in Australia? See what you pay for, how to stage a build, which rules apply and how to pick a partner. Book a free scoping call.
AI Insights
AI teammate for optometrists and audiologists: answer calls, book visits and hand clinical matters to your staff. Hosted in Australia, from $79/mo. Try NeoMind.