AI Insights
AI evals are automated tests for non-deterministic software. Instead of asserting that a function returns 42, an eval scores an AI feature’s output against a fixed set of realistic inputs and a definition of “good”, then tells you whether today’s version is better or worse than yesterday’s. If you are shipping anything built on a language model, evals are the single control that turns a demo into a product you can defend.
This guide is written for founders and product teams who have a working prototype and are now staring down the gap between “it worked when I tried it” and “it works for ten thousand strangers”. We build and ship these systems weekly at Neomeric, and the eval layer is consistently the part teams skip and later regret.
An eval is a repeatable measurement of an AI system’s behaviour on a fixed dataset. It has three parts: a set of inputs (the test cases), a definition of what a correct or acceptable output looks like, and a scorer that turns each output into a number or a label. Run the same eval against two versions of your prompt, model, or retrieval pipeline and you get a comparison instead of a hunch.
Traditional software testing does not transfer cleanly. A unit test assumes the same input always produces the same output. Language models do not offer that guarantee, and they degrade in ways that are invisible to type checkers: a slightly reworded prompt shifts tone, a model upgrade changes formatting, a new retrieval chunking strategy quietly drops the one document that mattered. None of that throws an exception. It just makes your product worse while every test stays green.
The measurement problem is not solved at the frontier either. Stanford HAI’s 2026 AI Index chapter on responsible AI examines safety, fairness, transparency and governance and is explicit that significant measurement gaps persist across the field. If the research community is still arguing about how to measure model behaviour reliably, the practical conclusion for a product team is not “wait for a standard” — it is “measure the narrow thing your product actually does, and measure it yourself”.
They fail because nobody can tell the difference between a change that helped and a change that felt like it helped. Without a scoreboard, prompt engineering becomes superstition, model upgrades become coin flips, and every regression is discovered by a customer.
The failure rate at the industry level is not subtle. Gartner has predicted that more than 40% of agentic AI projects will be cancelled by the end of 2027, citing escalating costs, unclear business value and inadequate risk controls. A widely reported MIT study of enterprise generative AI pilots found that the overwhelming majority showed no measurable effect on profit and loss, as covered by Forbes — a preliminary, non-peer-reviewed report, but one whose diagnosis matches what we see in the field.
The common thread in both is not model capability. It is the absence of a feedback loop tight enough to prove value or catch drift. “Unclear business value” is what it looks like from the boardroom when nobody instrumented the thing. Evals are how you make value legible.
There is a second, quieter failure mode: teams that do have evals but built them out of vibes. A twelve-example test set assembled by the founder on a Sunday will pass everything, because it encodes the founder’s mental model of the product rather than the messy reality of users. An eval set that never fails is not a safety net, it is decoration.
Honest cost benchmarks, the hidden costs vendors don’t quote, and a 10-line scoping worksheet.
Get the free Australian AI MVP Cost Guide 2026 — we’ll email it straight to you.
Measure the small number of behaviours that would embarrass you in front of a customer. For most AI products that comes down to four families of metric, and you almost certainly need all four.
Task success. Did the system do the thing? For a support assistant, that is whether the answer resolves the question. For an extraction pipeline, whether the fields are right. This is the only metric your customer cares about, and it is usually the hardest to automate — which is exactly why teams substitute easier proxies and then wonder why the numbers look good and the product feels bad.
Groundedness. Is every factual claim traceable to a retrieved source, or is the model filling gaps from memory? For retrieval-augmented systems this is the metric that separates a useful product from a liability. Score it per claim, not per response, or you will average away the single fabricated sentence that matters.
Format and contract compliance. Does the output parse? Does it match the schema, stay inside the length budget, use the tool call you expected? These are cheap to check with plain code and they catch a startling share of production incidents, particularly after a model version change.
Safety and scope. Does the system refuse what it should refuse, decline to give advice it is not qualified to give, and stay inside the topic boundary you sold? For Australian teams this is also where regulatory exposure lives — a system that confidently answers a question about someone’s entitlements or finances is a different risk category from one that books an appointment.
Latency and cost belong on the same dashboard. They are not quality metrics, but every quality improvement is a trade against them, and a team that only tracks quality will happily ship a 40-second response that nobody waits for.
A golden set is a curated collection of inputs with known-good expectations. It is the most valuable artefact your AI team will own, and it cannot be bought — it has to be built out of your own traffic and your own domain.
Start with fifty to a hundred cases. That is enough to detect meaningful regressions and small enough that a human can actually review the results. Sample them from four buckets in roughly equal measure:
Write the expectation as a rubric, not as an exact string. “Names the correct policy section and does not invent a clause number” is testable and survives paraphrasing; an exact-match assertion against a paragraph of prose will fail on every harmless rewording and train your team to ignore red builds.
Version the set in the same repository as your code, review changes to it in pull requests, and treat weakening a test case as a design decision requiring an explanation. The moment your golden set becomes editable by whoever is trying to ship, it stops being a control.
Use a model to grade a model when the quality you care about is a judgement call and you need to make thousands of them. Use a human when the judgement is the product.
LLM-as-a-judge is now a well-studied technique with a substantial research literature — see the survey on LLM-as-a-Judge for the landscape, and Judging the Judges for a careful look at where judge models align with human raters and where they are vulnerable. The consistent finding across that work is that judges can reach useful agreement with humans on well-specified criteria, and that they carry real biases — toward longer answers, toward their own family of models, toward whichever option appears first — which have to be controlled for rather than assumed away.
Practical rules that keep judges honest:
A practical hybrid works well: deterministic code checks the cheap things, a judge model screens everything, and a human reviews the disagreements and a random sample. Evidently AI’s practitioner guide to LLM-as-a-judge is a good reference for implementing the scoring layer.
An eval you run manually before a launch is a checklist. An eval that runs on every pull request is a control. The difference in outcomes is enormous and the engineering effort to get there is small.
The minimum viable setup has four pieces. First, evals run in CI on every change to a prompt, a model version, a retrieval parameter or a tool definition — prompts are code and belong under the same review discipline. Second, the pipeline fails the build on a regression against a threshold you agreed in advance, not one you negotiate after seeing the number. Third, you log real production traffic with inputs, outputs, retrieved context and the model version, so that any incident can be replayed. Fourth, you run a scheduled online eval against a sample of live traffic, because the distribution of what users actually ask drifts away from your golden set within weeks.
Close the loop by promoting production failures into the golden set. Every incident that reaches a customer should end its life as a permanent test case. Do that consistently for six months and you will have an asset no competitor can copy, because it is made of your users’ actual problems.
This discipline is the same one that separates a prototype from a system you can operate — the argument we make at length in our AI app production-ready checklist. It also constrains architecture: the more agentic your design, the more surface area there is to evaluate, which is a strong argument for the simplest pattern that solves the problem, as we set out in our guide to AI agent design patterns.
Less than you think to start, and far less than the alternative. A first golden set of a hundred cases with rubrics is typically two to four days of work from someone who understands the domain, plus a day of engineering to wire the runner into CI. The running cost is model tokens, and it is controllable: evals are the textbook batch workload, and providers price them accordingly. OpenAI’s Batch API documentation sets out a 50% discount against synchronous pricing in exchange for a 24-hour completion window, which suits an overnight eval run precisely.
The larger and more honest cost is human review time. Budget a few hours a week for a domain expert to look at failures and disagreements. Teams that skip this end up with a green dashboard and an unhappy customer base, which is the most expensive outcome available.
Weigh that against the counterfactual. The failure modes Gartner attributes to cancelled agentic projects — escalating cost, unclear value, inadequate risk controls — are all things an eval programme addresses directly. It is the cheapest insurance in the AI stack.
Ship that, then improve it. A crude eval running on every commit beats an elegant one that exists in a document.
A benchmark measures general model capability against a public dataset, so every lab can compare models. An eval measures your product’s behaviour on your data and your definition of success. Benchmarks help you shortlist a model; only evals tell you whether your application works.
Fifty to a hundred well-chosen cases is enough to catch meaningful regressions and small enough for a human to review the results. Coverage of your real failure modes matters far more than volume — a thousand near-identical happy-path cases will detect nothing.
For well-specified criteria with a clear rubric, yes, with controls. Research on LLM-as-a-judge shows judges can reach useful agreement with human raters, but also documents biases toward longer answers, toward a judge’s own model family, and toward whichever option is presented first. Randomise ordering, calibrate against human graders on a sample, and keep humans in the loop for safety-critical or regulated decisions.
They slow down the first week and speed up every week afterwards. Once a scoreboard exists, prompt and model changes can be merged on evidence in minutes instead of being argued about, and regressions are caught before a customer finds them.
Continuously. Add every production failure as a permanent case, and review the set quarterly against what users are actually asking. Real query distributions drift within weeks of launch, so a static set slowly stops representing your product.
Yes, and it is cheaper there. A single AI-powered field in an otherwise conventional app still has a failure mode, still degrades on a model upgrade, and still needs a handful of test cases and a schema check. The cost scales with the number of behaviours you care about, not the size of the app.
Neomeric, a Melbourne-based AI product and consulting company — and the team behind NeoMind, Australia’s onshore AI teammates platform — builds evaluation into every AI system we ship, because it is the only honest way to tell a client their product is working. If you want a second opinion on how to test what you are building, talk to us.
Neomeric is a Melbourne AI product studio — 7+ products shipped, including our own. Start with a free 15-minute scoping call, or a 2-week Build Sprint at A$6,900 fixed, fully credited toward your pilot.
What an AI MVP really costs in Australia in 2026 — line-item budgets, the traps that blow them out, and how to scope a build that pays for itself.
AI Insights
AI API costs climbing? Seven proven levers to cut LLM spend in 2026 — caching, batching, routing, output caps and metering. Start with the free cost guide.
AI Insights
AI for NDIS providers in Australia: where it helps, where it must not go, and the onshore compliance essentials. Read the 2026 practitioner guide now.
AI Insights
AI agent design patterns explained: prompt chaining, routing, orchestration, evaluator loops and more, plus how to pick the right one. Start building.