Online evals score your AI app’s real production traffic, so you find quality problems that a pre-launch test set never saw. They complement offline evals rather than replace them: offline tests protect you before a change ships, and online evals tell you what is happening after it does. Here is a practical way to set them up without drowning in data or cost.<\/p>\n
Offline evals run a fixed set of test cases against your system during development. Online evals score live traces from real users as they arrive. Anthropic’s engineering team describes production monitoring as something that “kicks in post-launch to detect distribution drift and unanticipated real-world failures”<\/a>, and notes that it reveals real user behaviour at scale but is reactive, since problems can reach users before anyone notices.<\/p>\n That last point is why you need both. Offline evals are your gate; online evals are your smoke alarm. If you have not built the offline layer yet, start with our guide to how to test AI products<\/a>, then come back here. The pilot stage, where this all begins, is covered in our AI pilot to production playbook<\/a>.<\/p>\n Score the few qualities that would hurt you if they slipped, not everything you can measure. A sensible starting set has four checks:<\/p>\n Write each check as a yes\/no question with a clear definition. Vague scores like “quality 1 to 10” drift, and they are hard to act on.<\/p>\n Honest cost benchmarks, the hidden costs vendors don’t quote, and a 10-line scoping worksheet.<\/p>\n Get the free guide<\/a><\/div>\n Use sampling and filters. Langfuse’s documentation on online evaluation<\/a> describes a rule that decides which incoming observations get scored, using filters, a sampling rate and one or more evaluators. For LLM-as-a-judge evaluators it advises checking the estimated cost and lowering the sampling rate if needed. The same idea works in any tooling: you do not need to judge every conversation to learn whether quality is moving.<\/p>\n A practical sampling plan looks like this:<\/p>\n Use an LLM judge for qualities that need language understanding, such as relevance, tone or whether an answer matches the source material. Langfuse describes core evaluation concepts<\/a> that include model-based scoring alongside manual review and user feedback, and these methods work best together. Use plain code for anything you can check exactly, such as format, required fields or banned phrases, because code is cheaper, faster and does not disagree with itself.<\/p>\n Whatever judge you use, read the evidence. Anthropic’s guidance is blunt that you will not know if your graders are working well unless you read the transcripts and grades from many trials, and it recommends sampling transcripts to read weekly. Treat the judge as an instrument that needs calibrating, not as the truth.<\/p>\n Connect scores to decisions before you need them. Set an alert threshold for each check, name who gets the alert and write down what they do first. A falling grounding score might trigger a review of the knowledge base; a rising handover failure rate might trigger a prompt change and a rerun of your offline tests.<\/p>\n Then close the loop: every real failure you find should become a new offline test case. Over time your offline set stops being a guess and becomes a record of what has actually gone wrong. Our AI observability and monitoring guide<\/a> covers the tracing layer that makes all of this possible.<\/p>\nWhat should you score on live traffic?<\/h2>\n
\n
Free: The Australian AI MVP Cost Guide 2026<\/h4>\n
How do you score production traces without scoring everything?<\/h2>\n
\n
When should you use an LLM as the judge?<\/h2>\n
How do you turn online scores into action?<\/h2>\n
What about privacy when you log and score real conversations?<\/h2>\n