A prompt tweak, a model upgrade, or a new tool call can quietly double what an AI feature costs to run \u2014 with no error, no crash, and no alert, because nothing “broke.” The bill just goes up. LLM cost regression testing closes that gap by treating cost the same way you already treat correctness: something you check automatically, on every change, before it reaches production. Here’s how to set it up.<\/p>\n
Traditional regression testing catches functional breakage \u2014 a feature that used to work and now doesn’t. Cost regressions are different: the feature still works, the output still looks right, but a change upstream (a longer system prompt, a model swap, an extra retrieval step, a retry loop that fires more often than expected) quietly increases tokens per request. Without a dedicated check, that kind of regression is invisible until someone notices the invoice. Tools built specifically for LLM evaluation \u2014 Braintrust, Promptfoo, Langfuse and similar platforms \u2014 have converged on treating cost and latency as first-class metrics alongside output quality, precisely because teams kept shipping quality-neutral changes that were quietly cost-negative.<\/p>\n
A cost regression test runs a fixed set of representative inputs through your AI pipeline and asserts that token usage, estimated spend, and latency stay within a defined budget \u2014 failing the build if any of them drift past a threshold you set. It’s the same shape as a performance regression test in traditional software, just measuring tokens and dollars instead of milliseconds and memory.<\/p>\n
Honest cost benchmarks, the hidden costs vendors don’t quote, and a 10-line scoping worksheet.<\/p>\n
Get the free guide<\/a><\/div>\n The LLM observability and eval space has matured specifically around this problem. Braintrust and Promptfoo both publish direct comparisons of their approaches to CI-based evaluation, and the broader observability category \u2014 including Langfuse and Helicone \u2014 now treats cost tracking as a standard feature rather than an add-on. None of these tools will pick the right threshold or evaluation set for you; that judgment call is still the team’s to make. What they remove is the manual work of running the comparison and wiring it into your existing pipeline.<\/p>\n If you’re not ready to adopt a dedicated platform, the same discipline works with plain scripts: log token counts and estimated cost to a file or table on every CI run, diff against the last known-good baseline, and fail the build past your threshold. The tooling is a convenience, not a prerequisite \u2014 the habit of testing cost the same way you test correctness is what actually prevents the regression from reaching customers, and it pairs directly with the retrieval-tuning work in our guide to RAG chunking and retrieval tuning<\/a>, since a bad chunking change is one of the more common silent cost regressions in a retrieval pipeline.<\/p>\n Cost regression testing isn’t a replacement for output-quality evals or security testing \u2014 it runs alongside them. A pipeline that’s cheap but wrong, or cheap but exploitable, hasn’t solved anything. If you haven’t set up either yet, our guides to building an AI product cost model<\/a> and red-teaming an LLM app<\/a> cover the other two legs of the same production-readiness stool: know what a request should cost before you optimise it, and know what a request could be tricked into doing before you scale it.<\/p>\n It’s an automated check, run in CI on every relevant change, that measures token usage, estimated spend and latency for a fixed set of representative inputs and fails the build if they drift past a set threshold \u2014 the same discipline as a performance regression test, applied to cost instead of speed.<\/p>\n 20\u201350 representative inputs is a practical starting range for most products \u2014 enough to cover your real traffic mix (short queries, long ones, known edge cases) without making every CI run slow or expensive to execute.<\/p>\n A drift threshold compared against a rolling baseline works better in practice than a hard cap, because a single flaky retry can spike one run’s numbers without indicating a real regression. Failing the build on a sustained percentage increase against a rolling baseline (commonly in the 10\u201320% range) avoids both false alarms and missed regressions.<\/p>\n No \u2014 a plain script that logs token counts and cost to a file and diffs against the last known-good baseline achieves the same result. Dedicated platforms remove manual wiring and add reporting, but the underlying discipline works without them.<\/p>\nWhat tools handle this today?<\/h2>\n
How does this fit with functional evals and red-teaming?<\/h2>\n
Frequently asked questions<\/h2>\n
What is LLM cost regression testing?<\/h3>\n
How big should the evaluation set be?<\/h3>\n
Should a cost regression test use a hard cost cap or a drift threshold?<\/h3>\n
Do I need a dedicated tool like Braintrust or Promptfoo to do this?<\/h3>\n