AI Insights
The fastest way to cut an AI API bill is to stop paying full price for tokens you send repeatedly, and stop paying real-time rates for work nobody is waiting on. Between prompt caching, batch processing, model routing and output limits, most production applications can take a large bite out of their inference spend without touching output quality. Here are seven levers, ordered by how much they typically return for the effort.
Because token consumption scales with context, not with customers. Every conversation turn resends the history. Every retrieval-augmented answer resends the retrieved documents. Every tool-using agent resends the full tool schema on each step. Traffic doubles and the bill quadruples, and the line item that grew is the one nobody is looking at — input tokens.
Cost is not a footnote to this problem either. Gartner has predicted that more than 40% of agentic AI projects will be cancelled by the end of 2027, with escalating costs named first among the causes. Unit economics are a product requirement, not a finance problem you can defer.
Prompt caching stores the stable prefix of your prompt — system instructions, tool definitions, style guides, a reference document — on the provider’s side, so subsequent calls that share that prefix are billed at a steeply reduced rate for the cached portion. Anthropic’s platform pricing documentation sets out how cache writes and cache reads are priced relative to standard input tokens; the read rate is a small fraction of the standard rate.
Two implementation details decide whether you actually get the discount. Put everything stable at the front of the prompt and everything variable at the end, because caching works on a prefix match — one dynamic timestamp at the top of your system prompt invalidates the whole thing. And keep the cached prefix genuinely identical across calls, including whitespace.
Honest cost benchmarks, the hidden costs vendors don’t quote, and a 10-line scoping worksheet.
Get the free Australian AI MVP Cost Guide 2026 — we’ll email it straight to you.
If a job can tolerate a delay, it should not be paying real-time prices. OpenAI’s Batch API documentation describes a 50% discount against synchronous pricing in exchange for a completion window of up to 24 hours, and Anthropic’s pricing documentation describes an equivalent batch discount. The engineering change is small: write your requests to a file, submit, collect results later.
Good batch candidates are everywhere once you look: nightly summarisation, backfilling embeddings, classifying yesterday’s support tickets, enriching a CRM, and — as we argue in our guide to AI evals — your entire evaluation suite, which is the textbook overnight workload.
Most teams pick one strong model and send everything to it. That is a reasonable default for week one and an expensive one by month six, because a large share of production traffic is classification, extraction, routing and short-form rewriting that a small model handles indistinguishably.
Build a cheap router: a small model or a few rules classify the request, easy traffic goes to the small model, hard traffic goes to the frontier model, and anything the small model is not confident about escalates. Measure the quality of both paths against the same eval set before you switch — the saving is only real if the output still passes. Our guide on choosing an AI model for your app covers how to run that comparison properly.
Output tokens are typically priced several times higher than input tokens across major providers, which makes verbosity the most expensive habit in your codebase. Set an explicit maximum output length on every call, ask for structured output rather than prose where a machine is the consumer, and stop asking models to “explain your reasoning” in responses that get parsed and thrown away.
This is also the lever with the best side effects. Shorter outputs are faster, cheaper and usually more useful to the user.
Retrieval pipelines quietly become the biggest line item in a RAG application because it is easier to send twenty chunks than to work out which three matter. Reduce top-k, rerank before you send rather than letting the model sort it out, deduplicate near-identical chunks, and cut chunk size to the smallest window that still answers your evaluation set.
Measure quality as you tighten. There is a point where trimming context starts costing you accuracy, and the only way to find it without guessing is a fixed eval set you can re-run.
Multi-agent architectures multiply token consumption, because context is re-sent at every handoff and each agent re-reads the shared state. The cost compounds silently with each additional hop. Independent analyses of caching, batching and routing strategies — including this practitioner breakdown from GMI Cloud — consistently find that architectural choices dominate per-token optimisations.
The cheapest fix is usually to use a simpler pattern. If a single call with good instructions solves the problem, do not orchestrate five agents to solve it more impressively. We set out the decision order in our guide to AI agent design patterns: stop at the first pattern that works.
You cannot optimise a number you do not have. Tag every API call with the feature, the customer or tenant, and the model version, then chart cost per feature and cost per active user weekly. Almost every team that does this for the first time discovers one feature consuming a wildly disproportionate share of the bill, and a handful of accounts driving costs that exceed what they pay.
That chart is also what turns an AI feature into a business case. Cost per successful task, tracked over time, is the number that answers “is this worth running?” — and it is the number an investor or a board will ask for.
Instrument first (lever 7), because everything else is guesswork without it. Then cap output tokens and cache your stable prefix — both are hours of work with immediate returns. Then batch your offline jobs. Only then take on routing and retrieval tuning, which need an eval set to do safely, and finally revisit architecture if agent handoffs are still dominating the bill.
Neomeric, a Melbourne-based AI product and consulting company — and the team behind NeoMind, Australia’s onshore AI teammates platform — runs this exercise on every system we take to production, because a product with unclear unit economics does not survive its first serious growth month.
No. Caching stores the processed prefix of your prompt so it does not have to be paid for at full input rates again. The model sees the same tokens and produces output the same way; only billing and latency change.
Latency. Batch requests are asynchronous and providers document completion windows of up to 24 hours, so batching only suits work no user is waiting on. It is unsuitable for anything in a live request path.
Not always. A smaller model that needs three attempts, longer prompts or heavier post-processing can cost more end to end than one call to a stronger model. Compare cost per successful task against a fixed evaluation set, not cost per token.
It depends entirely on the workload shape. Applications that resend large stable context on every call have the most to gain from caching; applications with heavy offline processing gain most from batching. Rather than trusting a headline percentage, instrument your own spend by feature first and size the opportunity from your own numbers.
Only at sustained high volume, and only after the cheap levers are exhausted. Self-hosting trades a variable API bill for fixed GPU capacity plus engineering and operations time, which usually raises total cost for small and mid-sized workloads even when the per-token number looks better.
Neomeric is a Melbourne AI product studio — 7+ products shipped, including our own. Start with a free 15-minute scoping call, or a 2-week Build Sprint at A$6,900 fixed, fully credited toward your pilot.
What an AI MVP really costs in Australia in 2026 — line-item budgets, the traps that blow them out, and how to scope a build that pays for itself.
AI Insights
AI evals explained: how to test an AI product before launch, build a golden set, use LLM-as-a-judge, and catch failures in CI. Read the 2026 builder guide.
AI Insights
AI for NDIS providers in Australia: where it helps, where it must not go, and the onshore compliance essentials. Read the 2026 practitioner guide now.
AI Insights
AI agent design patterns explained: prompt chaining, routing, orchestration, evaluator loops and more, plus how to pick the right one. Start building.