AI Insights
Prompt engineering asked “what should I say to the model?” Context engineering asks a better question: “what should be in the model’s context window at this moment, and what should not?” For AI agents — systems that run over many turns, call tools, and accumulate history — that question decides whether the agent stays sharp or slowly degrades as its window fills with noise. This how-to walks through the practical steps we apply when building agents at Neomeric, a Melbourne-based AI product and consulting company — and the team behind NeoMind, Australia’s onshore AI teammates platform.
Context engineering is the discipline of curating the set of tokens a model sees at inference time — system prompt, tool definitions, retrieved documents, message history, and tool results — so that every token earns its place. Anthropic’s engineering guide frames it as the natural progression of prompt engineering: context is a critical but finite resource, and the job is to find the configuration of context most likely to produce the behaviour you want. The failure mode it prevents is familiar to anyone who has run a long agent session: the model that was precise at turn three becomes vague at turn thirty, not because the model changed but because its window filled with stale tool output and irrelevant history.
Before optimising anything, log the full assembled context of a few real requests and read it. Most teams are surprised: duplicated instructions, entire documents retrieved for a one-line answer, verbose tool results kept verbatim across turns. You cannot curate what you have not seen. Make context inspection a debugging habit, the way you would read a SQL query plan before tuning a database.
Get the free Australian AI MVP Cost Guide 2026 — we’ll email it straight to you.
The system prompt should sit between two failure modes: hardcoded if-else brittleness at one extreme, and vague high-level guidance at the other. State the agent’s role, its non-negotiable rules, and the shape of a good outcome — then stop. Every edge case you enumerate in prose is an edge case the model may over-fit to; every rule you leave implicit is one it may miss. Start minimal, add instructions only in response to observed failures, and treat the prompt as versioned code with an eval suite behind it — our guide to testing AI products with evals covers how to catch regressions when you edit it.
Honest cost benchmarks, the hidden costs vendors don’t quote, and a 10-line scoping worksheet.
Tools are context too: every definition consumes tokens and every overlapping capability creates a decision point where the model can choose wrong. Prefer a small set of tools with crisp, non-overlapping purposes and descriptions written like good documentation — what it does, when to use it, what it returns. If a human engineer would need to ask which of two tools applies, the model will guess. Constrain tool results as well: return the fields the task needs, not the full API payload. We cover the architectural side in AI agent design patterns.
Loading every possibly relevant document up front is the most common context-bloat mistake. The better pattern is just-in-time retrieval: keep lightweight identifiers — file paths, record IDs, search queries — and let the agent fetch content when the task actually requires it. Retrieval quality then becomes the bottleneck, which is a pipeline problem: chunking, freshness, and hybrid search, covered in our RAG data pipeline how-to and the underlying RAG architecture guide.
For tasks that outlive the window, decide what survives: summarise completed sub-tasks, keep decisions and constraints, drop raw tool output that has already been acted on. A useful pattern is structured note-taking — the agent maintains a short running summary of state outside the message history, and stale turns are pruned. Compaction is lossy by design; the engineering judgement is choosing what the future turns will actually need.
Order the context so that stable content — system prompt, tool definitions, reference material — comes first and volatile content last, then use provider prompt caching on the stable prefix. Anthropic’s prompt caching reduces costs by up to 90% and latency by up to 85% for long prompts, and equivalents exist across major providers. Caching rewards exactly the discipline this guide teaches: a well-factored context with a stable prefix is also the cheapest one to serve. For the wider cost picture, see our guide to AI API cost optimisation.
Context engineering governs what goes in; it pairs with guardrails governing what comes out. Schema-constrained outputs, semantic validation, and human escalation triggers turn a well-fed agent into a trustworthy one — we cover that half of the discipline in this week’s companion piece on AI guardrails and structured outputs.
Prompt engineering focuses on writing effective instructions. Context engineering manages everything the model sees at inference time — system prompt, tools, retrieved documents, history, and tool results — and curates it across every turn of an agent’s operation, treating context as a finite resource.
Usually because the context window fills with stale tool output, duplicated instructions, and irrelevant history. The model is not degrading; its inputs are. Compaction, just-in-time retrieval, and pruning keep long-running agents sharp.
No. Larger windows raise the ceiling but do not remove the need for curation — irrelevant tokens still dilute attention, add cost, and slow responses. Efficient context use matters at every window size.
Log and read the fully assembled context of real requests, then remove what is not earning its place — verbose tool results and just-in-case documents are the usual offenders. Ordering stable content first and enabling prompt caching typically follows as the second win.
Neomeric is a Melbourne AI product studio — 7+ products shipped, including our own. Start with a free 15-minute scoping call, or a 2-week Build Sprint at A$6,900 fixed, fully credited toward your pilot.
What an AI MVP really costs in Australia in 2026 — line-item budgets, the traps that blow them out, and how to scope a build that pays for itself.
AI Insights
AI deployment options for Australian builders: direct API, Bedrock, Vertex, Azure, self-hosted or hybrid. What stays onshore, what it costs. Read the guide.
AI Insights
AI for gyms and fitness studios in Australia: what an AI teammate should handle, what stays human, onshore hosting and real setup time. Read the guide.
AI Insights
AI guardrails and structured outputs in 2026: schema-enforced JSON, output validation, rail frameworks and what Australian rules require. Read the guide.