AI Insights
Red-teaming an LLM app means deliberately attacking your own product the way a hostile user, a poisoned document or a compromised tool would, before a customer does it for you. It is not the same as writing evals: evals check that the app does what it should, red-teaming checks what it does when someone tries to make it misbehave. This how-to gives you a seven-step process you can run in a week with open-source tooling, sized for a small team shipping an AI product in Australia rather than a bank with a dedicated security function.
Because an LLM processes instructions and data through the same channel, and no amount of input validation fully separates them. Prompt injection has held the number one position in the OWASP Top 10 for LLM Applications since the list began, and the 2025 edition shifted its emphasis from users jailbreaking a chat box to indirect injection, where the attack arrives inside a web page, a PDF or an email your app was asked to read. A conventional penetration test will find your SQL injection and your broken auth. It will not find that a supplier’s invoice PDF can instruct your agent to approve itself.
The US standards body NIST makes the same point in its Generative AI Profile, NIST AI 600-1, which recommends red-teaming both before and after deployment and treats it as recurring work that repeats whenever the model, prompts or tools change. Neomeric, a Melbourne-based AI product and consulting company — and the team behind NeoMind, Australia’s onshore AI teammates platform — runs the process below on every agentic product before pilot, and again on every model upgrade.
Start with a one-page threat model, not a tool. List the app’s inputs (user chat, uploaded files, retrieved documents, tool results, other agents), its capabilities (which tools it can call, what data it can read, what actions it can take) and its hard limits (must never reveal another customer’s data, must never send an email without confirmation, must never give medical advice). Every red-team test you run later is a probe against one line on this page. If you cannot write the line, you cannot test for it.
Map each capability to the OWASP categories it exposes. An app that reads external web pages is exposed to indirect prompt injection; an app with tool access is exposed to excessive agency; an app with a long-running loop is exposed to unbounded consumption. Most small products have three or four live categories, not ten.
Honest cost benchmarks, the hidden costs vendors don’t quote, and a 10-line scoping worksheet.
Get the free Australian AI MVP Cost Guide 2026 — we’ll email it straight to you.
Use a scanner to sweep the obvious attacks before you spend human hours. NVIDIA’s open-source garak plays the role of a vulnerability scanner for LLMs, with a large library of probe modules covering injection, jailbreak templates, encoding bypasses, data leakage and toxic output; a 2026 comparison of the main open-source tools describes garak as the broadest first sweep, PyRIT as the tool for surgical follow-up and Promptfoo as the tool for keeping fixes from regressing. Point the scanner at your app’s actual endpoint, not the raw model, because your system prompt, retrieval layer and tool wiring are what you are testing.
Expect noise. A broad scan will flag things that are not real risks for your context. Triage against the threat model from Step 1 and keep only the findings that violate a written rule.
Indirect injection is where most real-world compromises now happen, so test every path where untrusted content reaches the model. Plant instructions in a test web page your app is asked to summarise, in a PDF it is asked to extract from, in a calendar invite, in a customer’s support ticket and in a tool’s JSON response. The instruction should try to do something observable: exfiltrate the system prompt, call a tool it should not, or change the answer given to a different user.
The fix is structural, not a better system prompt. Untrusted content should be clearly delimited, tool results should be treated as data, and any action with side effects should require a check that the instruction originated from the user, not from content. Our guide to AI guardrails and structured outputs covers the patterns that make this enforceable in code.
Single-shot probes miss attacks that unfold over a conversation. Microsoft’s open-source PyRIT framework, built from the scripts its own AI Red Team used from 2022, lets you compose an attacker model, converters that rewrite prompts (encoding, translation, persona framing) and scorers that judge whether the target misbehaved, then run them as an automated multi-turn orchestrator. Use it to chase the three or four highest-severity findings from Steps 2 and 3, and to test whether a patient attacker can get there in ten turns when they could not in one.
Keep the scoring honest. An LLM judging another LLM will produce false positives and false negatives; sample its verdicts by hand, especially anything marked safe.
If your app can call tools, test whether it can be talked into calling the wrong one, calling it with the wrong arguments, or calling it in a loop. Give the red-team account the same permissions as a real customer and try to reach data or actions outside that scope through the model. Then try to make the app expensive: recursive tool calls, enormous documents, prompts that trigger maximum-length outputs. Unbounded consumption is on the OWASP list because a runaway agent can cost more in an afternoon than a month of normal traffic.
Fixes here are boring and effective: least-privilege credentials per tool, hard caps on tool calls and tokens per session, and a kill switch you have actually tested. If you have not instrumented these limits yet, our guide to AI observability and monitoring explains what to log so you can see an attack while it is happening.
A red-team finding that is fixed once and never re-tested will come back on the next model upgrade. Convert each confirmed finding into a test case in your eval suite: the attack prompt, the context that made it work, and an assertion about what a safe response looks like. Promptfoo is well suited to this because it runs from the command line and slots into CI, so a pull request that reintroduces the vulnerability fails before it ships. If you already have an eval harness, the same harness should hold your adversarial cases; our guide to AI evals covers how to structure it.
Red-teaming is a cadence, not a milestone. Re-run the automated sweep on every model change, prompt change and new tool, and run a human-led session at least once a quarter or before any major release. Write down what you tested, what you found and what you fixed: for Australian businesses handling personal information, that record is also your evidence of reasonable steps under the Privacy Act if something later goes wrong. For a wider view of the controls around the model, see our AI app security guide.
Evals check that the app does what it should on expected inputs. Red-teaming checks what the app does when someone deliberately tries to make it misbehave, including through content it reads and tools it calls. You need both, and confirmed red-team findings should become permanent eval cases.
A broad scanner such as garak for the first sweep, Microsoft’s PyRIT for adaptive multi-turn follow-up on the serious findings, and Promptfoo to run the resulting regression tests in CI. All three are free and open source.
Run the automated sweep on every model, prompt or tool change, and a human-led session at least quarterly or before a major release. NIST’s Generative AI Profile recommends red-teaming before and after deployment as recurring work.
Yes. Prompt injection is ranked first in the OWASP Top 10 for LLM Applications, and the 2025 edition emphasises indirect injection through documents, web pages and tool results rather than only direct jailbreaks in the chat box.
No. The provider tests the raw model. Your risk lives in your system prompt, retrieval layer, tool wiring and permissions, so you must test your own endpoint end to end.
Neomeric is a Melbourne AI product studio — 7+ products shipped, including our own. Start with a free 15-minute scoping call, or a 2-week Build Sprint at A$6,900 fixed, fully credited toward your pilot.
What an AI MVP really costs in Australia in 2026 — line-item budgets, the traps that blow them out, and how to scope a build that pays for itself.
AI Insights
AI product pricing models compared: per-seat, usage-based and outcome-based, with a hybrid design method and an Australian lens. Read the guide first.
AI Insights
AI for dental practices in Australia: what AI teammates handle, what stays clinical, Ahpra and privacy rules, onshore hosting, setup time. Read the guide.
AI Insights
How to build an AI product cost model: trace one request, price tokens with caching, model real usage, add non-token costs, stress-test margin. Start now.