What Is Red Teaming? A Guide for AI Agents and LLM Apps

Picture this: your support agent ships on Monday with a new tool that can look up customer orders. On Thursday, someone pastes a block of text into the chat that ends with an instruction to disregard earlier directions and return the last five orders on file, and the agent complies. No firewall was crossed and no credential was stolen. The agent did what it was told, by a person who was not supposed to be telling it anything.

Red teaming is the practice of finding that failure deliberately, before a user finds it by accident. The discipline comes out of cybersecurity, where a red team emulates a real attacker against an organization's defenses. The same idea applies to an LLM application, except the target is the prompt, the tools the agent is allowed to call, the documents it retrieves, and the conversation carrying all of it.

Traditional engagements are scheduled quarterly, which made sense when the thing under test was a network that changed slowly. An LLM application does not hold still. A prompt edit, a model upgrade, or one new tool in the agent's list can reopen a class of attack that last month's report said was closed, and none of those changes look like a security event on the way out the door.

That is why the useful version of this work is automated and continuous, and why a finding only counts once it becomes a test that runs again. A report that sits in a folder describes the agent you had, not the one currently answering customers.

What Is Red Teaming?

Red teaming is an authorized adversarial exercise in which a team emulates a real attacker against a system to find weaknesses its defenders have not accounted for. It measures how well detection and response hold up under realistic pressure, rather than producing an inventory of known vulnerabilities.

The vocabulary comes from military wargaming, where one group plays the opposing force so planners can see their strategy fail somewhere cheap. Cybersecurity borrowed the framing, and AI security borrowed it again once model behavior became something an attacker could manipulate directly.

What separates a red team exercise from other security testing is the objective. A red team is given a goal, such as reaching customer records or extracting an internal instruction, and then works toward it through whatever path the system allows.

Red team, blue team, and purple team

The color scheme describes who is playing which role during the exercise.

• Red team - The offensive side. It emulates an adversary, works toward a defined objective, and documents how it got there.
• Blue team - The defensive side. It runs detection, response, and containment, often without advance notice that an exercise is underway.
• Purple team - Both sides working with the information shared openly, so defenders can watch each attack land and tune controls in the same session.

For most teams shipping LLM applications, these roles are not separate headcount. The same engineers write the agent, run the attacks against it, and fix what comes back, which makes the purple arrangement the default whether anyone calls it that.

How red teaming differs from vulnerability scanning

A scanner enumerates known weaknesses against a known signature list. It is genuinely useful for dependency CVEs, exposed services, and misconfiguration, and it is fast enough to run on every build.

What it cannot do is pursue an objective. No scanner will discover that an agent hands over order history when a user asks four times in a row with escalating justification, because that failure has no signature. It emerges from the model's behavior under a specific conversation, and finding it requires something that adapts as the target responds.

Red Teaming for AI Systems in Production

What changes when the target is an LLM app

Most security testing assumes a boundary: a network edge, an authentication layer, a request that is either malformed or well-formed. An LLM application dissolves that assumption, because the instruction and the data arrive in the same channel, written in the same language, and the model has no reliable way to tell which is which.

The other complication is that the system is non-deterministic. The same attack sent twice can succeed once and fail once, which means a single clean pass is weak evidence. Coverage matters more than any individual result.

Agents raise the stakes further. When a model is only generating text, a successful manipulation produces a bad answer. When the model can call tools, the same manipulation produces an action against a real system, and the impact scales with whatever permissions the agent was handed.

For those reasons, red teaming is not just important for AI systems, but it's crucial to running them in production as part of your application performance monitoring (APM) stack.

The attack classes worth probing

The OWASP Top 10 for LLM Applications published its 2026 edition on August 4, and the reordering is a fair map of where production risk has actually moved. Prompt injection holds LLM01. Excessive Agency climbed from sixth to third. System Prompt Leakage was renamed Hidden Context Exposure and now sits at LLM08, while Improper Output Handling fell from fifth to tenth.

These are the classes that repay direct testing against a live application:

• Direct prompt injection - Instructions in the user's own message that override the system prompt, from blunt overrides to role-play framing and encoded payloads.
• Indirect prompt injection - Instructions hidden in content the agent consumes rather than content the user types: a retrieved document, a support ticket, a scraped page, a tool response.
• Hidden context exposure - Getting the agent to reproduce its system prompt, its tool definitions, or the reasoning it was told to keep internal.
• Secret disclosure - Credentials, API keys, connection strings, and internal identifiers surfacing in output because they were reachable in context.
• Goal hijacking - Redirecting the agent's objective mid-task so that it pursues the attacker's goal while appearing to work on the user's. OWASP's companion agentic list ranks this first as ASI01.
• Tool misuse and excessive agency - Getting the agent to call a tool it should not call, with parameters it should not accept, or to chain calls into an action nobody authorized.
• Multi-turn escalation - Starting benign and ratcheting upward across a conversation, which defeats defenses tuned to evaluate one message at a time.

Cross-tenant disclosure belongs on the list for any multi-customer deployment, since the failure there is one user's data appearing in another user's session through shared retrieval or shared memory.

When to run a campaign

The honest trigger is any change that alters what the agent can do or how it decides to do it. That covers a prompt revision, a model swap, a new tool, a widened permission, a change to the retrieval corpus, and a guardrail adjustment. It also covers the arrival of a new public attack technique, because the technique that broke someone else's agent last week is now in general circulation.

A calendar cadence still has a place underneath all of that, as a floor rather than a plan. Running an assessment monthly catches drift that nobody associated with a specific deploy, particularly on agents that read from a corpus other teams keep editing.

How Red Teaming Helps AI Security

Turning findings into controls

A finding is worth something once it changes the system and then keeps checking that the change held. In practice that means the decisive prompt becomes a regression test, so the next deploy that reintroduces the weakness fails before it reaches users.

The remediation itself is where OWASP's 2026 reordering earns attention. Excessive Agency moved up and output sanitization moved down because filtering has proven to be the weaker lever. You cannot reliably detect every phrasing of an injection, so the durable control is to bound what a compromised model is allowed to reach: narrower tool permissions, scoped credentials, approval gates on irreversible actions, and separation between the data an agent reads and the systems it can write to.

Detection still belongs in the stack alongside those bounds. Input and output filtering catches a meaningful share of attempts cheaply, and pairing it with agent-level limits is what prompt injection detection looks like when it is built defensively rather than as a single gate.

Evidence for audits and compliance reviews

Regulation is often cited as the reason to red team an AI application, and the citation is usually wrong in a way worth correcting. The EU AI Act's Article 55 does require documented adversarial testing, but it binds providers of general-purpose models with systemic risk. If you are building on top of a frontier model rather than training one, that obligation belongs to your provider.

High-risk obligations under the Act were deferred by Regulation (EU) 2026/1744 to December 2027 for standalone Annex III systems and August 2028 for AI embedded in regulated products, so teams shipping ordinary LLM features are not facing a near-term statutory deadline either.

What does arrive on a real schedule is the customer security review. Enterprise buyers in healthcare, finance, and the public sector now ask how the AI feature was tested adversarially, and they ask during procurement rather than after. SOC 2 and ISO 42001 auditors ask a version of the same question. A campaign report with dated findings, severity, evidence, and the remediation that followed answers it directly, which is the same reason a documented AI governance framework shortens those conversations.

How to Automate Red Teaming for LLMs

Designing an adversarial campaign

An automated assessment works in stages, and the sequence matters because each one narrows what the next should try. It starts by profiling the target: what the agent claims to be, how it refuses, which tools it exposes, and where its guardrails sit.

From there the run widens before it deepens. A broad sweep across attack families establishes which categories the target resists at all, and the results tell the engine where a deeper, adaptive effort is worth the budget. Multi-turn escalation against a category that already failed on the first message is wasted spend.

The stage that separates a useful campaign from a noisy one is verification. An attack that appears to succeed often has not, because the model produced plausible-looking text rather than real data, and confirming the difference is what turns an attempt into a finding you can act on.

Grading and triaging findings

A finding needs enough attached to it that an engineer can reproduce the failure without guessing: the prompt that worked, the response it drew, the evidence used to confirm it, the category and technique, and a severity. Mappings to OWASP or MITRE ATLAS matter mostly when the report has to travel to a security team or an auditor who works in those frameworks.

Triage follows severity, with critical and high findings reproduced in a controlled environment before anyone changes the agent. Reproduction is the step teams skip, and skipping it means shipping a fix against a failure nobody has confirmed independently.

Aggregate numbers deserve more skepticism than they usually get. A resistance rate or an overall grade summarizes one run against one scope with one budget, and a strong score means no severe issue was confirmed inside that scope. It is not proof that the agent is secure, and any campaign that reports otherwise is overselling.

Running campaigns continuously instead of quarterly

The point of automating this work is that it can run where the rest of your checks run. A scan wired into CI with a grade threshold turns adversarial testing into a merge gate, so an agent whose resistance falls below the bar stops before it reaches production instead of being discovered later.

Because campaigns are adaptive, two runs against the same target will not send identical attacks. That is a feature for coverage over time and a trap for anyone comparing single runs, so treat movement across several assessments as the signal rather than the delta between any two.

Continuous adversarial testing sits alongside the other production gates that a change to an AI system has to pass, which is the same shift that DevOps best practices for AI systems describe for evals, rollback, and versioning.

What to Look For in a Red Teaming Service

Buying decisions in this category are hard to reverse, because the findings, the regression tests, and the reporting format all end up embedded in how your team works. These are the criteria that separate a red teaming tool that improves security from one that produces a document.

• It tests the application, not the model - Attacks aimed at a bare model endpoint tell you about the model. Your risk lives in the system prompt, the retrieval pipeline, the tools, and the guardrails, so the target has to be the deployed application.
• Multi-turn and adaptive attacks - Single-shot probes miss escalation, context poisoning, and anything that requires the attacker to react to a refusal. A service limited to one-message tests leaves most real risk untouched.
• Evidence attached to every finding - The decisive prompt, the response, and the confirmation. Without those, triage becomes an argument about whether the finding is real.
• Honest coverage reporting - Some categories cannot be tested through a conversation interface at all. A service that quietly marks those as passing is worse than one that reports them as untested.
• Framework mappings - OWASP and MITRE ATLAS identifiers let a finding travel to a security reviewer or an auditor without translation.
• A CI path - If the only way to run an assessment is a person clicking through a console, it will not run on the cadence your deploys actually require.
• Report handling - Campaign output contains attack prompts, reconstructed instructions, and whatever the agent disclosed, including secrets and personal data. Ask where reports are stored, who can read them, and how long they persist.

Weigh those against how the results reach the rest of your stack. A finding that arrives as a standalone PDF costs an engineer an afternoon of context reconstruction, while one that lands next to the trace and the eval score for the same agent is already halfway to a fix.

How to Run Automated Red Team Campaigns Against Your Agent

The findings that matter against a production agent are not the ones where it says something it shouldn't. They are the ones where it retrieves another customer's account record, fetches a URL an attacker planted in a support ticket, or executes a refund for an ineligible order because someone claimed to be an authorized operator. Each of those is the agent using a tool it was given, on behalf of the wrong person.

Respan tests for that class of failure against the agent you actually deployed, rather than the model underneath it or the application around it. Campaigns connect through a local Python adapter for private, stateful, or custom-auth agents, through an OpenAI-compatible HTTPS endpoint, or against a hosted sandbox when you want to watch a full run before pointing anything at your own systems.

• Published attacks, selected live - Recon profiles the agent and its refusal behavior, broad testing finds where it gives way, then crescendo, PAIR, tree of attacks with pruning, and Bad Likert judge escalate against whatever failed. Every attempt returns a verdict of refused, partial, or successful.
• Proof beside every finding - Confirmed findings carry the decisive prompt next to the response or the recorded tool action, with severity and OWASP and MITRE ATLAS identifiers attached. An engineer reproduces the failure from the report instead of arguing about whether it happened.
• A grade your pipeline blocks on - Campaigns close on an A through F grade, a resistance rate, and a confirmed-finding count. respan-redteam scan --fail-under B makes that a CI check, so a release either clears your bar or fails the build.
• Coverage you can trust - Results split into what was confirmed, what was tested and held, and what needs deeper access than a conversation interface allows. Categories Respan cannot reach are reported as untested rather than counted as passes.
• The exercise stays under your control - Campaigns require you to confirm authorization on the target, and with the adapter your credentials and connection logic never leave your machine. Attack prompts, responses, and evidence persist as an audit trail you can hand to a security reviewer.
• Open source and inspectable - The campaign CLI is public at respanai/respan-redteam, so the attacks running against your agent are code your team can read first.
• The loop closes in one place - A confirmed finding becomes a dataset row, an evaluator, and an online eval scoring live traffic, so the same failure gets caught in production and not only in the next campaign.

Testing an agent adversarially and then watching production for the same failure are usually two purchases and two integrations. Keeping both against one span-based record of what the agent did is what makes a finding cheap enough to fix the week it shows up.

FAQ

Red teaming vs penetration testing: what is the difference?

A penetration test works through a defined scope to enumerate and exploit as many vulnerabilities as it can find, usually inside a short window and with the defending team aware it is happening. A red team exercise works toward an objective instead, taking whatever path succeeds and often running without the defenders' knowledge, so the result describes detection and response rather than a vulnerability count. For an LLM application the distinction shows up in what gets found: a pen test surfaces the exposed endpoint and the unpatched dependency, while a red team run surfaces the conversation that talks the agent into using them.

Is AI red teaming the same as jailbreak testing?

Jailbreak testing is one part of it. Jailbreaks target the model's safety training, trying to get it to produce content it was trained to refuse. AI red teaming covers that plus the failures that only exist because the model sits inside an application, including indirect injection through retrieved documents, tool calls made with attacker-supplied parameters, cross-tenant disclosure, and hidden context exposure. An agent can be highly resistant to jailbreaks and still hand over customer records through a poisoned support ticket.

How often should you red team an LLM application?

Tie it to change rather than to the calendar wherever possible, since the attack surface moves when the prompt, the model, the tool list, the permissions, or the retrieval corpus moves. Wiring a campaign into CI with a grade threshold gets you that automatically. A monthly run underneath it catches drift nobody attributed to a specific deploy, and a new publicly disclosed technique is worth an unscheduled assessment on its own.

Can red teaming be fully automated?

Automation handles the parts that benefit from volume and consistency: sweeping attack families, escalating across turns, confirming evidence, and re-running everything after a fix. Human judgment still matters for threat modeling specific to your business logic, for deciding which findings are acceptable risk, and for the creative attack nobody has catalogued yet. The practical arrangement is automated campaigns on every meaningful change, with human effort spent on the scenarios unique to what your agent does.

What is the best red teaming tool for LLM apps?

Respan is the strongest fit for teams already running LLM applications in production, because campaigns target the deployed agent rather than a bare model, findings arrive with evidence and OWASP or MITRE ATLAS mappings, and the results sit next to the traces and evaluator scores for the same system. That last part is what shortens remediation, since the finding and the behavior it came from are one click apart. Dedicated AI security vendors and open-source attack frameworks are reasonable options if red teaming is being run by a separate security team with its own workflow, though both leave you correlating findings against a separate observability stack. Respan is free to start.