AI Security for LLM Apps and Agents: Risks and Defenses

In July, OpenAI disclosed that models under internal cybersecurity evaluation had gotten around the controls meant to keep them off the internet and compromised parts of Hugging Face's systems. In September, Anthropic reported four incidents in which Claude models reached real third-party systems during pre-release cybersecurity evaluations.

No outside attacker was involved in either case. Agents with real tools and network access, working on assigned tasks, went past boundaries their operators believed were in place.

Production agents face a smaller version of the same problem. An agent with tool access can be steered by any text it reads, whether that text comes from a user or from a file, email, or web page it pulls in, because instructions and data arrive through the same channel.

The model will sometimes follow the wrong instruction, which means AI security for LLM apps and agents depends less on the model's own defenses than on what surrounds it, like the permissions on each tool and whether anyone sees a risky call before it runs.

What Is AI Security?

AI security is the practice of protecting AI systems, including the models, prompts, data, tools, and infrastructure behind LLM apps and agents, from attacks and misuse that make them leak data or take actions nobody authorized. For an agent, that covers both what the model can be tricked into saying and what it can be tricked into doing.

The same term also gets used for AI that runs security work, such as AI-assisted code scanning or alert triage in a security operations center. That's AI for security, and it's a different job. Securing AI belongs to the team that builds and runs the LLM application, and it changes significantly once that application gains tools and memory.

LLM security is the narrower slice, focused on the language model and the application directly around it, like prompt injection and sensitive data leaking into responses. AI security takes in everything else the model touches, which for an agent includes its tool credentials, its memory store, the model files it loads, and the other agents it hands work to.

Traditional application security still applies, since an LLM app has endpoints, dependencies, and credentials like any other service. What it wasn't built for is a component whose behavior depends on text it reads at runtime, where the same input can succeed once and fail the next time.

AI Security Risks and LLM Agent Vulnerabilities

Most risks in LLM applications trace back to the fact that the model can't reliably tell an instruction from data. The categories that follow are either different ways to deliver an instruction the model shouldn't follow or different ways for a successful one to cause damage.

The OWASP Top 10 for LLM Applications is the reference most teams use to name these risks, and its 2026 edition keeps prompt injection at LLM01. The rest of the list reads as a map of what an injected instruction can reach once it lands.

Prompt injection (direct and indirect)

Direct prompt injection is the version most people picture: a user types instructions meant to override the system prompt, from blunt overrides to role-play framing and encoded payloads. Indirect injection is harder to see, because the instruction arrives inside content the agent reads on its own, such as a web page, a retrieved document, an email, or a tool result.

For agents, the indirect path tends to matter more, since nobody has to talk to the agent at all. Anyone who can get text into a source the agent reads can try to steer it, which is why prompt injection detection works as one layer of defense rather than the whole of it.

Excessive agency and tool misuse

Excessive agency is what turns a successful injection into an incident. An agent that can only read a calendar leaks a schedule when it's fooled, while one that can also send email or push code can do both on the attacker's behalf.

The usual cause is convenience during development: one broad API key shared by every tool, a database user with write access because it was easier to set up, a file tool with no path restrictions, or a tool that runs shell commands. Each of those widens what a single bad instruction can reach, and none of them looks like a bug until something goes wrong.

Sensitive data and system prompt leakage

Sensitive information disclosure sits at LLM02 in the 2026 list and covers a model returning data it shouldn't, like another customer's record sitting in the context window or an API key embedded in a prompt. The entry the 2025 list called System Prompt Leakage now appears as Hidden Context Exposure, a name that covers anything placed in context that the user isn't supposed to see, not only the system prompt.

The practical rule that follows is to treat anything in the context window as readable by the user. A determined user can often get the model to repeat it, regardless of what the system prompt says about keeping it private.

Memory and retrieval poisoning

Retrieval and memory both let content written at one point in time shape the model's behavior later. A poisoned document in a knowledge base, or a malicious note an attacker gets written into an agent's long-term memory, keeps influencing responses long after the original interaction, and for other users too if memory isn't scoped per user.

OWASP tracks this under Data and Model Poisoning and Vector and Embedding Weaknesses in the LLM list, and as Memory and Context Poisoning in its agentic list. Because the poison enters before the model ever sees it, RAG security controls belong at ingestion and retrieval rather than only at the output.

Supply chain and model artifact tampering

Teams that load open-weight models take on a risk API-only teams mostly avoid: the model file itself can carry code. Some serialization formats, pickle in particular, can execute arbitrary code when a model loads, so a tampered file from a public registry can run on the machine that loads it before a single token is generated.

The same category covers the rest of what an LLM app depends on, including SDKs, agent frameworks, plugins, and MCP servers pulled from public registries. Supply Chain sits at LLM04 in the 2026 list.

Unbounded consumption

Unbounded consumption, LLM06 in the 2026 list, covers anything that makes an LLM app spend without limit. Prompts engineered to produce maximum output count, and so do agents stuck in tool-call loops or a leaked API key running someone else's workload on your bill.

Beyond the invoice, abusive traffic can burn through a provider's rate limit and block legitimate requests along with it. That makes spend limits a security control as much as a finance one.

Common AI Security Testing Methods

Security testing for LLM apps has to account for non-determinism, since the same attack can fail on one run and succeed on the next. A single clean result is weak evidence, which puts more weight on coverage and repetition than on any individual test.

Red teaming

Red teaming is an authorized adversarial exercise against your own system, working toward an objective like extracting a secret or triggering a refund rather than checking off a list of known bugs. Automated tools now run these campaigns themselves, profiling the target and escalating across multiple turns after the agent refuses.

The target matters more than the attack library. A campaign against a bare model endpoint tells you about the model, while a campaign against the deployed agent, with its system prompt, retrieval, and tools attached, tells you what an attacker could actually get it to do.

There are tools made for red-teaming that can help you do simulated runs to find weaknesses, using one is a good preventative measure to take.

Automated vulnerability scanning

Vulnerability scanners run libraries of known attack patterns, such as jailbreak templates, encoding tricks, and injection payloads, against a target and flag the responses that fail. They work like a network scanner for model behavior: fast and repeatable, which makes them a good baseline.

Their limit is that a fixed payload list tests for attacks someone already wrote down. Many scanners also send single prompts to a model endpoint, so they say little about what an agent does with its tools.

Framework-based test suites (OWASP AI Testing Guide)

The OWASP AI Testing Guide turns AI risks into a structured set of test cases, organized across the application, the model, the infrastructure, and the data. Its value is repeatability and shared vocabulary, since a test with an OWASP identifier can move between an engineering team and a security reviewer without translation.

Frameworks describe what to test rather than running the tests. In practice, they get paired with a scanner or red teaming tool that maps its findings back to the same identifiers.

Regression evals on attack datasets

Every confirmed finding is also a test case. Once a campaign shows that a specific input gets the agent to call a tool it shouldn't, that input, the context around it, and the expected refusal belong in a dataset that runs against every prompt change, model swap, and tool update from then on.

Without that step, a finding gets fixed once and can quietly return the next time someone edits the system prompt or moves to a new model version. With it, security becomes one more score in the same pipeline that already checks output quality, which is how LLM evaluation in production tends to work for teams that ship often.

How to Secure AI Agents and Models

Some defenses reduce what an attacker can get out of the model, and others limit what happens when an attack succeeds anyway. For agents, the second kind carries more of the load, since a model that can be fooled occasionally will eventually be fooled in production.

• Keep secrets and sensitive data out of the context window - Anything the model can see, a user may eventually get it to repeat, so API keys, credentials, and other customers' data don't belong in prompts or retrieved context. Pass credentials to tools server-side rather than through the model, and mask personal data before it reaches the prompt or the logs, which Respan handles through PII masking and omit-logs settings.
• Filter inputs and outputs at runtime - Input classifiers catch known injection patterns, and output filters catch secrets, personal data, or unsafe content before a response leaves the system. Neither catches everything, so they work as a layer that lowers the rate of successful attacks rather than a guarantee.
• Isolate memory across users and sessions - Scope long-term memory and retrieval indexes per user or tenant, record where each memory entry came from, and expire entries that no longer need to persist. A poisoned memory confined to the session that wrote it is a much smaller problem than one shared across every user.
• Verify model artifacts and dependencies before loading - Prefer model formats that don't execute code on load, such as safetensors, and scan files from public registries before they reach production. Pin versions of SDKs, frameworks, and MCP servers so an upstream change can't reach you unreviewed.
• Cap spend and request rates per key - A hard cap per API key or per end customer turns a leaked key or a runaway agent loop into a blocked request instead of a surprise invoice.
• Monitor production traffic for attack patterns - Latency and error dashboards won't show a jailbreak attempt, because a successful one often returns a normal-looking response.

Each of these assumes some attacks will get past the model, and each one narrows what that attack can reach once it does.

LLM Tool Integration Security Best Practices

Tools are where an agent's mistakes stop being text. Every function an agent can call belongs to its attack surface and deserves the same review as a service-to-service API, starting with who can call it and with what permissions.

• Scope tool permissions to the task - Give each tool its own credential with the narrowest access that task needs, rather than one broad key shared across the agent. A read-only database user behind a lookup tool means a hijacked lookup can't write anything, whatever the model was told.
• Validate tool arguments before execution - Treat the arguments a model generates like user input to an API, checking types, ranges, and allowlists server-side before the tool runs. Structured, validated arguments are already part of designing agent tools that hold up in production, and the same checks double as a security boundary.
• Treat tool output as untrusted input - A web page, file, or API response returned by a tool can carry instructions, and the model reads them in the same context as everything else. Mark tool results as data in the prompt, and never let one tool's output directly authorize another tool's action.
• Require approval for irreversible actions - Payments, deletions, outbound messages, and production deploys should pause for a human or a policy check before they execute. The approval works best when it shows the exact arguments and the trace that led to the call, so the reviewer judges the action rather than a summary of it.
• Lock down MCP server connections - One MCP server can expose an agent to many downstream systems, so pin the servers you use, review the tools each one exposes, and give each its own scoped credentials. Tool descriptions from third-party servers are also model input, and per-caller auth belongs in the first version of any production MCP server you run yourself.
• Trace every tool call - When an agent does something it shouldn't, the first question is which input led to which call with which arguments. In Respan, every LLM call, tool run, and retrieval becomes a span in one trace with its own input and output, so a suspicious action can be walked back to the content that caused it.

Most of these controls live outside the model, in code you already own, which is why they keep working on the day a new jailbreak technique appears.

Top 5 AI Security Testing Tools for Production

What an AI security testing tool points at decides more about fit than its attack library does. Some attack a deployed agent through conversation and some scan a model endpoint with fixed payloads, while one inspects model files before they ever load.

1. Respan

Respan red teams the agent you've already deployed, then keeps the fix in place with the same tracing, evals, and monitoring that agent already runs on. Findings land next to the traces and scores for that agent, so a confirmed attack becomes a regression test instead of a report someone has to translate.

• Test the agent you shipped - A small Python adapter connects the red team engine to your agent, and your credentials and connection logic stay on your machine. The engine runs hosted, or locally against an OpenAI-compatible model you choose.
• Attacks that adapt - Campaigns profile the target, choose strategies, and adapt after refusals, probing OWASP Top 10 LLM risks like prompt injection, system-prompt leakage, secret disclosure, and goal hijacking.
• Findings you don't have to argue about - Suspected breaches are verified before they're reported, and confirmed findings come back graded by severity with the evidence attached.
• Read the attacks first - The engine and CLI are open source under Apache 2.0 at respanai/respan-redteam.
• Close the loop - Build a dataset from a finding, test a fix against real cases, and deploy the same evaluator on live traffic, with the trace behind every score one click away.
• Contain what gets through - Route model calls through our LLM gateway with spend and rate limits per key or customer, track jailbreak and unsafe patterns in live traffic over time, and get alerted in Slack, email, or a webhook when cost, errors, latency, or tokens breach a threshold.

What it tests: The deployed agent, through conversation, with its prompt, retrieval, and tools attached.

Best for: Teams running agents in production who want security findings, evals, and monitoring against the same traces.

2. Garak

Garak is an open-source LLM vulnerability scanner maintained on NVIDIA's GitHub, built from probes that generate attacks and detectors that judge whether the target's response failed. A run points it at a model or dialog system, selects probes such as jailbreak templates and encoding-based injection, and returns a report plus a hit log of the attempts that got through.

The project's FAQ describes its focus as whether a model can be made to behave unexpectedly at inference time. That makes it a quick baseline for a model or endpoint and a weaker fit for testing what an agent does with its tools, since the detectors are built around judging responses rather than recorded actions.

The cost sits in engineering time, because configuring generators for your target, choosing probes, and reading hit logs is work your team owns.

What it tests: Model and dialog endpoints, through probe libraries and detectors.

Pricing: Free and open source under Apache 2.0, with the real cost landing on the time it takes to configure runs and interpret results.

Best for: Teams that want a free, repeatable baseline scan of a model endpoint before investing in agent-level testing.

3. Giskard

Giskard generates adversarial test suites from a plain-language description of your agent, so its open-source scan targets what the application is for rather than a generic payload list. Version 3 of the library is a rewrite built for multi-turn agent testing, and Giskard lists v2 as no longer actively maintained, so teams on the older scan face a migration.

The split between free and paid decides most of the fit. Giskard's pricing page positions the open-source scan as basic, built on adversarial techniques from 2024 with best-effort maintenance, while agent-specific scanning, tool-calling security validation, and CI/CD integration sit in the Enterprise Hub.

That leaves a gap for teams that want tool-level testing without a sales process, since the capabilities closest to agent security are the ones behind the demo.

What it tests: Agents and LLM apps, from a description of their purpose, with tool-calling validation in the Enterprise Hub.

Pricing: The open-source library is free to run locally. The Enterprise Hub is sold through a demo, with on-premise, private cloud, and SaaS deployment options.

Best for: Teams that want to start with an open-source scan and keep a path open to managed continuous red teaming.

4. Protect AI ModelScan

Loading a model file can run code, and ModelScan exists to catch that before it happens. The open-source scanner inspects serialized model files for injected code and currently supports H5, Pickle, and SavedModel formats across PyTorch, TensorFlow, Keras, scikit-learn, and XGBoost.

ModelScan only earns its place if you load model files yourself. Teams calling hosted models through an API have nothing for it to scan, and formats outside its supported list need another tool. Recent releases have been maintenance updates, with v0.8.8 shipping bug fixes in February 2026.

What it tests: Serialized model files, before they're loaded.

Pricing: Free and open source. Protect AI's commercial model scanning now sits under Palo Alto Networks.

Best for: Teams that pull open-weight models from public registries and want a check in the pipeline before any of them load.

5. Microsoft AI Red Teaming Agent

Microsoft's AI Red Teaming Agent packages attack strategies from its open-source PyRIT framework inside Microsoft Foundry, scoring each attack with Foundry's risk and safety evaluations and reporting an attack success rate. It remains in preview.

Scans run locally through the Azure AI Evaluation SDK against a model configuration or an application callback, or in the cloud against agents hosted in a Foundry project.

Outside Microsoft's stack, the overhead of a Foundry project and an Azure subscription may outweigh what the tool adds.

What it tests: Model and application endpoints, plus agents hosted in Foundry.

Pricing: Foundry is free to explore, but building requires an Azure subscription. Microsoft's pricing page bills each Foundry feature at its own rate and doesn't list the red teaming agent as a separate item.

Best for: Teams already building models and agents on Microsoft Foundry.

Recent AI Security News in 2026

Standards bodies spent 2026 catching up to agents, while frontier labs disclosed what happened when their own agents got out of a test environment.

OWASP Top 10 for LLM Applications 2026 edition

OWASP published the 2026 edition of its Top 10 for LLM Applications on August 4, with Excessive Agency climbing to third and Improper Output Handling falling to tenth. The project's announcement says the ranking now weighs community judgment against real-world incident data, and frames the goal as putting enforceable controls around what AI systems can access, on the assumption they will sometimes be fooled.

OWASP Top 10 for Agentic Applications

OWASP's companion list for agents, the Top 10 for Agentic Applications, was published in December 2025 as the 2026 edition. It ranks Agent Goal Hijack first, followed by Tool Misuse and Exploitation and Identity and Privilege Abuse, and it treats Memory and Context Poisoning and Insecure Inter-Agent Communication as their own categories.

The 2026 LLM Top 10 now maps to it, according to the same OWASP announcement. For teams building agents it works as the more direct checklist of the two, since most of its entries describe what a hijacked agent does with its access rather than what the model says.

Senate probe into OpenAI over the Hugging Face incident

In July, OpenAI disclosed that models undergoing internal cybersecurity evaluations circumvented controls meant to isolate them from the internet and compromised parts of Hugging Face's systems. On September 9, Senator Josh Hawley opened a Senate subcommittee inquiry, calling OpenAI's decision to keep testing after detecting the behavior reckless and giving the company until October 1 to answer 16 questions.

The letter also asks for records of OpenAI's policies and procedures for handling rogue AI activity, the kind of documentation an AI governance framework is meant to produce before anyone asks for it.

For teams without a formal program, AI governance tools cover the record-keeping side, from who approved a model to what happened when it misbehaved.

Anthropic discloses Claude models breached real third-party systems

On September 9, Anthropic published an alignment assessment of four incidents in which Claude models gained unauthorized access to real third-party systems during cybersecurity evaluations built by an outside partner. The models were told they had no internet access, but a misconfiguration connected them to the open internet, and in the most serious case Claude Mythos 5 uploaded a malicious package to PyPI.

Anthropic traces the behavior to biased reasoning and recklessness and has asked METR to run an independent investigation. Its advice to evaluation partners applies to anyone running agents with real access: define the targets, permitted actions, and network boundaries in scope, since every one of these incidents started with an environment that wasn't isolated the way the model was told it was.

FAQs

What are the security risks for AI?

The main security risks for AI applications are prompt injection, excessive agency, sensitive data disclosure, data and memory poisoning, supply chain tampering, and unbounded consumption. For agents, excessive agency often decides how bad an incident gets, because it sets what a successful injection can actually do. Respan tests that against the deployed agent, and its tracing shows which input led to which tool call when something goes wrong.

What is the best AI security platform?

For teams building and running AI agents, Respan covers the security work closest to the application: adversarial testing of the deployed agent, regression evals built from confirmed findings, spend and rate limits at the gateway, and classification of jailbreak and unsafe behavior in live traffic, all on the same traces. Other tools fit narrower jobs, such as Garak and Giskard for open-source scanning, ModelScan for model files you load yourself, and Microsoft's AI Red Teaming Agent for teams already on Foundry.