AI Red Teaming: Why Security Testing Matters More Than Safety Testing

What Is AI Red Teaming, Really?

AI red teaming has spent most of its short history answering one question: will the model say something it shouldn't? That's a safety question, not a security one, and the difference is the central argument of this article. As AI agents move from answering questions to taking actions, querying databases, reading email, and updating records, the more consequential question becomes what an agent can be made to do, not what it can be made to say.

This distinction sounds simple once it's stated, but it isn't reflected in how most organizations approach AI red teaming today. Security teams evaluating an AI agent for an internal deployment often default to testing whether it produces biased, offensive, or policy-violating output, the same testing a public-facing chatbot needs. That's safety testing. It doesn't answer the question a CISO actually needs answered before an agent gets connected to a mailbox, a customer database, or a payment system: can this agent be manipulated into doing something that damages the business?

Safety Testing vs. Security Testing: The Distinction Most Programs Miss

AI security splits into two categories that get treated as one, and that conflation is where most red teaming programs go wrong.

AI safety testing asks whether a model will produce content it shouldn't: instructions for an illegal act, biased statements, or anything that creates reputational liability if a customer sees it. If an agent says something it shouldn't, that's a genuine problem, but the organization's data is still intact, its infrastructure is still up, and no one's identity was misused. It's a reputation risk, not an operational one.

AI security testing asks a different question: can the agent be coerced into taking an action that compromises data, infrastructure, or identity? This is the domain security organizations have spent decades and tens of billions of dollars building tooling around: DSPM, CSPM, and the rest of the posture management stack, because it maps to what they've always cared about: data not leaking and systems not being compromised. Applied to AI agents, security testing checks whether the agent can be convinced to exfiltrate data, delete a database, escalate its own access, or act outside its intended permissions.

Both matter, but not equally, and not for every deployment. A customer-facing agent for a large hospitality brand carries real reputational risk if it says something offensive to a guest, so safety testing is non-negotiable there. But that same organization should care just as much about whether its agent can be manipulated into exposing one guest's reservation details to another, or into approving free upgrades it was never authorized to grant, a security failure with a much sharper business impact than a bad headline.

The calculus shifts further for internal, non-customer-facing agents built on platforms like Microsoft Copilot Studio or ChatGPT's workspace agents. If one of these agents produces an offensive response, the reputational exposure sits with the platform vendor, not the deploying organization. But if that same agent exfiltrates internal data, the liability belongs entirely to the company that connected it. For this category, which covers the majority of enterprise AI agent deployments today, security testing should be the priority, not an afterthought behind safety.

The Lethal Trifecta: Why Agentic AI Changes the Threat Model

Security researcher Simon Willison coined the term “lethal trifecta” to describe the exact combination of conditions that makes an AI agent dangerous. When all three are present, an agent becomes capable of exfiltrating data without any code execution or privilege escalation in the traditional sense, using nothing but its own legitimate access.

• Untrusted data. The agent processes content it doesn't control: web pages, emails, calendar invites, shared documents, or comments on a listing.
• Sensitive information. The agent has access to data worth protecting: personal information, credentials, financial records, or internal business data.
• A write action. The agent can take an outbound action: sending an email, updating a database, posting a message, or calling an external system.

The danger isn't hypothetical. An agent with all three conditions can be hijacked by content it merely reads, such as an email, a document, or a webpage, into using its own legitimate access to route sensitive data to an attacker-controlled destination. No malware is involved. No account is compromised. The agent simply does what it was manipulated into believing was its task.

This is why threat modeling an agent should start before any adversarial prompt is written. If an agent reads untrusted content, holds sensitive data, and can act outward, that combination is a risk on its own. The risk exists the moment the trifecta is present, the same way handing database deletion access to an untrained user is a risk whether or not that user ever presses delete.

Beyond Attack Techniques: The Risks That Actually Matter

Most discussions of AI red teaming lead with attack techniques: prompt injection, jailbreaks, roleplay framing. These matter, but they're the delivery mechanism, not the risk. Prompt injection, direct or indirect, is how an attacker gets a malicious instruction in front of an agent. The risk is what happens after.

Goal misalignment is the core risk

Every risk below is a variation of the same underlying failure: the agent was told to do X and, whether through manipulation or simple confusion, did Y instead. Zenity uses the term goal misalignment rather than “goal manipulation” deliberately: manipulation implies an attacker is always present, and increasingly, one isn't. An agent operating with a long, cluttered context window can decide on its own that deleting a production database is the fastest way to “solve” a bug report. Nothing injected the idea. As agents become more autonomous, this unmanipulated version of goal misalignment becomes more common, not less.

Destructive actions and tool misuse

Any tool connected to an agent, such as a database connector, an email client, or a code execution sandbox, can be turned toward a destructive outcome. An agent with delete access to a production database using that access is often unrecoverable.

Harvesting and data exfiltration

Harvesting is the agent actively searching connected systems for sensitive information; exfiltration is what happens to that information once collected. An agent goal-misaligned into harvesting credentials, then routing them outward, is the same attack chain researchers have already demonstrated against production enterprise AI platforms.

Persistence mechanisms

Memory is a genuinely useful agent feature and a genuinely useful attacker foothold. An attacker who implants a false memory in an agent with persistent memory doesn't need to compromise anything again. The agent keeps acting on that instruction indefinitely, and the user talking to it has no way of knowing its behavior has been redirected.

User impersonation and identity misuse

Most organizations grant an agent its own identity and assume that solves the access problem. It doesn't. In practice, users connect agents to systems using their own credentials: a coding assistant connected to an MCP server inherits the developer's access, an agent connected to Google Drive inherits the user's Drive permissions. An agent induced to send a message, approve a request, or click a phishing link does so as that person, and the recipient has no way to tell the difference.

Frameworks for AI Red Teaming

Several frameworks give AI red teaming programs a shared vocabulary. Each is useful. None is complete alone, and security-focused testing benefits from looking past any single list.

MITRE ATLAS

MITRE ATLAS (Adversarial Threat Landscape for AI Systems) is the closest AI-security equivalent to ATT&CK: a structured knowledge base of adversary tactics, techniques, mitigations, and case studies. It currently documents 16 tactics, 84 techniques, and 56 sub-techniques, and it has expanded specifically to cover agentic AI, including techniques for harvesting credentials from connected systems and discovering tool definitions that expose exfiltration paths. ATLAS is the right tool for mapping which attack techniques apply to a given deployment and building detection logic against documented adversary patterns.

OWASP Top 10 for LLMs and Top 10 for Agentic Applications

The OWASP Top 10 for Agentic Applications extends the original LLM-focused list to autonomous agent risk, covering categories like Agent Goal Hijack and Tool Misuse and Exploitation. It's a strong baseline, and it's what most vendors and buyers reference first. Zenity has contributed directly to the OWASP Agentic Security Initiative's broader threat catalog, which documents a wider set of agent risk categories than the top ten format can hold, including persistence and harvesting patterns that don't map cleanly onto the public list.

NIST AI Risk Management Framework

The NIST AI Risk Management Framework places adversarial testing inside its govern-map-measure-manage structure and defines red teaming as adversarial testing under stress conditions to surface failure modes. It's governance-centric rather than technique-centric: useful for structuring a program, less useful as a hands-on testing checklist.

The practical takeaway: no framework in wide circulation was purpose-built to separate security risk from safety risk. Programs that rely on a single list, however authoritative, will end up testing safety by default, simply because that's what the format makes easiest to enumerate.

The AI Red Teaming Workflow

A structured workflow keeps testing systematic rather than ad hoc. The five phases below hold for manual and automated red teaming alike, though automating each one gets substantially more complex once security, not safety, is the target.

Phase 1: Reconnaissance and scoping

Map the system before testing it: the model in use, every tool integration, every connected data source, the trust boundaries between user input and system instructions, and any existing defenses already in place, including classifiers and rate limiting.

Phase 2: Attack generation

Generate adversarial test cases mapped to the relevant risk categories. Automated tooling can produce volume; human red teamers add the business-logic-specific scenarios that require actual knowledge of the target application.

Phase 3: Execution and testing

Run the test cases and record results systematically. Because AI systems behave probabilistically, a single run rarely tells the full story; each category needs enough repetitions to establish a reliable success rate.

Phase 4: Validation and impact analysis

Not every successful exploit is a meaningful risk, and not every failed attempt means the system is defended. Validation asks what an attacker would realistically need to execute the attack in production, and what the plausible business impact is if they succeed.

Phase 5: Mitigation and re-test

Prioritize findings by exploitability and impact, remediate, then re-test to confirm the fix holds and hasn't opened a new gap. For agentic systems operating in environments that change constantly, this cycle should run continuously rather than as a periodic event.

The State of Automated AI Red Teaming Today

Ask most security teams what tool they use for AI red teaming and the answer is an open source project: NVIDIA's Garak, Microsoft's PyRIT, or Promptfoo. All three are useful, and all three were built to test safety, not security.

That's not a criticism of the tools; it's a reflection of what's actually feasible to automate. Safety testing looks roughly the same for every agent: no agent should describe how to commit a crime, regardless of what it does or who built it. That uniformity is what makes it possible to ship a repository of adversarial prompts and call it done. Security testing doesn't work that way. Every agent has a different set of tools, connected data, and failure modes, which means automated security red teaming needs a bigger model of the system under test, not a bigger prompt library.

Consider what it actually takes to automate a test for indirect prompt injection leading to data exfiltration. The tool needs to connect the target agent to a real inbox, generate a payload delivered through that inbox, trigger the agent to process it, and confirm whether the resulting action reached an attacker-controlled endpoint. Getting that pipeline working, before writing a single adversarial payload, is a materially harder engineering problem than sending a prompt and grading the response. That's a meaningful reason security testing has lagged: the infrastructure it requires is harder to build and harder to open source.

There's a useful workaround that doesn't require a live indirect injection to prove a point. Whatever an agent can be told to do directly, it can also be manipulated into doing maliciously. If a direct prompt can get an agent to render an image using data pulled from a parameter, that alone demonstrates a zero-click exfiltration path exists. Security teams evaluating an agent's exposure don't need to wait for a fully automated indirect-injection harness to start finding real risk; they need to ask what the agent can do, then assume an attacker will eventually find a way to make it do that on command.

Where AI Red Teaming Is Headed

The gap between safety-focused and security-focused automated red teaming won't stay this wide. As agentic AI becomes the default rather than the exception, the tooling gets forced to catch up, because safety-only testing will visibly fail to catch the incidents that actually cost organizations money and trust.

The next few years should bring automated red teaming tools that are themselves agentic: systems with enough context about a target agent's tools, permissions, and data access to reason about where its security boundaries actually sit, rather than firing a static list of prompts and grading whether any of them landed. That shift changes what a red teaming report looks like. Instead of a pass/fail count against a prompt library, a mature security-focused red teaming agent should identify that a target has a lethal trifecta, generate an exploit chain specific to that agent's actual tools and data, and explain the business impact in terms a CISO can act on immediately, including quarantining an agent that's demonstrated real capability for harm.

Building that kind of testing agent is a different scale of problem than maintaining a prompt repository, which is also why it hasn't shown up as an open source project the way safety testing has. It requires deep context about the target environment and a substantial amount of accumulated expertise about how agents actually fail in production. As that expertise gets encoded into automated tooling over the next couple of years, expect the center of gravity in AI red teaming to shift decisively from safety to security, simply because that's where the uncovered risk has been sitting all along.

FAQs About AI Red Teaming

What's the difference between AI safety testing and AI security testing?

Safety testing checks whether a model produces harmful, biased, or policy-violating content, leading to a reputational risk. Security testing checks whether an agent can be manipulated into taking an action that compromises data, infrastructure, or identity, leading to an operational risk. Most automated red teaming tools today test safety almost exclusively.

What is the lethal trifecta in AI agent security?

A term coined by security researcher Simon Willison for the combination of conditions that makes an agent capable of data exfiltration: access to untrusted data, access to sensitive information, and the ability to take an outbound action. An agent with all three can leak data using nothing but its own legitimate access.

What is goal misalignment, and why does it matter more than prompt injection?

Goal misalignment is when an agent pursues an outcome other than the one it was given, whether an attacker manipulated it or it drifted there on its own inside a long or corrupted context. Prompt injection is one route to goal misalignment. Testing only for injection misses the cases where no attacker was involved at all.

Are Garak, PyRIT, and Promptfoo enough for AI red teaming?

They're strong for safety testing at scale, and Promptfoo has begun extending into security-adjacent categories. None was built to test tool misuse, data exfiltration, or identity abuse in a way that reflects a specific agent's actual tools and data access.

What frameworks should an AI red teaming program be built around?

MITRE ATLAS for threat modeling and technique vocabulary, the OWASP Top 10 for Agentic Applications for a baseline risk taxonomy, and the NIST AI RMF for governance structure. Treat all three as a starting point rather than a complete checklist, since none were built specifically to separate safety risk from security risk.

Should an organization prioritize safety or security in its AI red teaming program?

It depends on where the liability sits. Customer-facing agents that an organization builds and owns carry real reputational risk, so safety testing matters there too. Internal agents built on third-party platforms like Copilot Studio or ChatGPT shift reputational liability to the vendor, but data and access liability stays with the deploying organization, which is why security testing should be the priority for most enterprise agent deployments today.

What will automated AI red teaming look like in the next few years?

Expect a shift from static prompt libraries toward agentic red teaming tools that build context about a target agent's tools, data access, and permissions, then reason about where its actual security boundaries sit.