Resilience by Design: Architecting Agentic Systems to Fail Safely, Not Catastrophically

Agentic AI fails differently than traditional software. When a service crashes, alerts trigger, processes restart, and humans intervene. In contrast, an agent encountering a problem continues operating, reasoning past faults, improvising around missing inputs, and proceeding with actions. In these systems, failure redirects execution rather than stopping it. Many AI roadmaps have not yet fully considered this design challenge.

In my thirty years building security programs, I've seen one principle repeatedly: resilience must be built in from the start, not added after a breach. Implementing resilience is an architectural decision made before you write code or buy tools. Agentic AI heightens this need, as we now deploy systems authorized to act (move money, change configurations, contact customers, or revoke access) without human confirmation. Granting such authority without designing for safe failure means delegating judgment to a system that lacks it, and leaving the organization without accountability.

I wrote this article to present resilience from two perspectives. For executives and boards, it addresses consequences and key questions to consider before AI deployment. For CISOs and security architects, it focuses on mechanisms: the controls that make "fail safe, not catastrophic" actionable in design and implementation. I have also included a checklist to go with this article to make it actionable for those tasked with agentic AI governance and security.

The Infinite Game, Applied to Agents

The Infinite Game framework I have referenced in previous published articles on the internet is directly relevant here. Cybersecurity is best viewed lens of an infinite game mindset, where the game has no end and no designated winner. Instead it's a disciplined approach to play a game where the rules continually change. To be effective in this type of game requires a focus on the basics like security hygiene. With regard to agentic resilience, security hygiene is not a one-time effort to eliminate AI risk; it’s an ongoing discipline maintained through every model update, integration, and permission expansion. Agentic AI does not change this requirement; it increases the consequences of neglecting it.

A finite mindset deploys the agent, measures adoption, and considers the task complete. An infinite mindset continually asks: when the agent is wrong, what’s the potential impact, and who defined the boundaries to contain it?

Catastrophic Failure Is a Design Choice

Catastrophic failure in agentic systems rarely stems from a single dramatic error. Instead, it results from a chain of plausible but incorrect outputs, each accepted as ground truth and compounding through workflows lacking checkpoints.

For example, an agent may generate a fictitious vendor account number and proceed to act on it. A downstream agent then trusts this output and initiates payment. By the time a human intervenes, multiple systems have already acted on false information. These failures occur quietly and sequentially, escalating a model error into a board-level incident.

The root cause is that agentic systems don’t follow the failure model assumed by most tools. Traditional services fail by stopping, while agents fail by continuing. When an agent receives ambiguous or adversarial input, it may confidently reach an incorrect conclusion and execute it, since it has tool access without human oversight. The fusion of control and reasoning planes creates this risk, and effective controls are needed to separate them.

Therefore, resilience by design must be tailored specifically for agentic systems, rather than relying on standard application security checklists.

The Four Planes You Actually Control

Architecture cannot prevent an LLM from hallucinating. Still, you can control its environment, including what it can access, when it must pause for review, what forensic records it creates, and how it responds to uncertainty. These are the four planes of control.

1. Bounded Authority: capability scoping for non-human actors

Limit an agent's permissions to what it needs for its specific task, rather than assigning broad roles for convenience. Treat each agent as a distinct non-human identity with its own credentials, rather than as a process under a service account with excessive permissions. This approach prevents privilege aggregation, where an agent with broad access becomes a significant risk if its reasoning fails.

• Per-agent identity and short-lived credentials. Scope tokens to the specific tools and data an agent needs for its declared task, issue them just-in-time, and expire them quickly. No standing broad access grants.
• Capability-based tool access. Gate each tool call through an authorization layer the agent cannot rewrite. The agent requests an action; a separate policy engine decides whether that action, with those parameters, is permitted right now.
• Parameter-level constraints. Implement parameter-level constraints, not just tool-level controls. For example, allowing an agent to “call the payments API” is insufficient. Restricting it to payments below a certain threshold and only to approved counterparties provides stronger oversight.
• Separate read agents from write agents. When possible, divide retrieval and analysis from action-taking, ensuring the component processing untrusted content does not also possess write credentials.

2. Circuit Breakers: human gating at irreversible boundaries

A circuit breaker is a defined condition that halts autonomous execution and waits for confirmation. The key consideration is placement, determined by consequence and confidence. Classify each agent action by reversibility and potential impact and require human or independent verification for actions that cannot be reversed.

Use a clear decision model: for each action, assess its reversibility and consequence. Allow automatic execution for reversible, low-consequence actions, but require confirmation for irreversible or high-consequence actions, regardless of the agent's confidence. Self-reported confidence is not a reliable safety indicator.

if action.reversible and action.consequence == LOW:
     execute(action)
 elif action.consequence in (HIGH, IRREVERSIBLE):
     require_human_confirmation(action, context, reasoning_trace)
 else:
     require_independent_verification(action)  # second agent / rules check

• Classify actions as irreversible by default. Payments, access changes, data deletion, external communications, production configuration changes, and contract actions should all require gating unless explicitly justified otherwise.
• Implement rate and anomaly limiters. Even permitted actions require velocity caps. For example, an agent initiating hundreds of refunds per minute signals a failure, even if each action meets policy.
• Adopt a fail-closed approach. If the policy engine, confirmation channel, or verification step is unavailable, the system should halt rather than proceed without supervision.

3. Observable Reasoning: provenance you can replay

When an agentic incident occurs, ordinary logs can answer "what happened," but rarely explain "why it happened." This distinction is critical for postmortem reviews, regulatory inquiries, and remediation. Agent observability must capture decision provenance, not just side effects, to avoid reconstructing intent from incomplete data.

• Maintain full decision traces. Log the agent's inputs, retrieved context, intermediate plans, tool calls, and authorization outcomes, linking them as a single causal chain per task rather than dispersing them across multiple log instances.
• Ensure audit trails are tamper-evident and inaccessible. The agent should not be able to read, summarize, or modify audit records. Treat these logs as independent evidence.
• Ensure decision paths are replayable. The system should allow reconstruction of whether the same input would produce a similar action, which is essential for effective remediation. Since agents are deterministic, output is never the same, but you should be able to reconstruct the decision path with similar results.
• Track provenances for all retrieved content. In RAG and tool-augmented workflows, record the source of each piece of context. Without proper tagging, tracing issues from poisoned or spoofed sources becomes impossible.

4. Graceful Degradation: designed fallbacks for ambiguity

Many agentic deployments assume the agent will always find a viable solution. Resilient design, however, plans for uncertainty. Define system behavior when the agent is uncertain, blocked, or beyond its competence, and engineer fallbacks rather than improvise them.

• Establish explicit states for low confidence and ambiguity. Route such cases to a human, a deterministic rules path, or a safe no-op, focusing on defined outcomes rather than relying on the model to improvise. Low confidence states are typically flagged by agents set by human review and established boundaries.
• Bounded autonomy on retries. Set strict limits on retry logic. Without hard ceilings, agents may repeatedly re-plan in response to persistent failures, leading to excessive tool calls and escalating costs and risks.
• Degrade to a less capable, more predictable path. Design systems to degrade to less capable but more predictable workflows. When failures occur, fallback to deterministic processes is preferable to broader autonomous improvisation, which can produce unpredictable results.

The Threats These Planes Actually Answer

Align these controls with adversary tactics; otherwise, they remain theoretical. The primary agentic threat classes identified in OWASP's agentic and LLM security guidance, NIST's AI 100-2 E2025, and MITRE ATLAS correspond directly to the four planes.

Resilient agentic systems aren't the ones that never fail. They're the ones engineered so the failure hits a boundary instead of a blast radius.

What Boards Aren’t Yet Asking

Most boards I advise continue to focus on adoption questions such as speed, scope, and return on investment. Few are considering containment: what occurs when an autonomous system acts on incorrect information at machine speed?

This sequence is misaligned. Rapid adoption without corresponding investment in failure architecture often leads organizations to explain incidents to regulators after they occur, rather than demonstrating controls to auditors in advance. The four planes provide boards with specific questions:

• What can this agent access?
• Where must it pause for approval?
• Can we trace its decision-making?
• How does it respond to uncertainty?

None of This Is New, and That's the Point

These controls are not unique to AI. They reflect long standing principles such as capability-based security, least privilege, defense in depth, and fail-safe defaults. Additionally, safety-critical engineering emphasizes that components will fail and must do so safely. Fields like aviation, industrial control, and medical-device engineering have practiced this for decades. Resilient systems are engineered to fail safely, not to avoid failure entirely.

The CISO's mandate is evolving, regardless of organizational structure. While data security was the primary focus for two decades, the current priority is governing autonomous decision-makers. This shift represents a new discipline, more aligned with industrial safety engineering than traditional cybersecurity. Security leaders must be proficient in both technology and business to ensure these principles are integral to agentic design reviews before deployment, rather than added in response to incidents.

Sustained, Not Solved

The Infinite Game mindset is relevant at the technical level as well. Agentic resilience is an ongoing process, not a completed task. Each new tool integration expands the attack surface, every model update alters failure patterns, and each additional permission introduces new boundaries to manage.

Fail-safe design is not a limitation on agentic AI's value; it is essential for effective large-scale deployment. Organizations that prioritize safe failure can confidently deploy agents in critical workflows, knowing containment is validated. Those that neglect this step may limit adoption due to risk tolerance or face significant consequences after an uncontained incident.

Agentic resilience benefits organizations that remain operational after inevitable failures. Implement the four planes proactively, while boundaries can still be intentionally designed and failure remains a manageable choice rather than a public issue.

To aid you in implementing your agentic resilience initiatives I have condensed the four above controls into a checklist for you to use for your resilience and business continuity programs. Remember resilience is not about never having another breach or incident, it's about having a process in place for when you do have your next incident the blast radius is limited and the impact to the business is properly managed. Good luck!