Toward a Dynamic Stackelberg Game-Theoretic Framework for Agentic AI Defense Against LLM Jailbreaking

TL;DR

A game theoretic framework that models the interaction between prompt engineers and large language models (LLMs) as a two player extensive form game coupled with a Rapidly exploring Random Trees (RRT) search over prompt space offers a principled foundation for evaluating, interpreting, and hardening LLM guardrails.

Abstract

This paper proposes a game theoretic framework that models the interaction between prompt engineers and large language models (LLMs) as a two player extensive form game coupled with a Rapidly exploring Random Trees (RRT) search over prompt space. The attacker incrementally samples, extends, and tests prompts, while the LLM chooses to accept, reject, or redirect, leading to terminal outcomes of Safe Interaction, Blocked, or Jailbreak. Embedding RRT exploration inside the extensive form game captures both the discovery phase of jailbreak strategies and the strategic responses of the model. Furthermore, we show that the defender behavior can be interpreted through a local Stackelberg equilibrium condition, which explains when the attacker can no longer obtain profitable prompt deviations and provides a theoretical lens for understanding the effectiveness of our Purple Agent defense. The resulting game tree thus offers a principled foundation for evaluating, interpreting, and hardening LLM guardrails.

Figures (4)

• Figure 1: Conceptual framework of the Purple Agent: "Think Red to Act Blue". The left panel illustrates the Attacker's (Red) objective to explore prompt trajectories (e.g., $P1 \to P4 \text{ or } P3$) leading to jailbreaks. The right panel depicts the Defender's (Blue) goal to intercept these threats via blocking mechanisms. The Purple Agent (bottom) unifies these perspectives by internalizing adversarial search (Thinking Red) to proactively deploy safety boundaries (Acting Blue), thereby neutralizing potential attacks before they materialize.
• Figure 2: Example of the Stackelberg game dynamics. A benign query (A1) leads to a safe state. An adversarial role-play (A2) forces the Defender to choose. A naive "Redirect" strategy, while avoiding immediate failure, may open a path for a contextual follow-up (A2') that leads to a delayed jailbreak, illustrating the need for multi-turn anticipation.
• Figure 3: RRT exploration process and the corresponding extensive-form game tree construction at different stages. The figure shows how RRT sampling expands the prompt space and how the game tree is updated during the search.
• Figure 4: t-SNE visualizations of jailbreak prompts. (a) Attacker-only: Dense clusters indicate unstable regions where Fragile Safety prevails ($\bar{v} \approx 1$). (b) Purple Agent: Sparse, isolated points confirm that anticipatory blocking has neutralized risky neighborhoods, achieving a Robust Local Equilibrium ($\bar{v} \to 0$).

Theorems & Definitions (2)

• Definition 1: Subgame-Perfect Stackelberg Equilibrium (SPSE)
• Definition 2: Local $\varepsilon$-Equilibrium