Breaking Agent Backbones: Evaluating the Security of Backbone LLMs in AI Agents

TL;DR

The paper addresses how the choice of backbone LLM affects security in AI agents. It introduces threat snapshots, a formal framework that isolates LLM vulnerabilities at specific agent states, and builds the b^3 benchmark from crowdsourced attacks to evaluate 31 backbone LLMs. The findings show that enhanced reasoning improves security while model size does not reliably predict security, and closed-weight systems tend to be more secure, offering actionable guidance for backbone selection. By treating security as a first-class evaluation dimension alongside capability benchmarks, the work provides a practical path for practitioners and researchers to strengthen agent security.

Abstract

AI agents powered by large language models (LLMs) are being deployed at scale, yet we lack a systematic understanding of how the choice of backbone LLM affects agent security. The non-deterministic sequential nature of AI agents complicates security modeling, while the integration of traditional software with AI components entangles novel LLM vulnerabilities with conventional security risks. Existing frameworks only partially address these challenges as they either capture specific vulnerabilities only or require modeling of complete agents. To address these limitations, we introduce threat snapshots: a framework that isolates specific states in an agent's execution flow where LLM vulnerabilities manifest, enabling the systematic identification and categorization of security risks that propagate from the LLM to the agent level. We apply this framework to construct the $\operatorname{b}^3$ benchmark, a security benchmark based on 194331 unique crowdsourced adversarial attacks. We then evaluate 31 popular LLMs with it, revealing, among other insights, that enhanced reasoning capabilities improve security, while model size does not correlate with security. We release our benchmark, dataset, and evaluation code to facilitate widespread adoption by LLM providers and practitioners, offering guidance for agent developers and incentivizing model developers to prioritize backbone security improvements.

Figures (11)

• Figure 1: (left) Illustration of how inputs flow within an AI agent, alternating between an LLM step that calls the backend LLM $m$ with the current model context and a processing step that calls the processing function $f_{\operatorname{proc}}$ until the final response is produced. (right) The $\operatorname{b}^3$ benchmark, which uses threat snapshots to isolate an LLM step from the context-output flow on the left. (right top) There are 30 threat snapshots in total based on 10 application with three levels $\operatorname{L1}$, $\operatorname{L2}$ and $\operatorname{L3}$. (right bottom) Each threat snapshot is evaluated against the set of attacks where we evaluate each attack $N$ times which is used to account for the variance in responses.
• Figure 2: (top left) Model ranking is roughly preserved across key slices of threat snapshots, with some models standing out on tasks involving content safety and tool use. We plot models that perform the best or the worst in at least one category. (bottom left) LLMs with reasoning enabled have lower total vulnerability scores (lower is better). (right) Ranking based on total vulnerability scores for all models -- lower score is better.
• Figure 3: Threat snapshots provide an abstraction to fully describe an instance of the LLM vulnerability.
• Figure 4: Overall ranking are not heavily influenced by the method used to select attacks. We plot the Spearman's rho rank correlation between the selected attack dataset and other choices in the benchmark construction. The box plot on the left shows Spearman's rho for random rankings.
• Figure 5: Vulnerability scores for differently sized models of the same families. There is no clear trend indicating that large models are more secure.

Theorems & Definitions (3)

• Example 2.1: AI Coding Assistant
• Definition 2.2: LLM vulnerability
• Example 2.3: AI Coding Assistant -- continued