ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoning

TL;DR

ShieldAgent introduces a novel guardrail agent that enforces explicit safety policy compliance for other LLM-based agents through probabilistic, verifiable reasoning. It builds an Action-based Safety Policy Model (ASPM) by extracting verifiable rules from policy documents into action/state predicates and $LTL_f$ rules, optimizes the rule set, and uses a probabilistic framework (Markov Logic Network) to decide action safety with an adjustable threshold. The framework includes a shielding pipeline, tool library, and hybrid memory to enable efficient verification, and it is validated on ShieldAgent-Bench, a large agent-guardrail benchmark across multiple environments and risk categories, where ShieldAgent achieves state-of-the-art accuracy and substantial efficiency gains. The work demonstrates meaningful improvements in guarding LLM agents against agent-based and environment-based attacks, with practical implications for deploying safer autonomous AI in real-world tasks.

Abstract

Autonomous agents powered by foundation models have seen widespread adoption across various real-world applications. However, they remain highly vulnerable to malicious instructions and attacks, which can result in severe consequences such as privacy breaches and financial losses. More critically, existing guardrails for LLMs are not applicable due to the complex and dynamic nature of agents. To tackle these challenges, we propose ShieldAgent, the first guardrail agent designed to enforce explicit safety policy compliance for the action trajectory of other protected agents through logical reasoning. Specifically, ShieldAgent first constructs a safety policy model by extracting verifiable rules from policy documents and structuring them into a set of action-based probabilistic rule circuits. Given the action trajectory of the protected agent, ShieldAgent retrieves relevant rule circuits and generates a shielding plan, leveraging its comprehensive tool library and executable code for formal verification. In addition, given the lack of guardrail benchmarks for agents, we introduce ShieldAgent-Bench, a dataset with 3K safety-related pairs of agent instructions and action trajectories, collected via SOTA attacks across 6 web environments and 7 risk categories. Experiments show that ShieldAgent achieves SOTA on ShieldAgent-Bench and three existing benchmarks, outperforming prior methods by 11.3% on average with a high recall of 90.1%. Additionally, ShieldAgent reduces API queries by 64.7% and inference time by 58.2%, demonstrating its high precision and efficiency in safeguarding agents.

Figures (19)

• Figure 1: Overview of ShieldAgent.(Top) From AI regulations (e.g. EU AI Act) and platform-specific safety policies, ShieldAgent first extracts verifiable rules and iteratively refines them to ensure each rule is accurate, concrete, and atomic. It then clusters these rules and assembles them into an action-based safety policy model, associating actions with their corresponding constraints (with weights learned from real or simulated data). (Bottom) During inference, ShieldAgent retrieves relevant rule circuits w.r.t. the invoked action and performs action verification. By referencing existing workflows from a hybrid memory module, it first generates a step-by-step shielding plan with operations supported by a comprehensive tool library to assign truth values for all predicates, then produces executable code to perform formal verification for actions. Finally, it runs probabilistic inference in the rule circuits to provide a safety label and explanation and reports violated rules.
• Figure 2: Pipeline for curating ShieldAgent-Bench. We adopt the AWM web agent wang2024agent and collect safe trajectories by executing instructions with full policy compliance. For risky trajectories, we attack the agent with two SOTA agent-based and environment-based algorithms and produce unsafe trajectories across seven risk categories.
• Figure 3: Performance comparison of ShieldAgent with rule traverse and GuardAgent baselines on ST-WebAgentBench. We report the individual guardrail accuracy for each risk category.
• Figure 4: The number of rules during each iteration step for GitLab policy. Specifically, the orange bar denotes the number of rules after each verifiability refinement step, and the blue bar denotes the number of rules after each redundancy pruning step.
• Figure 5: The number of predicates during each iteration step for GitLab policy. Specifically, the orange bar denotes the number of predicates after each verifiability refinement step, and the blue bar denotes the number of predicates after each redundancy pruning step.