Agentic Moderation: Multi-Agent Design for Safer Vision-Language Models

TL;DR

This work tackles safety for large vision-language systems by addressing the limitations of static and post-hoc defenses against cross-modal jailbreaks. It introduces Agentic Moderation, a model-agnostic, multi-agent framework that coordinates Shield, Responder, Evaluator, and Reflector to provide context-aware and interpretable moderation. Across five datasets and four LVLMs, the approach reduces attack success rate while maintaining or improving safety and utility metrics, demonstrating robust, scalable governance for multimodal safety. The modular design supports flexible policy updates and deployment in latency-constrained or high-stakes environments, outlining a path toward adaptive, agent-driven safety in multimodal AI systems.

Abstract

Agentic methods have emerged as a powerful and autonomous paradigm that enhances reasoning, collaboration, and adaptive control, enabling systems to coordinate and independently solve complex tasks. We extend this paradigm to safety alignment by introducing Agentic Moderation, a model-agnostic framework that leverages specialised agents to defend multimodal systems against jailbreak attacks. Unlike prior approaches that apply as a static layer over inputs or outputs and provide only binary classifications (safe or unsafe), our method integrates dynamic, cooperative agents, including Shield, Responder, Evaluator, and Reflector, to achieve context-aware and interpretable moderation. Extensive experiments across five datasets and four representative Large Vision-Language Models (LVLMs) demonstrate that our approach reduces the Attack Success Rate (ASR) by 7-19%, maintains a stable Non-Following Rate (NF), and improves the Refusal Rate (RR) by 4-20%, achieving robust, interpretable, and well-balanced safety performance. By harnessing the flexibility and reasoning capacity of agentic architectures, Agentic Moderation provides modular, scalable, and fine-grained safety enforcement, highlighting the broader potential of agentic systems as a foundation for automated safety governance.

Figures (6)

• Figure 1: Overview of the agentic moderation framwork. The system integrates multiple modular agents, including Shield, Responder, Reflector, or other modules to enforce flexible, policy-driven safety checks and task-specific moderation. Various LVLMs and tools are supported, enabling extensible and compositional implementation.
• Figure 2: SHIELD: The Shield Agent classifies multimodal inputs into predefined policy categories and assigns moderation actions (block, reframe, or forward) along with contextual “should do” and “should not do” guidance for safe response generation.
• Figure 3: Evaluator: The ThreatsEval signature used for multimodal safety evaluation of user requests and model responses.
• Figure 4: Reflector: Identifies harmful or policy-violating responses and provides corrective feedback that informs safe regeneration.
• Figure 5: Results across models. Each point denotes a configuration; black-edged dots ($\bullet$) mark baselines, and circled dots ($\circ$) indicate setups achieving the best trade-off between low Attack Success Rate (ASR) and Non-Following Rate (NF). Lower-left points represent safer and more compliant behavior.)