Risk Awareness Injection: Calibrating Vision-Language Models for Safety without Compromising Utility

TL;DR

Risk Awareness Injection (RAI) tackles safety vulnerabilities in vision-language models caused by Risk Signal Dilution during visual-text alignment. It presents a training-free, token-level safety calibration that constructs an Unsafe Prototype Subspace from unsafe keywords, localizes high-risk visual tokens via cosine similarity, and injects risk signals at the model's earliest layer (Layer 0) to boost safety cues without harming cross-modal reasoning. Across image and video benchmarks on multiple architectures, RAI achieves substantial reductions in Attack Success Rate (ASR) while preserving near-original utility in MM-E and MM-Vet tasks, and with modest inference overhead. The approach is lightweight, domain-robust, and easily generalizable, offering a scalable safety guardrail for multimodal systems without retraining.

Abstract

Vision language models (VLMs) extend the reasoning capabilities of large language models (LLMs) to cross-modal settings, yet remain highly vulnerable to multimodal jailbreak attacks. Existing defenses predominantly rely on safety fine-tuning or aggressive token manipulations, incurring substantial training costs or significantly degrading utility. Recent research shows that LLMs inherently recognize unsafe content in text, and the incorporation of visual inputs in VLMs frequently dilutes risk-related signals. Motivated by this, we propose Risk Awareness Injection (RAI), a lightweight and training-free framework for safety calibration that restores LLM-like risk recognition by amplifying unsafe signals in VLMs. Specifically, RAI constructs an Unsafe Prototype Subspace from language embeddings and performs targeted modulation on selected high-risk visual tokens, explicitly activating safety-critical signals within the cross-modal feature space. This modulation restores the model's LLM-like ability to detect unsafe content from visual inputs, while preserving the semantic integrity of original tokens for cross-modal reasoning. Extensive experiments across multiple jailbreak and utility benchmarks demonstrate that RAI substantially reduces attack success rate without compromising task performance.

Figures (10)

• Figure 1: RAI achieves robust defense without compromising utility. The figure compares the behavior of the baseline model (red box) and our method (green box). While the baseline is susceptible to malicious queries in both static image and dynamic video contexts, RAI successfully aligns risk semantics to refuse harmful requests. Furthermore, as shown in the VLMs' Utility column, RAI preserves precise reasoning capabilities, avoiding the performance degradation often associated with safety alignment.
• Figure 2: Micro-level Layer-wise Cosine Similarity Analysis (Qwen2-VL). We track the cosine similarity between visual tokens and the Unsafe Prototype Subspace across transformer layers. The Red Line (Defensive Failure) consistently exhibits lower similarity compared to the Green Line (Defensive Success). This persistent semantic gap indicates that in successful attacks, the visual risk signal is too weak to trigger the model's latent safety mechanisms, providing a rationale for our proposed early injection strategy.
• Figure 3: Impact of Injection Layer and Ratio on Safety and Utility. This figure presents the performance trade-off for Qwen2-VL-7B across layers 0–28. The ASR (MM-SafetyBench) is plotted for injection ratios from 0.01% to 50%, alongside the Perception Score (MME). Effective defense is achieved by modulating only a minimal fraction of high-risk visual tokens (0.01%–1%). Further increasing the injection ratio yields diminishing security returns while degrading perception performance. Deeper interventions progressively reduce the Perception Score, indicating impaired visual understanding.
• Figure 4: Overview of the RAI Framework. The framework consists of two main phases: (1) Risk-Aware Injection: It includes (a) Risk Perception &Sparse Gating, which constructs an Unsafe Prototype Subspace and identifies the most relevant risk categories (e.g., fraud with score 0.08), and (b) Risk Signal Injection, which explicitly injects the selected risk prototype vectors into the visual tokens via a weighted additive operation. (2) Safety Activation: The resulting risk-enhanced visual tokens, along with textual tokens, are fed into the LLM backbone, which activates the safety mechanism (e.g., safe refusal) based on the enriched risk semantics.
• Figure 5: Case 1: Illegal Activity . The model correctly identifies the chemical apparatus in the visual input and refuses to provide instructions for manufacturing dangerous substances.