Omni-Safety under Cross-Modality Conflict: Vulnerabilities, Dynamics Mechanisms and Efficient Alignment

TL;DR

This work reveals a critical cross-modality safety gap in omni-modal LLMs by decoupling modality from semantics with AdvBench-Omni, uncovering a Mid-layer Dissolution of refusal signals and a predominant shrinkage in refusal-vector magnitude as cross-modal inputs unfold. It identifies a modality-invariant pure refusal direction via SVD and introduces OmniSteer, an adaptive, layer-wise steering mechanism using lightweight adapters to apply this direction dynamically. Across three OLLMs and eight datasets, OmniSteer raises the Refusal Success Rate from a challenging baseline to over 91% while preserving Benign Acceptance Rate and overall capabilities, validating its effectiveness and efficiency. The approach provides mechanistic insight into cross-modal safety and offers a practical, plug-in tool for safer omni-modal systems, with potential for integration with training-time defenses in future work.

Abstract

Omni-modal Large Language Models (OLLMs) greatly expand LLMs' multimodal capabilities but also introduce cross-modal safety risks. However, a systematic understanding of vulnerabilities in omni-modal interactions remains lacking. To bridge this gap, we establish a modality-semantics decoupling principle and construct the AdvBench-Omni dataset, which reveals a significant vulnerability in OLLMs. Mechanistic analysis uncovers a Mid-layer Dissolution phenomenon driven by refusal vector magnitude shrinkage, alongside the existence of a modal-invariant pure refusal direction. Inspired by these insights, we extract a golden refusal vector using Singular Value Decomposition and propose OmniSteer, which utilizes lightweight adapters to modulate intervention intensity adaptively. Extensive experiments show that our method not only increases the Refusal Success Rate against harmful inputs from 69.9% to 91.2%, but also effectively preserves the general capabilities across all modalities. Our code is available at: https://github.com/zhrli324/omni-safety-research.

Figures (12)

• Figure 1: The construction pipeline of AdvBench-Omni.
• Figure 2: t-SNE dimensionality reduction analysis of hidden states across different modal inputs. We sampled data from AdvBench-Omni and AdvBench-MM to perform a t-SNE analysis.
• Figure 3: Cosine similarity between hidden states across various modal inputs and the text modality. Experiments were conducted on Qwen2.5-Omni-7B, comparing the similarities of inputs from AdvBench-Omni and AdvBench-MM against the text inputs.
• Figure 4: Safety evaluations on two OLLMs using AdvBench-Omni. We employ RSR to assess the models' refusal capabilities across various modal inputs. Figure \ref{['fig:rsr_qwen']} shows results for Qwen2.5-Omni-7B, and Figure \ref{['fig:rsr_minicpm']} shows results for MiniCPM-o-2.6.
• Figure 5: Layer-wise evolution curves of normalised projection values for inputs across different modalities.