Cross-Modal Safety Mechanism Transfer in Large Vision-Language Models

TL;DR

This work uncovers why safety mechanisms from text LLMs do not automatically transfer to vision in LVLMs, revealing that activation occurs at specific transformer layers and is hindered by weak hidden-state alignment between vision and language. It introduces Cross-Modal Safety Mechanism Transfer and a Text-Guided vision-language Alignment (TGA) method that uses retrieved text to guide hidden-state alignment, enabling the transfer of textual safety to vision without safety fine-tuning on visuals. The proposed approach achieves improved defense against toxic images (higher $DSR$) while maintaining competitive performance on vision tasks, and analyses reveal the importance of aligning hidden-state representations rather than solely outputs. Overall, TGA offers a practical, data-efficient path to safer LVLMs with broader impact for real-world multimodal systems.

Abstract

Vision-language alignment in Large Vision-Language Models (LVLMs) successfully enables LLMs to understand visual input. However, we find that existing vision-language alignment methods fail to transfer the existing safety mechanism for text in LLMs to vision, which leads to vulnerabilities in toxic image. To explore the cause of this problem, we give the insightful explanation of where and how the safety mechanism of LVLMs operates and conduct comparative analysis between text and vision. We find that the hidden states at the specific transformer layers play a crucial role in the successful activation of safety mechanism, while the vision-language alignment at hidden states level in current methods is insufficient. This results in a semantic shift for input images compared to text in hidden states, therefore misleads the safety mechanism. To address this, we propose a novel Text-Guided vision-language Alignment method (TGA) for LVLMs. TGA retrieves the texts related to input vision and uses them to guide the projection of vision into the hidden states space in LLMs. Experiments show that TGA not only successfully transfers the safety mechanism for text in basic LLMs to vision in vision-language alignment for LVLMs without any safety fine-tuning on the visual modality but also maintains the general performance on various vision tasks (Safe and Good).

Figures (14)

• Figure 1: (a): Hidden states at the specific local transformer layers in LVLMs play a crucial role in the successful activation of safety mechanism. (b): Current vision-language alignment methods cannot effectively align vision with its semantics in text modality at hidden states level. (c): Insufficient alignment at hidden states level shifts the semantics of image and misleads the layers for safety.
• Figure 2: Location of safety mechanism activation and attention map on toxic tokens varies with layer in LVLMs. The blue line is the proportion $R$ of attention from token $x$ to the toxic token set $\mathcal{C}$ over the entire attention map varies with layer. The pink region are the layers where safety mechanism is activated. The left and right boundaries of the region are the minimum and maximum activation layer on the entire sample set respectively.
• Figure 3: Defense Success Rate (DSR) for the toxic text input in the condition that information flow from toxic tokens are masked in every $5$ layers. The yellow bars are the original DSR without any truncating. The pink bars are truncating at layers where safety mechanism is activated determined by our method. The blue bars are truncating at other layers.
• Figure 4: Cosine similarity between hidden sates of text and image input with the same semantics varies with layer in LVLMs. The pink region are the layers where safety mechanism is activated on text. The left and right boundaries of the region are the minimum and maximum activation layer on the entire sample set respectively. The purple dashed line is the cosine similarity between the output representations of text-image pairs obtained by CLIP ViT-H/14.
• Figure 5: Defense Success Rate (DSR) for the toxic image input in the condition that hidden states in every $5$ layers are added by mean pooled hidden states of the text. The yellow bars are the original DSR without any operation. The pink bars are adding in layers where safety mechanism is activated determined by our method. The blue bars are adding in other layers.