How Does Vision-Language Adaptation Impact the Safety of Vision Language Models?

TL;DR

This study examines how VL adaptation influences safety and evaluates the impact of safety fine-tuning methods, and suggests the weight merging approach as an optimal solution effectively reducing safety degradation while maintaining helpfulness.

Abstract

Vision-Language adaptation (VL adaptation) transforms Large Language Models (LLMs) into Large Vision-Language Models (LVLMs) for multimodal tasks, but this process often compromises the inherent safety capabilities embedded in the original LLMs. Despite potential harmfulness due to weakened safety measures, in-depth analysis on the effects of VL adaptation on safety remains under-explored. This study examines how VL adaptation influences safety and evaluates the impact of safety fine-tuning methods. Our analysis reveals that safety degradation occurs during VL adaptation, even when the training data is safe. While safety tuning techniques like supervised fine-tuning with safety datasets or reinforcement learning from human feedback mitigate some risks, they still lead to safety degradation and a reduction in helpfulness due to over-rejection issues. Further analysis of internal model weights suggests that VL adaptation may impact certain safety-related layers, potentially lowering overall safety levels. Additionally, our findings demonstrate that the objectives of VL adaptation and safety tuning are divergent, which often results in their simultaneous application being suboptimal. To address this, we suggest the weight merging approach as an optimal solution effectively reducing safety degradation while maintaining helpfulness. These insights help guide the development of more reliable and secure LVLMs for real-world applications.

Figures (9)

• Figure 1: Responses according to each safety type. In text-only safety (Left) and multimodal safety (Mid), VL adaptation causes the LVLM to produce harmful responses. In exaggerated safety (Right), safety tuning leads the LVLM to refuse to answer even harmless questions.
• Figure 2: Performance dynamics of LLaMA-2-Chat-VL on safety benchmarks during VL adaptation. The text-only safety benchmark (blue) and multimodal safety benchmark (red) use Attack Success Rate (solid line) as a metric, while the exaggerated safety benchmark (green) uses Refusal Rate (dotted line) as a metric. Lower values are better for both metrics.
• Figure 3: Performance dynamics of LLaMA-2-Chat-MTL on safety benchmarks when VL adaptation and safety tuning are applied simultaneously. The text-only safety benchmark (blue) and multimodal safety benchmark (red) use Attack Success Rate (solid line) as a metric, while the exaggerated safety benchmark (green) uses Refusal Rate (dotted line) as a metric. Lower values are better for both metrics.
• Figure 4: Cosine similarities between corresponding layers of LLaMA-2 Chat 7B and its VL-adapted counterpart, LLaMA-2-Chat-VL, at early (blue), mid (red), and late (green) stages of VL adaptation. The shaded region highlights the safety layers (layers 6 to 14) identified by li2024safety.
• Figure 5: Cosine similarities between corresponding layers of LLaMA-2-Chat-VL and its fine-tuned counterparts, LLaMA-2-Chat-VL-Chatty and LLaMA-2-Chat-VL-SL. The blue line represents the similarity between LLaMA-2-Chat-VL and LLaMA-2-Chat-VL-Chatty, and the red line shows the similarity between LLaMA-2-Chat-VL and LLaMA-2-Chat-VL-SL.