Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Harmonious Parameter Adaptation in Continual Visual Instruction Tuning for Safety-Aligned MLLMs

Ziqi Wang, Chang Che, Qi Wang, Hui Ma, Zenglin Shi, Cees G. M. Snoek, Meng Wang

TL;DR

The paper tackles the problem of continual visual instruction tuning for safety-aligned multimodal LLMs, where post-safety-alignment CVIT risks both task forgetting and safety degradation. It introduces Harmonious Parameter Adaptation (HPA), a post-training framework with focusing-based parameter partition, harmoniously balanced parameter selection, and orthogonal parameter adjustment to preserve safety while learning new tasks. Key contributions include Hessian-based parameter importance for focus scoring, adaptive per-layer retention strategies, and an orthogonal update mechanism to mitigate forgetting, validated on CVIT and safety benchmarks where HPA outperforms baselines in both task performance and safety preservation. The work offers a practical and scalable approach for maintaining safety guarantees during continual adaptation in real-world MLLMs.

Abstract

While continual visual instruction tuning (CVIT) has shown promise in adapting multimodal large language models (MLLMs), existing studies predominantly focus on models without safety alignment. This critical oversight ignores the fact that real-world MLLMs inherently require such mechanisms to mitigate potential risks. In this work, we shift our focus to CVIT for safety-aligned MLLMs and observe that during continual adaptation, the model not only suffers from task forgetting but also exhibits degradation in its safety. Achieving a harmonious balance between safety and task performance remains a crucial challenge. To address this, we propose Harmonious Parameter Adaptation (HPA), a post-training framework composed of focusing-based parameter partition, harmoniously balanced parameter selection, and orthogonal parameter adjustment. Specifically, HPA partitions parameters into two types based on their focus on safety or task performance, and selects the focused ones to preserve from a balanced perspective. In addition, HPA imposes orthogonality constraints on parameter updates to further alleviate catastrophic forgetting. Extensive experiments on the CVIT benchmark and safety evaluation datasets demonstrate that HPA better maintains high safety and mitigates forgetting than existing baselines.

Harmonious Parameter Adaptation in Continual Visual Instruction Tuning for Safety-Aligned MLLMs

TL;DR

The paper tackles the problem of continual visual instruction tuning for safety-aligned multimodal LLMs, where post-safety-alignment CVIT risks both task forgetting and safety degradation. It introduces Harmonious Parameter Adaptation (HPA), a post-training framework with focusing-based parameter partition, harmoniously balanced parameter selection, and orthogonal parameter adjustment to preserve safety while learning new tasks. Key contributions include Hessian-based parameter importance for focus scoring, adaptive per-layer retention strategies, and an orthogonal update mechanism to mitigate forgetting, validated on CVIT and safety benchmarks where HPA outperforms baselines in both task performance and safety preservation. The work offers a practical and scalable approach for maintaining safety guarantees during continual adaptation in real-world MLLMs.

Abstract

While continual visual instruction tuning (CVIT) has shown promise in adapting multimodal large language models (MLLMs), existing studies predominantly focus on models without safety alignment. This critical oversight ignores the fact that real-world MLLMs inherently require such mechanisms to mitigate potential risks. In this work, we shift our focus to CVIT for safety-aligned MLLMs and observe that during continual adaptation, the model not only suffers from task forgetting but also exhibits degradation in its safety. Achieving a harmonious balance between safety and task performance remains a crucial challenge. To address this, we propose Harmonious Parameter Adaptation (HPA), a post-training framework composed of focusing-based parameter partition, harmoniously balanced parameter selection, and orthogonal parameter adjustment. Specifically, HPA partitions parameters into two types based on their focus on safety or task performance, and selects the focused ones to preserve from a balanced perspective. In addition, HPA imposes orthogonality constraints on parameter updates to further alleviate catastrophic forgetting. Extensive experiments on the CVIT benchmark and safety evaluation datasets demonstrate that HPA better maintains high safety and mitigates forgetting than existing baselines.

Paper Structure

This paper contains 15 sections, 14 equations, 8 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Problem Formulation
  5. Overview of HPA Framework
  6. Focusing-Based Parameter Partition
  7. Harmoniously Balanced Parameter Selection
  8. Orthogonal Parameter Adjustment
  9. Experiments
  10. Benchmark Setup
  11. Experimental Setup
  12. Main Results
  13. Ablation Study
  14. Further Analysis
  15. Conclusion

Figures (8)

  • Figure 1: Unsafety Variation (on the safety benchmark) and Accuracy Variation (on the first tuning dataset AD) under Post-SA CVIT for different methods. The black dashed line indicates that the MLLM remains unsafe throughout Pre-SA CVIT due to missing safety alignment.
  • Figure 2: Overview of HPA Framework. HPA consists of three components: (1) Focusing-based parameter partition which partitions parameters into safety- and task-focused types; (2) Harmoniously balanced parameter selection which balances the selection of focused parameters; (3) Orthogonal parameter adjustment which applies orthogonal constraints during parameter updates to mitigate forgetting.
  • Figure 3: Effect of different parameter retention rates $p$ on performance and safety.
  • Figure 4: Effect of fixed versus adaptive coefficient $\alpha$ on performance and safety.
  • Figure 5: Layer-wise proportion of safety-focused parameters located not in shared-focused positions among all selected safety-focused parameters.
  • ...and 3 more figures