Visual-Guided Key-Token Regularization for Multimodal Large Language Model Unlearning

TL;DR

This work tackles privacy-preserving unlearning in multimodal LLMs by addressing token-level importance and visual cues. It proposes Visual-Guided Key-Token Regularization (ViKeR), which uses irrelevant reference images to estimate ideal token distributions and applies KL-based regularization to guide unlearning toward forgetting key tokens while preserving normal tokens. The approach yields better forgetting, retention, and coherence on MLLMU and CLEAR benchmarks, with theoretical analysis of token-level gradient reweighting clarifying why key tokens are prioritized. Practically, ViKeR enables more reliable unlearning in vision-language models without sacrificing general language capabilities or response quality.

Abstract

Unlearning in Multimodal Large Language Models (MLLMs) prevents the model from revealing private information when queried about target images. Existing MLLM unlearning methods largely adopt approaches developed for LLMs. They treat all answer tokens uniformly, disregarding their varying importance in the unlearning process. Moreover, these methods focus exclusively on the language modality, disregarding visual cues that indicate key tokens in answers. In this paper, after formulating the problem of unlearning in multimodal question answering for MLLMs, we propose Visual-Guided Key-Token Regularization (ViKeR). We leverage irrelevant visual inputs to predict ideal post-unlearning token-level distributions and use these distributions to regularize the unlearning process, thereby prioritizing key tokens. Further, we define key tokens in unlearning via information entropy and discuss ViKeR's effectiveness through token-level gradient reweighting, which amplifies updates on key tokens. Experiments on MLLMU and CLEAR benchmarks demonstrate that our method effectively performs unlearning while mitigating forgetting and maintaining response coherence.

Figures (14)

• Figure 1: Problem formulation for unlearning. Given the forget set (i.e., target visual--question--answer triples to be unlearned), the MLLM is expected to forget the targeted content while preserving other knowledge and generating coherent responses.
• Figure 2: Probability distributions of the predicted answer tokens for a visual--question pair to be unlearned. a) Output of the full MLLM (i.e., LLM before unlearning). Gray indicates normal tokens, while red indicates key tokens. b) Output of MLLM after unlearning with GA, where normal tokens are forgotten instead of key tokens. c) Averaged output of the full MLLM, using irrelevant reference images as inputs, which approaches the ideal distribution for unlearning, and can be considered as a good reference for the unlearning process.
• Figure 3: ViKeR Pipeline. The left half shows GA, while the right depicts modifications of ViKeR. ViKeR has two stages: (1) visual-guided token distribution estimation (in blue), where irrelevant visual inputs are fed into the pre-unlearning gradient-free full model to estimate ideal token distributions, and (2) token-level regularization (in orange), using the predicted distribution to regularize GA.
• Figure 4: Visualization results of token distribution after unlearning with different methods. Compared to other methods, ViKeR preserves the prediction of normal tokens (such as '_person') while successfully unlearning key tokens (such as '_civil').
• Figure 5: Hyperparameter analysis of $\lambda$ and $k$.

Theorems & Definitions (8)

• Definition 1
• Proposition 1
• Definition 2
• Proposition 2
• Proposition 3
• proof
• proof
• proof