Knowledge Vector Weakening: Efficient Training-free Unlearning for Large Vision-Language Models

TL;DR

This work tackles the high computational cost of unlearning in large vision–language models by introducing Knowledge Vector Weakening (KVW), a training-free, forward-propagation–only method. KVW identifies forget-related knowledge vectors in MLPs via a Forget Knowledge Accessor and progressively weakens their contributions using a gate $g(\mathcal{A})=\exp(-\gamma\mathcal{A})$, without gradient updates or retraining. Through experiments on LVLM benchmarks MLLMU-Bench and CLEAR, KVW demonstrates a more stable forget–retain trade-off and substantial computational savings compared with gradient-based and LoRA-based unlearning, closely approaching oracle forget performance while preserving retain accuracy. The method relies on a memory-like interpretation of FFNs, leveraging knowledge coefficients and a contrast between forget and retain activations, and shows robust performance across hyperparameters with clear practical advantages for safe and scalable deployment.

Abstract

Large Vision-Language Models (LVLMs) are widely adopted for their strong multimodal capabilities, yet they raise serious concerns such as privacy leakage and harmful content generation. Machine unlearning has emerged as a promising solution for removing the influence of specific data from trained models. However, existing approaches largely rely on gradient-based optimization, incurring substantial computational costs for large-scale LVLMs. To address this limitation, we propose Knowledge Vector Weakening (KVW), a training-free unlearning method that directly intervenes in the full model without gradient computation. KVW identifies knowledge vectors that are activated during the model's output generation on the forget set and progressively weakens their contributions, thereby preventing the model from exploiting undesirable knowledge. Experiments on the MLLMU and CLEAR benchmarks demonstrate that KVW achieves a stable forget-retain trade-off while significantly improving computational efficiency over gradient-based and LoRA-based unlearning methods.

Figures (9)

• Figure 1: Forget accuracy (%) on CLEAR forget05 for GD and NPO under different ranks $r$. Evaluation follows a 2-fold validation protocol with hyperparameters selected on a cross-validation split and tested on a held-out split to avoid overestimation from fixed settings. The figure reveals strong rank sensitivity.
• Figure 2: Knowledge Vector Weakening. In this figure, $x_f$ and $x'_f$ denote the hidden representations of the forget input and the next layer output, respectively; $v_i$, $c_i$, and $k_i$ denote knowledge vectors, their corresponding contribution coefficients, and the associated keys. (a) Before applying KVW, forget-related knowledge vectors contribute heavily to the hidden state of the next layer, forming a dominant pathway that drives output generation. (b) After applying KVW, this dominant pathway is effectively weakened by selectively scaling down forget-related knowledge vectors.
• Figure 3: Comparison of computational cost across unlearning methods in terms of FLOPs, time, and memory usage. FLOPs denote the number of floating-point operations required to process a single batch. Time measures the total wall-clock time for the unlearning process over all batches. Memory indicates the maximum VRAM consumption during unlearning.
• Figure 4: Qualitative results on identity unlearning. Red captions expose key identity information, while orange captions hallucinate incorrect identities. In contrast, green captions produce safe descriptions that avoid revealing identity. This figure shows that KVW prevents the model from relying on identity-related knowledge and leading to non-identifying visual descriptions.
• Figure 5: Sensitivity to $\gamma$. Retain accuracy is reported as the average across Retain, RealFace, and RealWorld. The shaded region indicates the feasible range of $\gamma$ that satisfies effective forgetting under the retain constraint.