PULSE: Practical Evaluation Scenarios for Large Multimodal Model Unlearning

TL;DR

This paper addresses the practical evaluation gap for unlearning in large multimodal models by introducing the PULSE protocol, which targets two realistic scenarios: forgetting pre-trained knowledge and sustaining performance under sequential unlearning requests. It formalizes the evaluation with $\, abla$D-sets for unlearned and retained data and tests several unlearning methods (GA, GA+KLR, NPO) on LLaVA-v1.5-13B with LoRA, revealing that forgetting pre-trained knowledge severely harms general capability (MMBench) while forgetting fine-tuned knowledge has a smaller impact. The study also demonstrates that sequential unlearning quickly erodes both unlearning effectiveness and retention, highlighting sustainability challenges. Overall, PULSE provides a practical framework and empirical insights to guide the design of robust unlearning techniques suitable for real-world deployment in multimodal settings.

Abstract

In recent years, unlearning techniques, which are methods for inducing a model to "forget" previously learned information, have attracted attention as a way to address privacy and copyright concerns in large language models (LLMs) and large multimodal models (LMMs). While several unlearning benchmarks have been established for LLMs, a practical evaluation framework for unlearning in LMMs has been less explored. Specifically, existing unlearning benchmark for LMMs considers only scenarios in which the model is required to unlearn fine-tuned knowledge through a single unlearning operation. In this study, we introduce PULSE protocol for realistic unlearning scenarios for LMMs by introducing two critical perspectives: (i) Pre-trained knowledge Unlearning for analyzing the effect across different knowledge acquisition phases and (ii) Long-term Sustainability Evaluation to address sequential requests. We then evaluate existing unlearning methods along these dimensions. Our results reveal that, although some techniques can successfully unlearn knowledge acquired through fine-tuning, they struggle to eliminate information learned during pre-training. Moreover, methods that effectively unlearn a batch of target data in a single operation exhibit substantial performance degradation when the same data are split and unlearned sequentially.

Figures (5)

• Figure 1: Our PULSE Pipelines.
• Figure 2: Dataset Construction of Pre-trained Knowledge Unlearning. We selected individuals with high performance on LLaVA and created dataset of pre-trained knowledge unlearning.
• Figure 3: Example of the Multimodal Task and the Text-only Task. The multimodal task includes person's face image, while the text-only task only has text prompt.
• Figure 4: The Effect of the Source of Unlearning Target. The $\mathcal{D}_{\text{unlearn}}$ axis shows what percentage of the model's pre-unlearning knowledge (set as 100) has been forgotten. For the $\mathcal{D}_{\text{retain}}$ and MMBench axes, it shows what percentage of pre-unlearning knowledge has been retained. All methods exhibit a substantial drop in MMBench score when unlearning pre-trained knowledge.
• Figure 5: The Transition of Accuracy Over Multiple Requests. All methods show a proper decrease in accuracy on $\mathcal{D}_{\text{unlearn}}$ as the number of unlearning requests increases, but at the same time accuracy on $\mathcal{D}_{\text{retain}}$ and MMBench also drops significantly, indicating that these methods fail to handle sequential requests sustainably.