Single Image Unlearning: Efficient Machine Unlearning in Multimodal Large Language Models

TL;DR

The paper addresses the problem of unlearning specific visual concepts in multimodal LLMs and introduces Single Image Unlearning (SIU), which forgets a concept's visual recognition by fine-tuning a single image with a novel Multifaceted Fine-tuning Data strategy and a Dual Masked KL-divergence Loss. It also benchmarks MU in MLLMs with MMUBench, a dataset and evaluation suite assessing efficacy, generality, specificity, fluency, diversity, and defenses against privacy and misuse attacks. Experimental results demonstrate SIU's superior forgetting performance while preserving utility, and reveal robustness to membership inference and jailbreak attacks. Overall, the work pioneers MU in MLLMs and provides a practical, data-efficient framework for concept unlearning along with a comprehensive benchmark for future research.

Abstract

Machine unlearning empowers individuals with the `right to be forgotten' by removing their private or sensitive information encoded in machine learning models. However, it remains uncertain whether MU can be effectively applied to Multimodal Large Language Models (MLLMs), particularly in scenarios of forgetting the leaked visual data of concepts. To overcome the challenge, we propose an efficient method, Single Image Unlearning (SIU), to unlearn the visual recognition of a concept by fine-tuning a single associated image for few steps. SIU consists of two key aspects: (i) Constructing Multifaceted fine-tuning data. We introduce four targets, based on which we construct fine-tuning data for the concepts to be forgotten; (ii) Jointly training loss. To synchronously forget the visual recognition of concepts and preserve the utility of MLLMs, we fine-tune MLLMs through a novel Dual Masked KL-divergence Loss combined with Cross Entropy loss. Alongside our method, we establish MMUBench, a new benchmark for MU in MLLMs and introduce a collection of metrics for its evaluation. Experimental results on MMUBench show that SIU completely surpasses the performance of existing methods. Furthermore, we surprisingly find that SIU can avoid invasive membership inference attacks and jailbreak attacks. To the best of our knowledge, we are the first to explore MU in MLLMs. We will release the code and benchmark in the near future.

Figures (24)

• Figure 1: Overview of the Unlearning Process in MLLMs Using SIU. The process starts with a user request to unlearn the visual recognition of concepts, utilizing MMUBench (introduced in Section \ref{['sec:mmu']}) to provide concepts for unlearning. SIU has two elements which are Multifaceted Fine-tuning Data and Dual Masked KL-divergence Loss. After unlearning, the unlearned MLLM is evaluated for generality, specificity, diversity, fluency, and resistance to membership inference and jailbreak attacks.
• Figure 2: Visualization of various metrics across different methods over steps using LLAVA7B.
• Figure 3: Visualization of various metrics across different methods over steps using LLAVA13B.
• Figure 4: EM performance comparison of methods SIU, GA+KL, PO, and GA across different concepts.
• Figure 5: The output distribution of LLAVA when queried about the visual recognition of unseen concepts.