SafeEraser: Enhancing Safety in Multimodal Large Language Models through Multimodal Machine Unlearning

TL;DR

SafeEraser formalizes safety-focused unlearning for multimodal LLMs and introduces PD Loss to curb over-forgetting while preserving model utility. The benchmark comprises 3,000 images and 28.8K VQA pairs across six harmful categories, organized into forget, retain, and prompt decouple sets. Empirical results show that PD Loss, when combined with existing MU methods, reduces Safe Answer Refusal Rate by 79.5% and maintains forget quality and utility, illustrating a practical path toward safer MLLMs. The work provides a comprehensive framework and metrics for evaluating safety unlearning, with implications for deploying trustworthy multimodal AI systems in real-world settings.

Abstract

As Multimodal Large Language Models (MLLMs) develop, their potential security issues have become increasingly prominent. Machine Unlearning (MU), as an effective strategy for forgetting specific knowledge in training data, has been widely used in privacy protection. However, MU for safety in MLLM has yet to be fully explored. To address this issue, we propose SAFEERASER, a safety unlearning benchmark for MLLMs, consisting of 3,000 images and 28.8K VQA pairs. We comprehensively evaluate unlearning methods from two perspectives: forget quality and model utility. Our findings show that existing MU methods struggle to maintain model performance while implementing the forget operation and often suffer from over-forgetting. Hence, we introduce Prompt Decouple (PD) Loss to alleviate over-forgetting through decouple prompt during unlearning process. To quantitatively measure over-forgetting mitigated by PD Loss, we propose a new metric called Safe Answer Refusal Rate (SARR). Experimental results demonstrate that combining PD Loss with existing unlearning methods can effectively prevent over-forgetting and achieve a decrease of 79.5% in the SARR metric of LLaVA-7B and LLaVA-13B, while maintaining forget quality and model utility. Our code and dataset will be released upon acceptance. Warning: This paper contains examples of harmful language and images, and reader discretion is recommended.

Figures (26)

• Figure 1: The Target of Safety Unlearning in MLLMs.
• Figure 2: Variety of keywords across six categories.
• Figure 3: The construction pipeline of SafeEraser.
• Figure 4: The big picture of SafeEraser. It consists of a forget set and a retain set. Notably, retain set is divided into three levels: concept-level, image-level, and prompt-decoupled structure, designed for the fine-grained removal of unsafe capabilities of MLLMs while maintaining normal performance. Additionally, we evaluate MLLMs with diverse evaluation metrics. Compared to traditional unlearning methods, we propose a novel PD Loss, which successfully ensures the precise forgetting of harmful knowledge while preserving normal behavior.
• Figure 5: Visualization of various metrics across different methods over steps using LLaVA-v1.5-7B.