Unlearning Sensitive Information in Multimodal LLMs: Benchmark and Attack-Defense Evaluation

TL;DR

This work addresses the risk of sensitive information retention in multimodal LLMs by proposing UnLOK-VQA, a benchmark for targeted multimodal unlearning, and an attack–defense framework. It combines LoRA-based model editing with a rigorous evaluation on efficacy, generalization, and specificity using rephrase and neighborhood data derived from OK-VQA. The study shows multimodal extraction attacks are more effective than unimodal ones, and that the strongest defense (Head Projection/Max-Entropy) significantly mitigates leakage, with larger models demonstrating greater post-editing robustness. The results underscore the importance of targeted knowledge deletion in MLLMs and provide a practical benchmark and insights to guide future defense strategies and scaling considerations.

Abstract

LLMs trained on massive datasets may inadvertently acquire sensitive information such as personal details and potentially harmful content. This risk is further heightened in multimodal LLMs as they integrate information from multiple modalities (image and text). Adversaries can exploit this knowledge through multimodal prompts to extract sensitive details. Evaluating how effectively MLLMs can forget such information (targeted unlearning) necessitates the creation of high-quality, well-annotated image-text pairs. While prior work on unlearning has focused on text, multimodal unlearning remains underexplored. To address this gap, we first introduce a multimodal unlearning benchmark, UnLOK-VQA (Unlearning Outside Knowledge VQA), as well as an attack-and-defense framework to evaluate methods for deleting specific multimodal knowledge from MLLMs. We extend a visual question-answering dataset using an automated pipeline that generates varying-proximity samples for testing generalization and specificity, followed by manual filtering for maintaining high quality. We then evaluate six defense objectives against seven attacks (four whitebox, three blackbox), including a novel whitebox method leveraging interpretability of hidden states. Our results show multimodal attacks outperform text- or image-only ones, and that the most effective defense removes answer information from internal model states. Additionally, larger models exhibit greater post-editing robustness, suggesting that scale enhances safety. UnLOK-VQA provides a rigorous benchmark for advancing unlearning in MLLMs.

Figures (6)

• Figure 1: Illustration of (1) information leakage in MLLMs, and (2, 3) the attack-defense framework. This demonstrates that while defense methods can mitigate information leakage from MLLMs, malicious adversaries may still extract sensitive information from them.
• Figure 2: Pipeline for UnLOK-VQA generation: (1) We utilize the OK-VQA dataset as a basis for evaluating the efficacy of editing methods in removing knowledge from MLLMs; (2) We employ multiple techniques to produce rephrase data with different levels, which we use in blackbox attacks to assess the generalizability of the unlearning methods; (3) We create various levels of neighborhood data to check whether the editing methods target the intended information without altering the outputs of neighboring data.
• Figure 3: Effect of scaling the LLaVA-v1.5's size from 7B to 13B on attack success of HP attack (whitebox) and Multimodal Rephrase Attack (blackbox) against the Fact Erasure defense. We find that scaling makes the models more robust against the attacks.
• Figure 4: Distribution of question categories in UnLOK-VQA. It consists of samples belonging to diverse categories and covers all the categories in the original OK-VQA dataset.
• Figure 5: Average distance of the random, neighborhood image and rephrase image points from the original data point. Neighborhood points are closer to the target data point being deleted compared to random points on which other unlearning datasets evaluate specificity. Rephrase points are closer compared to both neighborhood and random data points. This is also reflected by higher Image Neighborhood $\Delta$-Acc compared to Random $\Delta$-Acc .