Beyond Superficial Unlearning: Sharpness-Aware Robust Erasure of Hallucinations in Multimodal LLMs

TL;DR

This work identifies a robustness gap in multimodal LLM hallucination unlearning: standard methods push models into sharp loss basins where hallucinations can resurface under weight shifts. It introduces SARE, a two-stage framework combining automated data curation with Targeted-SAM to achieve robust, geometry-aware erasure by flattening the loss landscape around hallucinated concepts. Through extensive experiments on multiple MLLMs and diverse evaluation metrics, SARE achieves significantly stronger erasure, preserves general generation and reasoning capabilities, and resists relearning, LoRA fine-tuning, and adversarial prompting. The approach offers a practical, data-efficient path to trustworthy multimodal generation with durable, perturbation-resilient hallucination suppression.

Abstract

Multimodal LLMs are powerful but prone to object hallucinations, which describe non-existent entities and harm reliability. While recent unlearning methods attempt to mitigate this, we identify a critical flaw: structural fragility. We empirically demonstrate that standard erasure achieves only superficial suppression, trapping the model in sharp minima where hallucinations catastrophically resurge after lightweight relearning. To ensure geometric stability, we propose SARE, which casts unlearning as a targeted min-max optimization problem and uses a Targeted-SAM mechanism to explicitly flatten the loss landscape around hallucinated concepts. By suppressing hallucinations under simulated worst-case parameter perturbations, our framework ensures robust removal stable against weight shifts. Extensive experiments demonstrate that SARE significantly outperforms baselines in erasure efficacy while preserving general generation quality. Crucially, it maintains persistent hallucination suppression against relearning and parameter updates, validating the effectiveness of geometric stabilization.

Figures (7)

• Figure 1: The vulnerability of unlearned MLLMs against relearning attacks. (a) Lightweight relearning can easily reactivate suppressed hallucinations. (b) The hallucination rate of EFUF exhibits a rapid resurgence as the number of relearning samples increases.
• Figure 2: Overview of the SARE framework. The top-right panel illustrates Stage 1, where an automated pipeline curates training subsets ($D_{neg}, D_{pos}, D_{sent}$). The bottom panel depicts Stage 2, contrasting the fragile sharp minima of standard unlearning (left) with the robust flat loss landscape of SARE (right).
• Figure 3: Assessment of General Capabilities on GQA, SQA, QBench, and MME. SARE effectively maintains foundational reasoning and comprehension.
• Figure 4: Training dynamics of SARE. Rapid convergence is achieved at Epoch 1, while further training leads to grounding collapse.
• Figure 5: Efficiency comparison. SARE achieves significant speedup over DPO and NPO with competitive latency relative to EFUF.