MAD: Modality-Adaptive Decoding for Mitigating Cross-Modal Hallucinations in Multimodal Large Language Models

TL;DR

Cross-modal hallucinations in multimodal LLMs arise from improper modality interaction, hindering reliable multimodal reasoning. MAD introduces a training-free, modality-aware decoding framework that first prompts the model to self-assess modality relevance and then uses weighted contrastive decoding with per-modality strengths α_m = γ w_m to fuse modality-specific distributions. By combining four modality configurations through an adaptive weighting scheme, MAD suppresses cross-modal interference while preserving relevant information, with strong empirical gains on CMM and AVHBench across multiple AV-LLMs. The approach demonstrates that explicit, task-aware modality awareness at decoding time enhances robustness and can extend to broader multi-sensor setups beyond audio–video.

Abstract

Multimodal Large Language Models (MLLMs) suffer from cross-modal hallucinations, where one modality inappropriately influences generation about another, leading to fabricated output. This exposes a more fundamental deficiency in modality-interaction control. To address this, we propose Modality-Adaptive Decoding (MAD), a training-free method that adaptively weights modality-specific decoding branches based on task requirements. MAD leverages the model's inherent ability to self-assess modality relevance by querying which modalities are needed for each task. The extracted modality probabilities are then used to adaptively weight contrastive decoding branches, enabling the model to focus on relevant information while suppressing cross-modal interference. Extensive experiments on CMM and AVHBench demonstrate that MAD significantly reduces cross-modal hallucinations across multiple audio-visual language models (7.8\% and 2.0\% improvements for VideoLLaMA2-AV, 8.7\% and 4.7\% improvements for Qwen2.5-Omni). Our approach demonstrates that explicit modality awareness through self-assessment is crucial for robust multimodal reasoning, offering a principled extension to existing contrastive decoding methods. Our code is available at \href{https://github.com/top-yun/MAD}{https://github.com/top-yun/MAD}

Figures (9)

• Figure 1: Cross-modal hallucinations and their mitigation through Modality-Adaptive Decoding (MAD). The base model hallucinates non-existent visual content (red text) and audio events (blue text) when describing audio-visual inputs. MAD eliminates these hallucinations by adaptively suppressing cross-modal interference, producing accurate descriptions grounded in actual content.
• Figure 2: Overall MAD pipeline. Given audio-visual inputs and a question, MAD extracts modality-adaptive weights by querying the model to identify relevant modalities [Step 1]. During generation, MAD fuses contrastive logits computed from four modality configurations using these weights to dynamically emphasize relevant modalities [Step 2]. In this example, despite a hammer being visible in the video, MAD correctly predicts "No" by prioritizing audio content, whereas the baseline predicts "Yes" due to cross-modal interference. This demonstrates MAD's effectiveness in mitigating video-driven audio hallucinations through adaptive, question-aware decoding.
• Figure 3: Distribution of extracted modality weights $(w_v, w_a, w_{av})$ across question types on video. The weights align with intuitive modality dependencies, confirming that MAD correctly identifies the required modalities without supervision.
• Figure 4: Impacts of $\gamma$ in VideoLLaMA2-AV
• Figure 6: Analysis on AVHBench