The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audio

TL;DR

This work systematically investigates hallucinations in large multimodal models across language, visual, and audio modalities. It identifies two main causes—unimodal priors overreliance and spurious inter-modality correlations—and introduces the Curse of Multi-modality (CMM) benchmark to diagnose them via object- and event-level probes with PA and HR metrics. The study benchmarks multiple LMMs, revealing persistent vulnerabilities, especially for audio-language and cross-modal interactions, and highlights the need for balanced cross-modal learning and improved mitigation strategies. The findings offer a concrete diagnostic framework and practical directions to enhance the reliability of multimodal systems in real-world applications.

Abstract

Recent advancements in large multimodal models (LMMs) have significantly enhanced performance across diverse tasks, with ongoing efforts to further integrate additional modalities such as video and audio. However, most existing LMMs remain vulnerable to hallucinations, the discrepancy between the factual multimodal input and the generated textual output, which has limited their applicability in various real-world scenarios. This paper presents the first systematic investigation of hallucinations in LMMs involving the three most common modalities: language, visual, and audio. Our study reveals two key contributors to hallucinations: overreliance on unimodal priors and spurious inter-modality correlations. To address these challenges, we introduce the benchmark The Curse of Multi-Modalities (CMM), which comprehensively evaluates hallucinations in LMMs, providing a detailed analysis of their underlying issues. Our findings highlight key vulnerabilities, including imbalances in modality integration and biases from training data, underscoring the need for balanced cross-modal learning and enhanced hallucination mitigation strategies. Based on our observations and findings, we suggest potential research directions that could enhance the reliability of LMMs.

Figures (4)

• Figure 1: Demonstrations of overreliance on unimodal priors.
• Figure 2: Validation experiments on overreliance on unimodal priors.
• Figure 3: Validation experiments on spurious inter-modality correlations caused by co-occurrences.
• Figure 4: Statistics of object and event frequencies in our probing questions.