XModBench: Benchmarking Cross-Modal Capabilities and Consistency in Omni-Language Models

TL;DR

XModBench proposes a tri-modal, modality-balanced benchmark to evaluate cross-modal consistency in omni-modal LLMs across audio, vision, and text. The authors curate 60,828 MCQ pairs spanning five task families and instantiate each item in six modality configurations to diagnose task competence, modality disparity, and directional imbalance, using a three-stage data pipeline and LLM-assisted filtering with human verification. Empirical results show persistent modality disparities, particularly with audio, and clear directional imbalances, indicating current OLLMs lack true modality-invariant reasoning. The benchmark provides a scalable diagnostic framework and will release data and tools to foster progress toward robust cross-modal grounding and reasoning.

Abstract

Omni-modal large language models (OLLMs) aim to unify audio, vision, and text understanding within a single framework. While existing benchmarks primarily evaluate general cross-modal question-answering ability, it remains unclear whether OLLMs achieve modality-invariant reasoning or exhibit modality-specific biases. We introduce XModBench, a large-scale tri-modal benchmark explicitly designed to measure cross-modal consistency. XModBench comprises 60,828 multiple-choice questions spanning five task families and systematically covers all six modality compositions in question-answer pairs, enabling fine-grained diagnosis of an OLLM's modality-invariant reasoning, modality disparity, and directional imbalance. Experiments show that even the strongest model, Gemini 2.5 Pro, (i) struggles with spatial and temporal reasoning, achieving less than 60% accuracy, (ii) reveals persistent modality disparities, with performance dropping substantially when the same semantic content is conveyed through audio rather than text, and (iii) shows systematic directional imbalance, exhibiting lower consistency when vision serves as context compared to text. These findings indicate that current OLLMs remain far from truly modality-invariant reasoning and position XModBench as a fundamental diagnostic tool for evaluating and improving cross-modal competence. All data and evaluation tools will be available at https://xingruiwang.github.io/projects/XModBench/.

Figures (7)

• Figure 1: Overview of XModBench. (a) Instances are built from aligned text–image–audio triplets; (b) instantiated into six modality configurations by permuting context and candidate modalities; (c) spanning five domains with 17 subtasks and 60,828 question–answer pairs; and (d) illustrated with example multiple-choice questions under balanced modality settings.
• Figure 2: Distribution of XModBench's questions across five task families with specific subtasks.
• Figure 3: XModBench task examples. We show sample questions from six subtasks in the benchmark. Each question includes possible contexts from different modalities, and for the vision-context example, the candidates are given in either text or audio.
• Figure 4: Modality disparity across different configurations. Negative scores indicate performance gaps, with the largest disparities observed between audio and text.
• Figure 5: Directional imbalance: accuracy gaps between paired inverse settings among audio, vision and text. Models show clear asymmetries, especially in vision–text and audio–text pairs.