OmnixR: Evaluating Omni-modality Language Models on Reasoning across Modalities

TL;DR

OmnixR presents a unique evaluation towards assessing OLMs over a diverse mix of modalities, such as a question that involves video, audio, and text, providing a rigorous cross-modal reasoning testbed unlike any existing benchmarks.

Abstract

We introduce OmnixR, an evaluation suite designed to benchmark SoTA Omni-modality Language Models, such as GPT-4o and Gemini. Evaluating OLMs, which integrate multiple modalities such as text, vision, and audio, presents unique challenges. Particularly, the user message might often consist of multiple modalities, such that OLMs have to establish holistic understanding and reasoning across modalities to accomplish the task. Existing benchmarks are limited to single modality or dual-modality tasks, overlooking comprehensive multi-modal assessments of model reasoning. To address this, OmnixR offers two evaluation variants: (1)synthetic subset: a synthetic dataset generated automatically by translating text into multiple modalities--audio, images, video, and hybrids (Omnify). (2)realistic subset: a real-world dataset, manually curated and annotated by experts, for evaluating cross-modal reasoning in natural settings. OmnixR presents a unique evaluation towards assessing OLMs over a diverse mix of modalities, such as a question that involves video, audio, and text, providing a rigorous cross-modal reasoning testbed unlike any existing benchmarks. Our experiments find that all state-of-the-art OLMs struggle with OmnixR questions that require integrating information from multiple modalities to answer. Further analysis highlights differences in reasoning behavior, underscoring the challenges of omni-modal AI alignment.

Figures (6)

• Figure 1: Reasoning Behavior of a OLM Varies across Modalities. Taking Gemini-1.5-Flash as an example, on text question, the reasoning behaviour is expected and the answer is correct. When the same question is rendered to an image, the model generate a reasonable reasoning but incorrect answer. On the video or audio representation of the same question, the model generates no reasoning and produces incorrect answers.
• Figure 2: The overview of Omni$\times$R$_\text{synth}$ and Omni$\times$R$_\text{real}$.
• Figure 3: We propose Omnify! to create the synthetic omni-modality evaluation data from the original text benchmarks.
• Figure 4: Visualization of Examples in the Omni$\times$R-Real set.
• Figure 5: We include two figures to illustrate which is a better format image. The upper one is the image with better format. The lower one is the image with the original format.