Evaluating Cross-Modal Reasoning Ability and Problem Characteristics with Multimodal Item Response Theory

TL;DR

A multi-modal and multidimensional item response theory framework (M3IRT) that extends classical IRT by decomposing both model ability and item difficulty into image-only, text-only, and cross-modal components, enabling compact, high-quality subsets that better reflect multimodal reasoning.

Abstract

Multimodal Large Language Models (MLLMs) have recently emerged as general architectures capable of reasoning over diverse modalities. Benchmarks for MLLMs should measure their ability for cross-modal integration. However, current benchmarks are filled with shortcut questions, which can be solved using only a single modality, thereby yielding unreliable rankings. For example, in vision-language cases, we can find the correct answer without either the image or the text. These low-quality questions unnecessarily increase the size and computational requirements of benchmarks. We introduce a multi-modal and multidimensional item response theory framework (M3IRT) that extends classical IRT by decomposing both model ability and item difficulty into image-only, text-only, and cross-modal components. M3IRT estimates cross-modal ability of MLLMs and each question's cross-modal difficulty, enabling compact, high-quality subsets that better reflect multimodal reasoning. Across 24 VLMs on three benchmarks, M3IRT prioritizes genuinely cross-modal questions over shortcuts and preserves ranking fidelity even when 50% of items are artificially generated low-quality questions, thereby reducing evaluation cost while improving reliability. M3IRT thus offers a practical tool for assessing cross-modal reasoning and refining multimodal benchmarks.

Figures (20)

• Figure 1: Questions with the highest or lowest cross-modal difficulty $b_{j}^{\rm cross}$ detected by M3IRT. Questions with high cross-modal difficulty require both modalities to find the correct answer. However, those with low difficulty allow us to solve using only the image or text.
• Figure 2: M2IRT investigates the modality-specific and cross-modal difficulties of questions that enables to contract a tailored, compact, and high-quality subset for evaluating a new MLLM.
• Figure 3: Distributions of $\theta$ estimated by M3IRT sorted in descending order.
• Figure 4: The average and standard deviation of Spearman's rank correlations between model rankings on the original benchmark and those estimated on extracted question subsets with different sizes.
• Figure 5: The average and standard deviation of the proportions of the low-quality questions in extracted question subsets $\gamma$ with different sizes.