Same Content, Different Answers: Cross-Modal Inconsistency in MLLMs

TL;DR

The paper investigates cross-modal inconsistency in multimodal large language models by introducing REST and REST+ benchmarks, designed to measure reasoning across identical content presented as text, image, or mixed formats under OCR-controlled conditions. It systematically evaluates 15 frontier MLLMs and reveals substantial modality-dependent inconsistencies, with the text modality often yielding the best results even when OCR is correct. The authors also analyze internal multimodal representations and find that alignment between modalities correlates with higher cross-modal consistency, suggesting a mechanistic link between representation similarity and reasoning reliability. REST/REST+ expose gaps in current models and provide a practical framework for benchmarking and guiding improvements in cross-modal reasoning under diverse visual conditions.

Abstract

We introduce two new benchmarks REST and REST+(Render-Equivalence Stress Tests) to enable systematic evaluation of cross-modal inconsistency in multimodal large language models (MLLMs). MLLMs are trained to represent vision and language in the same embedding space, yet they cannot perform the same tasks in both modalities. Our benchmarks contain samples with the same semantic information in three modalities (image, text, mixed) and we show that state-of-the-art MLLMs cannot consistently reason over these different modalities. We evaluate 15 MLLMs and find that the degree of modality inconsistency varies substantially, even when accounting for problems with text recognition (OCR). Neither rendering text as image nor rendering an image as text solves the inconsistency. Even if OCR is correct, we find that visual characteristics (text colour and resolution, but not font) and the number of vision tokens have an impact on model performance. Finally, we find that our consistency score correlates with the modality gap between text and images, highlighting a mechanistic interpretation of cross-modal inconsistent MLLMs.

Figures (14)

• Figure 1: Summary of our work. Left: Our REST benchmark measures whether MLLMs can consistently reason over identical information across modalities. We first verify text recognition (OCR) capability, then evaluate the same question in three modalities (text, image, mixed). Cross-modal inconsistency occurs when models produce different answers depending on the input format. Center: RER consistency score measures the degree to which a model outputs the same answer in all modalities. We evaluate 15 MLLMs (on OCR correct, sorted on REST) and find that the degree of cross-modal inconsistency varies substantially across models even when controlling for OCR. Right: Matching samples (i.e., different modalities containing the same information) show higher cosine similarity than non-matching ones, and the extent of this difference correlates with the consistency score on our benchmark.
• Figure 2: Cross-modal inconsistency leaves model potential untapped. This figure shows the cumulative distribution of correctly solved questions across sets of modalities (OCR-correct subset). From left to right, the bars represent: the percentage of questions that can be solved in all three modalities, followed by including questions that can only be solved in fewer modalities, ending with the Max Modal Coverage (green), which shows the percentage of questions that can be solved in at least one modality. Results are shown for GSM8k-Symbolic, which contains open-ended questions that cannot be solved by guessing.
• Figure 3: Models generally achieve higher text accuracy despite using fewer text tokens. Current MLLMs need more vision tokens than text tokens to achieve the same accuracy, except for Qwen2.5-VL-32B, where fewer vision tokens obtain higher accuracy (OCR-correct subset).
• Figure 4: Colored text makes models perform better. Relative improvements from either red or yellow text compared to black (OCR-correct subset).
• Figure 5: Samples from our 3-type Imagenet categories dataset for evaluating the intermediate representations of MLLMs.