IsoBench: Benchmarking Multimodal Foundation Models on Isomorphic Representations

TL;DR

IsoBench introduces a broad, isomorphism-based benchmark to probe how input representation—image versus text—affects reasoning in multimodal foundation models across math, science, algorithms, and chess. The study finds a consistent textual bias across leading models, even when problem content is semantically isomorphic across representations. To address this gap, the authors propose IsoCombination and IsoScratchPad prompting strategies that fuse representations or translate visuals to text, yielding notable performance gains in several tasks. The work highlights substantial modality limitations in current vision-language integrations and offers practical calibration techniques with potential to inform future multimodal design and evaluation. Overall, IsoBench provides a fine-grained, data-centric lens on how input form shapes multimodal reasoning capabilities and opens directions for more robust, representation-agnostic systems.

Abstract

Current foundation models exhibit impressive capabilities when prompted either with text only or with both image and text inputs. But do their capabilities change depending on the input modality? In this work, we propose $\textbf{IsoBench}$, a benchmark dataset containing problems from four major areas: math, science, algorithms, and games. Each example is presented with multiple $\textbf{isomorphic representations}$ of inputs, such as visual, textual, and mathematical presentations. IsoBench provides fine-grained feedback to diagnose performance gaps caused by the form of the representation. Across various foundation models, we observe that on the same problem, models have a consistent preference towards textual representations. Most prominently, when evaluated on all IsoBench problems, Claude-3 Opus performs 28.7 points worse when provided with images instead of text; similarly, GPT-4 Turbo is 18.7 points worse and Gemini Pro is 14.9 points worse. Finally, we present two prompting techniques, $\textit{IsoCombination}$ and $\textit{IsoScratchPad}$, which improve model performance by considering combinations of, and translations between, different input representations.

Figures (17)

• Figure 1: Do multimodal foundation models treat every modality equally? In this example, a model is provided with either an image representation or a text representation isomorphic to the image, where the instructions are kept identical. Surprisingly, multimodal models often give different responses for these isomorphic inputs (e.g., in the figure above, only the response to the text representation is correct). In IsoBench, we scale such examples into four domains (Math, Science, Algorithms, Games) and find a consistent preference towards text across many popular multimodal foundation models.
• Figure 2: IsoBench contains four major domains: Mathematical Functions, Science Questions, Graph Algorithms, and Chess Games. For each domain, there are two or three subtasks. All examples within IsoBench are provided with one image representation and several textual representations that are isomorphic to each other.
• Figure 3: Illustration of IsoCombination (IsoCB) and IsoScratchPad (IsoSP). IsoCB combines all representations provided by a user and constructs one unified prompt for a foundation model. IsoSP is a two-step prompting method, where a foundation model first describes an image and then uses the textual description as the sole representation for a given task.
• Figure 4: Distribution of science problem categories. We introduced three new categories that are absent from the ScienceQA dataset: Electric Circuit, Organic Compound, and Chemical equations.
• Figure 5: Sample response from GPT-4 to a parity problem in IsoBench. GPT-4 was able to analyze the parity of the rational function with the correct reasoning and computation.