Format Matters: The Robustness of Multimodal LLMs in Reviewing Evidence from Tables and Charts

TL;DR

This work investigates the cross-format robustness of multimodal LLMs in scientific claim verification by extending two datasets, SciTabAlign and ChartMimic, to SciTabAlign+ and ChartMimic+. It evaluates 12 open-source multimodal LLMs under table-only, chart-only, and combined inputs across four chart types, revealing that models consistently perform better with tables and struggle with charts, even when the data are semantically equivalent. Human evaluators, in contrast, perform robustly across both formats, highlighting a gap in current model capabilities rather than task difficulty. The study provides a clear direction for improving chart understanding and cross-format reasoning in multimodal LLMs to support more reliable AI-assisted scientific review.

Abstract

With the growing number of submitted scientific papers, there is an increasing demand for systems that can assist reviewers in evaluating research claims. Experimental results are a core component of scientific work, often presented in varying formats such as tables or charts. Understanding how robust current multimodal large language models (multimodal LLMs) are at verifying scientific claims across different evidence formats remains an important and underexplored challenge. In this paper, we design and conduct a series of experiments to assess the ability of multimodal LLMs to verify scientific claims using both tables and charts as evidence. To enable this evaluation, we adapt two existing datasets of scientific papers by incorporating annotations and structures necessary for a multimodal claim verification task. Using this adapted dataset, we evaluate 12 multimodal LLMs and find that current models perform better with table-based evidence while struggling with chart-based evidence. We further conduct human evaluations and observe that humans maintain strong performance across both formats, unlike the models. Our analysis also reveals that smaller multimodal LLMs (under 8B) show weak correlation in performance between table-based and chart-based tasks, indicating limited cross-modal generalization. These findings highlight a critical gap in current models' multimodal reasoning capabilities. We suggest that future multimodal LLMs should place greater emphasis on improving chart understanding to better support scientific claim verification.

Figures (3)

• Figure 1: An example of the claim verification task in our experiment, featuring both types of evidence: table and chart formats that represent the same information. It is noted that the original example is from the SciTab dataset. To ensure a fair comparison between table- and chart-evidence formats, we modified claims by replacing references to tables with figures (e.g., "Table 7" → "Figure 7") to make them compatible with chart-format evidence.
• Figure 2: An example of four chart types used in SciTabAlign+.
• Figure 3: Input structure used for zero-shot CoT prompting with combined input from both the chart and the table.