Does Table Source Matter? Benchmarking and Improving Multimodal Scientific Table Understanding and Reasoning

TL;DR

This paper tackles the challenge of scientific table understanding and numerical reasoning by proposing MMSci, a domain-focused multimodal framework with dynamic input resolutions. It introduces three components—MMSci-Pre (52K table-structure samples), MMSci-Ins (12K reasoning-tuning samples), and MMSci-Eval (3,114 numerical-reasoning test samples)—and demonstrates that domain-specific, high-quality data outperforms larger general-domain datasets. A two-model evaluation using Qwen2-VL-7B-Instruct and LLaVA-NeXT-7B confirms that dynamic-resolution table processing plus explicit reasoning steps yield strong generalization to held-out benchmarks and robust numerical reasoning capabilities. The work highlights the importance of data quality and targeted instruction tuning for scientific multimodal table tasks and provides publicly available code and data to foster reproducibility and further research.

Abstract

Recent large language models (LLMs) have advanced table understanding capabilities but rely on converting tables into text sequences. While multimodal large language models (MLLMs) enable direct visual processing, they face limitations in handling scientific tables due to fixed input image resolutions and insufficient numerical reasoning capabilities. We present a comprehensive framework for multimodal scientific table understanding and reasoning with dynamic input image resolutions. Our framework consists of three key components: (1) MMSci-Pre, a domain-specific table structure learning dataset of 52K scientific table structure recognition samples, (2) MMSci-Ins, an instruction tuning dataset with 12K samples across three table-based tasks, and (3) MMSci-Eval, a benchmark with 3,114 testing samples specifically designed to evaluate numerical reasoning capabilities. Extensive experiments demonstrate that our domain-specific approach with 52K scientific table images achieves superior performance compared to 150K general-domain tables, highlighting the importance of data quality over quantity. Our proposed table-based MLLMs with dynamic input resolutions show significant improvements in both general table understanding and numerical reasoning capabilities, with strong generalisation to held-out datasets. Our code and data are publicly available at https://github.com/Bernard-Yang/MMSci_Table.

Figures (7)

• Figure 1: Overview of the proposed framework, which consists of four key stages: (1) Table Image Generation; (2) MMSci Dataset Construction; (3) Table Structure Learning; and (4) Visual Instruction Tuning.
• Figure 2: Performance scaling with increasing instruction tuning data size on three MMSci tasks.
• Figure 3: Evaluation of generated data of MMSci-Ins and MMSci-Eval dataset. Correct refers to the data verified correctly by human annotators.
• Figure 4: MMSci-Pre Dataset example
• Figure 5: MMSci-Ins and MMSci-Eval Dataset example