CoReTab: Improving Multimodal Table Understanding with Code-driven Reasoning

TL;DR

CoReTab tackles the lack of multi-step reasoning supervision in multimodal table understanding by introducing a code-driven reasoning framework that pairs natural-language explanations with executable Python code. It builds a large, verified 115K-sample dataset across 11 MTU tasks and trains a Qwen2.5-VL-based model with LoRA adapters in three stages (table recognition pretraining, instruction tuning, and reinforcement learning). The approach yields substantial gains across 17 MMTab benchmarks, improving table question answering, fact verification, and table structure understanding, while providing transparent and verifiable reasoning traces. The framework offers scalable supervision that enhances both accuracy and interpretability, with practical implications for robust multimodal table reasoning in real-world applications.

Abstract

Existing datasets for multimodal table understanding, such as MMTab, primarily provide short factual answers without explicit multi-step reasoning supervision. Models trained on these datasets often generate brief responses that offers insufficient accuracy and limited interpretability into how these models arrive at the final answer. We introduce CoReTab, a code-driven reasoning framework that produces scalable, interpretable, and automatically verifiable annotations by coupling multi-step reasoning with executable Python code. Using the CoReTab framework, we curate a dataset of 115K verified samples averaging 529 tokens per response and fine-tune open-source MLLMs through a three-stage pipeline. We evaluate the resulting model trained on CoReTab across 17 MMTab benchmarks spanning table question answering, fact verification, and table structure understanding. Our model achieves significant gains of +6.2%, +5.7%, and +25.6%, respectively, over MMTab-trained baselines, while producing transparent and verifiable reasoning traces. These results establish CoReTab as a robust and generalizable supervision framework for improving multi-step reasoning in multimodal table understanding.

Figures (22)

• Figure 1: (Left-top) Typical response annotation from MMTab MMTab which provides limited reasoning supervision. (Left-bottom) Average length of responses of three datasets. (Middle) A pair of ⟨table image, question⟩. (Right) Our CoReTab annotation example with step-by-step reasoning and executable Python code.
• Figure 2: Overview of the CoReTab framework. It uses an LLM to generate responses as new annotations with multi-step reasoning and Python code, which can be executed and verified against the ground truth with minimal human review. CoReTab annotations provide a rich and interpretable supervision signal for training MLLMs.
• Figure 3: Our three-stage training pipeline: (1) pretraining on table recognition, (2) instruction tuning on CoReTab, and (3) optionally RL optimization on CoReTab. Separate LoRA adapters are updated in each stage.
• Figure 4: Comparison of responses generated by our CoReTab-Qwen2.5-VL and the Table-Qwen2.5-VL baseline.
• Figure 5: Descriptions of pre-defined functions used for TSU tasks to the (M)LLM.