Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Thinking with Tables: Enhancing Multi-Modal Tabular Understanding via Neuro-Symbolic Reasoning

Kun-Yang Yu, Zhi Zhou, Shi-Yu Tian, Xiao-Wen Yang, Zi-Yi Jia, Ming Yang, Zi-Jian Cheng, Lan-Zhe Guo, Yu-Feng Li

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated remarkable reasoning capabilities across modalities such as images and text. However, tabular data, despite being a critical real-world modality, remains relatively underexplored in multimodal learning. In this paper, we focus on the task of Tabular-Vision Multi-Modal Understanding (TVMU) and identify three core challenges: (1) high structural variability and data incompleteness in tables, (2) implicit and complex feature dependencies, and (3) significant heterogeneity in problem-solving pipelines across downstream tasks. To address these issues, we propose Thinking with Tables (TWT). TWT employs a program-aided code-based neuro-symbolic reasoning mechanism that facilitates key operations, such as information extraction and element modeling, by interacting with external environments. We evaluate TWT on eight representative datasets. Experimental results demonstrate that TWT consistently outperforms existing baselines by an average of 10\% in accuracy, achieving performance comparable to, or even surpassing, proprietary commercial SOTA LLMs on TVMU tasks. Models and codes are available at https://github.com/kunyang-YU/Thinking-with-Tables

Thinking with Tables: Enhancing Multi-Modal Tabular Understanding via Neuro-Symbolic Reasoning

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated remarkable reasoning capabilities across modalities such as images and text. However, tabular data, despite being a critical real-world modality, remains relatively underexplored in multimodal learning. In this paper, we focus on the task of Tabular-Vision Multi-Modal Understanding (TVMU) and identify three core challenges: (1) high structural variability and data incompleteness in tables, (2) implicit and complex feature dependencies, and (3) significant heterogeneity in problem-solving pipelines across downstream tasks. To address these issues, we propose Thinking with Tables (TWT). TWT employs a program-aided code-based neuro-symbolic reasoning mechanism that facilitates key operations, such as information extraction and element modeling, by interacting with external environments. We evaluate TWT on eight representative datasets. Experimental results demonstrate that TWT consistently outperforms existing baselines by an average of 10\% in accuracy, achieving performance comparable to, or even surpassing, proprietary commercial SOTA LLMs on TVMU tasks. Models and codes are available at https://github.com/kunyang-YU/Thinking-with-Tables
Paper Structure (21 sections, 9 equations, 6 figures, 3 tables)

This paper contains 21 sections, 9 equations, 6 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Multi-Modal Tabular Understanding
  4. Neuro-Symbolic Reasoning
  5. Problem and Analysis
  6. Problem Definition
  7. Problem Analysis
  8. Methods
  9. Environment Setup
  10. Task-Oriented Supervised Fine-Tuning
  11. Adaptive Loss-Scaled Reinforcement Learning
  12. Experiments
  13. Experimental Setup
  14. Main Results
  15. Further Analysis
  16. ...and 6 more sections

Figures (6)

  • Figure 1: Performance on TAT-QA dataset with different situations. FT means a full table message is provided.
  • Figure 2: Predictions from different models. Yellow highlights indicate insufficient feature modeling by existing methods, while red highlights show that our model performs more comprehensive feature modeling through code-based interactions.
  • Figure 3: TWT framework. It realizes robust and good performance via code-base neuro-symbolic reasoning in TVMU tasks.
  • Figure 4: Comparison of code execution accuracy of TWT w and w/o AL-GRPO
  • Figure 5: < image >
  • ...and 1 more figures