Texts or Images? A Fine-grained Analysis on the Effectiveness of Input Representations and Models for Table Question Answering

TL;DR

The paper addresses how input representations (table images vs. textual tables) and model types (LLMs vs. MLLMs) influence table question answering under controlled conditions. It introduces a 1600-instance benchmark across four settings defined by question complexity and table size, sourced from six datasets, to systematically compare model-representation combinations. Across seven open-weight MLLMs with decoupled LLM decoders, the study finds that large models favor image-based inputs, while smaller models exhibit varied optimal configurations; a new method, Fres, dynamically selects representations and achieves about a 10% improvement in exact-match accuracy while reducing input tokens. These results highlight the importance of model size and input modality in multimodal TQA and provide a practical mechanism to balance performance and efficiency in real-world systems.

Abstract

In table question answering (TQA), tables are encoded as either texts or images. Prior work suggests that passing images of tables to multi-modal large language models (MLLMs) performs comparably to or even better than using textual input with large language models (LLMs). However, the lack of controlled setups limits fine-grained distinctions between these approaches. In this paper, we conduct the first controlled study on the effectiveness of several combinations of table representations and models from two perspectives: question complexity and table size. We build a new benchmark based on existing TQA datasets. In a systematic analysis of seven pairs of MLLMs and LLMs, we find that the best combination of table representation and model varies across setups. We propose FRES, a method selecting table representations dynamically, and observe a 10% average performance improvement compared to using both representations indiscriminately.

Figures (6)

• Figure 1: Varying exact match (EM) for models and table representations under different settings (i and t stand for image and text representations of tables). We categorize our investigation into four settings based on table size (small or big) and question complexity (retrieval or reasoning).
• Figure 2: Evaluation of table size robustness. The bar plot shows the number of instances sampled for each bin, and the line plots show the performance of different approaches against varying table sizes.
• Figure 3: Different table templates.
• Figure 4: Resolution distribution of MMTab.
• Figure 5: Evaluation of table size robustness. The bar plot shows the number of instances sampled for each bin and the line plots show the performance of different approaches against varying table sizes.