Uncovering Limitations of Large Language Models in Information Seeking from Tables

TL;DR

TabIS delivers a reliable benchmark for evaluating Large Language Models on table information seeking by converting TTG data into single-choice QA with high-quality distractors. It introduces three progressively challenging subsets (B-TIS, SU-TIS, M-TIS) and a robust option-generation pipeline (Modify-Input, Modify-Output, Exam-Judge) to ensure difficult, faithful evaluations. Across 12 LLMs, GPT-4-turbo achieves the strongest performance but still lags behind on complex structural tasks and robustness to pseudo-relevant tables, revealing core limitations in table understanding and retrieval-augmented resilience. The work provides extensive data, code, and analysis to guide future research on improving LLMs’ TIS capabilities and reliability.

Abstract

Tables are recognized for their high information density and widespread usage, serving as essential sources of information. Seeking information from tables (TIS) is a crucial capability for Large Language Models (LLMs), serving as the foundation of knowledge-based Q&A systems. However, this field presently suffers from an absence of thorough and reliable evaluation. This paper introduces a more reliable benchmark for Table Information Seeking (TabIS). To avoid the unreliable evaluation caused by text similarity-based metrics, TabIS adopts a single-choice question format (with two options per question) instead of a text generation format. We establish an effective pipeline for generating options, ensuring their difficulty and quality. Experiments conducted on 12 LLMs reveal that while the performance of GPT-4-turbo is marginally satisfactory, both other proprietary and open-source models perform inadequately. Further analysis shows that LLMs exhibit a poor understanding of table structures, and struggle to balance between TIS performance and robustness against pseudo-relevant tables (common in retrieval-augmented systems). These findings uncover the limitations and potential challenges of LLMs in seeking information from tables. We release our data and code to facilitate further research in this field.

Figures (15)

• Figure 1: Above: A simplified table-to-text generation example illustrating the unreliable evaluation issue. Higher values on surface-level metrics like BLEU and ROUGE do not guarantee better results. Target cells are highlighted. Below: Our benchmark presented in a single-choice format.
• Figure 2: Simplified Examples of B-TIS subset, SU-TIS subset, and M-TIS subset. For each B-TIS sample, we generate one SU-TIS sample and one M-TIS sample with some modifications.
• Figure 3: Model performance in different option generation strategies. Averaged over 12 LLMs.
• Figure 4: Averaged accuracy and TSU variation score for 12 models, tested and averaged on 6 TSU tasks. Model names are simplified for illustration.
• Figure 5: TIS Robustness against pseudo-relevant tables and averaged accuracy for 12 models, tested and averaged on ToTTo and HiTab. Model names are simplified for illustration.