Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

How well do LLMs reason over tabular data, really?

Cornelius Wolff, Madelon Hulsebos

TL;DR

The paper investigates whether general-purpose LLMs truly reason over tabular data and how to evaluate this reliably. It critiques common free-form metrics and introduces LLM-as-a-judge as a more dependable evaluation method, calibrated against human annotations. By revising the TQA-Bench and adding downscaled, realistic tabular inputs, it reveals a substantial gap in tabular reasoning accuracy and shows sensitivity to missing values, duplicates, and structural variations. The work highlights the need for robust evaluation methods and improved model capabilities to ensure trustworthy, real-world tabular reasoning with LLMs.

Abstract

Large Language Models (LLMs) excel in natural language tasks, but less is known about their reasoning capabilities over tabular data. Prior analyses devise evaluation strategies that poorly reflect an LLM's realistic performance on tabular queries. Moreover, we have a limited understanding of the robustness of LLMs towards realistic variations in tabular inputs. Therefore, we ask: Can general-purpose LLMs reason over tabular data, really?, and focus on two questions 1) are tabular reasoning capabilities of general-purpose LLMs robust to real-world characteristics of tabular inputs, and 2) how can we realistically evaluate an LLM's performance on analytical tabular queries? Building on a recent tabular reasoning benchmark, we first surface shortcomings of its multiple-choice prompt evaluation strategy, as well as commonly used free-form text metrics such as SacreBleu and BERT-score. We show that an LLM-as-a-judge procedure yields more reliable performance insights and unveil a significant deficit in tabular reasoning performance of LLMs. We then extend the tabular inputs reflecting three common characteristics in practice: 1) missing values, 2) duplicate entities, and 3) structural variations. Experiments show that the tabular reasoning capabilities of general-purpose LLMs suffer from these variations, stressing the importance of improving their robustness for realistic tabular inputs.

How well do LLMs reason over tabular data, really?

TL;DR

The paper investigates whether general-purpose LLMs truly reason over tabular data and how to evaluate this reliably. It critiques common free-form metrics and introduces LLM-as-a-judge as a more dependable evaluation method, calibrated against human annotations. By revising the TQA-Bench and adding downscaled, realistic tabular inputs, it reveals a substantial gap in tabular reasoning accuracy and shows sensitivity to missing values, duplicates, and structural variations. The work highlights the need for robust evaluation methods and improved model capabilities to ensure trustworthy, real-world tabular reasoning with LLMs.

Abstract

Large Language Models (LLMs) excel in natural language tasks, but less is known about their reasoning capabilities over tabular data. Prior analyses devise evaluation strategies that poorly reflect an LLM's realistic performance on tabular queries. Moreover, we have a limited understanding of the robustness of LLMs towards realistic variations in tabular inputs. Therefore, we ask: Can general-purpose LLMs reason over tabular data, really?, and focus on two questions 1) are tabular reasoning capabilities of general-purpose LLMs robust to real-world characteristics of tabular inputs, and 2) how can we realistically evaluate an LLM's performance on analytical tabular queries? Building on a recent tabular reasoning benchmark, we first surface shortcomings of its multiple-choice prompt evaluation strategy, as well as commonly used free-form text metrics such as SacreBleu and BERT-score. We show that an LLM-as-a-judge procedure yields more reliable performance insights and unveil a significant deficit in tabular reasoning performance of LLMs. We then extend the tabular inputs reflecting three common characteristics in practice: 1) missing values, 2) duplicate entities, and 3) structural variations. Experiments show that the tabular reasoning capabilities of general-purpose LLMs suffer from these variations, stressing the importance of improving their robustness for realistic tabular inputs.
Paper Structure (33 sections, 3 figures, 6 tables)

This paper contains 33 sections, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Analysis of Tabular Reasoning Capabilities
  4. Evaluation Metrics for Tabular Reasoning
  5. The TQA-Bench and Revisions
  6. TQA-Bench reasoning tasks
  7. Lookup queries
  8. Aggregation queries
  9. Complex calculations
  10. TQA-Bench Revisions
  11. Query Refinements
  12. Tabular Data Downscaling
  13. Towards Reliable Evaluation of Tabular Reasoning Capabilities
  14. LLM-generated answers
  15. Free-form text evaluation metrics
  16. ...and 18 more sections

Figures (3)

  • Figure 1: Distribution of SacreBleu and BERT-scores obtained by comparing LLM-generated answers, from which we know their (in)correctness, against ground-truth answers. To be reliable as a metric, the distributions should be clearly separable, which is not the case for both metrics.
  • Figure 2: Performance of LLMs on calculating the average of columns, across varying table sizes. The accuracy of all models gradually decreases with table size.
  • Figure 3: Performance of LLMs on calculating a subtraction across columns, across varying table sizes. The accuracy significantly drops after 4K input size except for GPT-4o-mini.