Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Text2Analysis: A Benchmark of Table Question Answering with Advanced Data Analysis and Unclear Queries

Xinyi He, Mengyu Zhou, Xinrun Xu, Xiaojun Ma, Rui Ding, Lun Du, Yan Gao, Ran Jia, Xu Chen, Shi Han, Zejian Yuan, Dongmei Zhang

TL;DR

The paper fills a gap in tabular data research by introducing Text2Analysis, a benchmark for advanced analytics tasks and unclear queries that go beyond Text2SQL and TableQA. It provides five annotation methods leveraging LLMs to generate dense, executable NL2Code data across 347 tables, yielding 2249 query-code-result samples. Five state-of-the-art models are evaluated using Executable Code Ratio, pass@1, and regression metrics; results show GPT-4 excels on clear tasks but struggles with complex libraries and unclear queries, underscoring the need for improved field recommendation and parameter handling. This benchmark offers a challenging benchmark to drive future research in automated table analysis and real-world query understanding.

Abstract

Tabular data analysis is crucial in various fields, and large language models show promise in this area. However, current research mostly focuses on rudimentary tasks like Text2SQL and TableQA, neglecting advanced analysis like forecasting and chart generation. To address this gap, we developed the Text2Analysis benchmark, incorporating advanced analysis tasks that go beyond the SQL-compatible operations and require more in-depth analysis. We also develop five innovative and effective annotation methods, harnessing the capabilities of large language models to enhance data quality and quantity. Additionally, we include unclear queries that resemble real-world user questions to test how well models can understand and tackle such challenges. Finally, we collect 2249 query-result pairs with 347 tables. We evaluate five state-of-the-art models using three different metrics and the results show that our benchmark presents introduces considerable challenge in the field of tabular data analysis, paving the way for more advanced research opportunities.

Text2Analysis: A Benchmark of Table Question Answering with Advanced Data Analysis and Unclear Queries

TL;DR

The paper fills a gap in tabular data research by introducing Text2Analysis, a benchmark for advanced analytics tasks and unclear queries that go beyond Text2SQL and TableQA. It provides five annotation methods leveraging LLMs to generate dense, executable NL2Code data across 347 tables, yielding 2249 query-code-result samples. Five state-of-the-art models are evaluated using Executable Code Ratio, pass@1, and regression metrics; results show GPT-4 excels on clear tasks but struggles with complex libraries and unclear queries, underscoring the need for improved field recommendation and parameter handling. This benchmark offers a challenging benchmark to drive future research in automated table analysis and real-world query understanding.

Abstract

Tabular data analysis is crucial in various fields, and large language models show promise in this area. However, current research mostly focuses on rudimentary tasks like Text2SQL and TableQA, neglecting advanced analysis like forecasting and chart generation. To address this gap, we developed the Text2Analysis benchmark, incorporating advanced analysis tasks that go beyond the SQL-compatible operations and require more in-depth analysis. We also develop five innovative and effective annotation methods, harnessing the capabilities of large language models to enhance data quality and quantity. Additionally, we include unclear queries that resemble real-world user questions to test how well models can understand and tackle such challenges. Finally, we collect 2249 query-result pairs with 347 tables. We evaluate five state-of-the-art models using three different metrics and the results show that our benchmark presents introduces considerable challenge in the field of tabular data analysis, paving the way for more advanced research opportunities.
Paper Structure (26 sections, 3 equations, 7 figures, 3 tables)

This paper contains 26 sections, 3 equations, 7 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Problem Definition
  3. Analysis Operations and Tasks
  4. Unclear Queries
  5. Text2Analysis Dataset
  6. Data Collection
  7. Forward Annotation:
  8. Reverse Generation from Codes Snippets:
  9. Reverse Generation from Results:
  10. Expansion with New Tables:
  11. Expansion Unclear Queries:
  12. Annotation Quality Assurance
  13. Iterative Annotation and Refinement
  14. Human Evaluation
  15. Data Statistics and Distribution
  16. ...and 11 more sections

Figures (7)

  • Figure 1: Examples of Text2Analysis Benchmark.
  • Figure 2: Advanced Analysis consists of Advanced Operations and Visualizations that are not covered by Rudimentary Operations across descriptive, diagnostic, predictive, and prescriptive analytics.
  • Figure 3: Collection and Generation of (table, query, code, result) Tuples in Text2Analysis.
  • Figure 4: Analysis Task Distribution of All Queries.
  • Figure 5: Task & Parameter Distribution of Unclear Queries.
  • ...and 2 more figures