Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HCT-QA: A Benchmark for Question Answering on Human-Centric Tables

Mohammad S. Ahmad, Zan A. Naeem, Michaël Aupetit, Ahmed Elmagarmid, Mohamed Eltabakh, Xiasong Ma, Mourad Ouzzani, Chaoyi Ruan

TL;DR

HCT-QA introduces a large benchmark for question answering on human centric tables embedded in documents, addressing the challenge of complex HCT layouts by evaluating a broad spectrum of LLMs and VLMs. The authors construct real world and synthetic HCT datasets with thousands of QA pairs and ground truth answers, and provide a canonical CSV representation to standardize extraction. Through extensive experiments, they show that while large closed weight models achieve the best recall around 0.68, there is substantial room for improvement, especially on nested structures and aggregations; HTML input formats and semantic quality of table names significantly impact performance. Fine tuning with real and synthetic data yields dramatic gains, underscoring the value of their synthetic generator and the potential for pretraining on HCTs. The work demonstrates both the current limitations and the practical viability of LLM/VLM based QA on HCTs, and points to future directions in pretraining, cross table QA, and improved table extraction that can unlock the value of large document repositories.

Abstract

Tabular data embedded within PDF files, web pages, and other document formats are prevalent across numerous sectors such as government, engineering, science, and business. These human-centric tables (HCTs) possess a unique combination of high business value, intricate layouts, limited operational power at scale, and sometimes serve as the only data source for critical insights. However, their complexity poses significant challenges to traditional data extraction, processing, and querying methods. While current solutions focus on transforming these tables into relational formats for SQL queries, they fall short in handling the diverse and complex layouts of HCTs and hence being amenable to querying. This paper describes HCT-QA, an extensive benchmark of HCTs, natural language queries, and related answers on thousands of tables. Our dataset includes 2,188 real-world HCTs with 9,835 QA pairs and 4,679 synthetic tables with 67.5K QA pairs. While HCTs can be potentially processed by different type of query engines, in this paper, we focus on Large Language Models as potential engines and assess their ability in processing and querying such tables.

HCT-QA: A Benchmark for Question Answering on Human-Centric Tables

TL;DR

HCT-QA introduces a large benchmark for question answering on human centric tables embedded in documents, addressing the challenge of complex HCT layouts by evaluating a broad spectrum of LLMs and VLMs. The authors construct real world and synthetic HCT datasets with thousands of QA pairs and ground truth answers, and provide a canonical CSV representation to standardize extraction. Through extensive experiments, they show that while large closed weight models achieve the best recall around 0.68, there is substantial room for improvement, especially on nested structures and aggregations; HTML input formats and semantic quality of table names significantly impact performance. Fine tuning with real and synthetic data yields dramatic gains, underscoring the value of their synthetic generator and the potential for pretraining on HCTs. The work demonstrates both the current limitations and the practical viability of LLM/VLM based QA on HCTs, and points to future directions in pretraining, cross table QA, and improved table extraction that can unlock the value of large document repositories.

Abstract

Tabular data embedded within PDF files, web pages, and other document formats are prevalent across numerous sectors such as government, engineering, science, and business. These human-centric tables (HCTs) possess a unique combination of high business value, intricate layouts, limited operational power at scale, and sometimes serve as the only data source for critical insights. However, their complexity poses significant challenges to traditional data extraction, processing, and querying methods. While current solutions focus on transforming these tables into relational formats for SQL queries, they fall short in handling the diverse and complex layouts of HCTs and hence being amenable to querying. This paper describes HCT-QA, an extensive benchmark of HCTs, natural language queries, and related answers on thousands of tables. Our dataset includes 2,188 real-world HCTs with 9,835 QA pairs and 4,679 synthetic tables with 67.5K QA pairs. While HCTs can be potentially processed by different type of query engines, in this paper, we focus on Large Language Models as potential engines and assess their ability in processing and querying such tables.
Paper Structure (73 sections, 47 figures, 18 tables)

This paper contains 73 sections, 47 figures, 18 tables.

Table of Contents

  1. Introduction
  2. Challenging layouts.
  3. Untapped huge potential.
  4. A new perspective.
  5. Related work
  6. HCT-QA benchmark datasets
  7. Real-world dataset collection
  8. Data sources:
  9. Extraction and transformation:
  10. Characterization of table properties:
  11. Query and ground-truth answer generation
  12. Characterization of query properties:
  13. Human-Generated Queries:
  14. Model-Generated Queries:
  15. Human Verification:
  16. ...and 58 more sections

Figures (47)

  • Figure 1: Examples of Human-Centric Tables () from different real-world data sources exhibiting various levels of structural and semantic complexity: column nesting (L1); row nesting (L2); row group labels (L3); row aggregation (L4); column aggregation (L5).
  • Figure 2: Main properties of the HCTs and QA pairs in our benchmark datasets.
  • Figure 3: Example of Balanced Column Nesting
  • Figure 4: Example of Unbalanced Column Nesting
  • Figure 5: Example of Symmetric Column Nesting
  • ...and 42 more figures