Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hierarchical structure understanding in complex tables with VLLMs: a benchmark and experiments

Luca Bindini, Simone Giovannini, Simone Marinai, Valeria Nardoni, Kimiya Noor Ali

TL;DR

CHiTab introduces a benchmark for evaluating hierarchical header understanding in complex scientific tables using PubTables-1M. It defines two QA tasks, VLQA and SHQA, to quantify how well Vision-Large Language Models infer header hierarchies, and evaluates off-the-shelf and fine-tuned VLLMs, along with human baselines, across diverse prompt styles. Results show that VLLMs can learn partial hierarchical reasoning, with leaf-level counting remaining challenging in zero-shot settings, but targeted QLoRA fine-tuning can dramatically improve performance toward human levels. CHiTab highlights current limitations in structured data understanding within general-purpose VLLMs and provides a concrete, extensible path for integrating hierarchical table reasoning into multimodal models.

Abstract

This work investigates the ability of Vision Large Language Models (VLLMs) to understand and interpret the structure of tables in scientific articles. Specifically, we explore whether VLLMs can infer the hierarchical structure of tables without additional processing. As a basis for our experiments we use the PubTables-1M dataset, a large-scale corpus of scientific tables. From this dataset, we extract a subset of tables that we introduce as Complex Hierarchical Tables (CHiTab): a benchmark collection of complex tables containing hierarchical headings. We adopt a series of prompt engineering strategies to probe the models' comprehension capabilities, experimenting with various prompt formats and writing styles. Multiple state-of-the-art open-weights VLLMs are evaluated on the benchmark first using their off-the-shelf versions and then fine-tuning some models on our task. We also measure the performance of humans to solve the task on a small set of tables comparing with performance of the evaluated VLLMs. The experiments support our intuition that generic VLLMs, not explicitly designed for understanding the structure of tables, can perform this task. This study provides insights into the potential and limitations of VLLMs to process complex tables and offers guidance for future work on integrating structured data understanding into general-purpose VLLMs.

Hierarchical structure understanding in complex tables with VLLMs: a benchmark and experiments

TL;DR

CHiTab introduces a benchmark for evaluating hierarchical header understanding in complex scientific tables using PubTables-1M. It defines two QA tasks, VLQA and SHQA, to quantify how well Vision-Large Language Models infer header hierarchies, and evaluates off-the-shelf and fine-tuned VLLMs, along with human baselines, across diverse prompt styles. Results show that VLLMs can learn partial hierarchical reasoning, with leaf-level counting remaining challenging in zero-shot settings, but targeted QLoRA fine-tuning can dramatically improve performance toward human levels. CHiTab highlights current limitations in structured data understanding within general-purpose VLLMs and provides a concrete, extensible path for integrating hierarchical table reasoning into multimodal models.

Abstract

This work investigates the ability of Vision Large Language Models (VLLMs) to understand and interpret the structure of tables in scientific articles. Specifically, we explore whether VLLMs can infer the hierarchical structure of tables without additional processing. As a basis for our experiments we use the PubTables-1M dataset, a large-scale corpus of scientific tables. From this dataset, we extract a subset of tables that we introduce as Complex Hierarchical Tables (CHiTab): a benchmark collection of complex tables containing hierarchical headings. We adopt a series of prompt engineering strategies to probe the models' comprehension capabilities, experimenting with various prompt formats and writing styles. Multiple state-of-the-art open-weights VLLMs are evaluated on the benchmark first using their off-the-shelf versions and then fine-tuning some models on our task. We also measure the performance of humans to solve the task on a small set of tables comparing with performance of the evaluated VLLMs. The experiments support our intuition that generic VLLMs, not explicitly designed for understanding the structure of tables, can perform this task. This study provides insights into the potential and limitations of VLLMs to process complex tables and offers guidance for future work on integrating structured data understanding into general-purpose VLLMs.

Paper Structure

This paper contains 16 sections, 1 equation, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. CHiTab Benchmark
  4. Table Filtering
  5. Header Hierarchy Construction
  6. Ground Truth Generation for QA Tasks
  7. VLQA:
  8. SHQA:
  9. Benchmark Statistics
  10. Experiments
  11. Prompt tuning
  12. Zero shot
  13. Fine-tuning
  14. Human baseline
  15. Conclusions
  16. ...and 1 more sections

Figures (5)

  • Figure 1: CHiTab Benchmark outline. Steps 1 – 4 represent the pipeline stages.
  • Figure 2: Histogram of the number of questions per table.
  • Figure 3: Distribution of the overall accuracy of the four VLLMs over the test split.
  • Figure 4: Examples of tables from CHiTab which are either completely misundersood or completely understood by all the tested VLLMs. All table images are shown in original resolution as found in PubTables-1M.
  • Figure 5: Accuracy and stability per group.