RUST-BENCH: Benchmarking LLM Reasoning on Unstructured Text within Structured Tables

TL;DR

RUST-BENCH targets the gap in evaluating LLMs on real-world tabular data by combining long semi-structured tables with domain-specific content and multi-hop reasoning. It introduces a hybrid symbolic–semantic QA generation pipeline and validates data quality via human-in-the-loop, producing 7,966 questions over 2,031 tables from science and sports domains. Experiments show current LLMs struggle with scale, heterogeneity, and multi-hop inference, even with advanced prompting and table-reasoning baselines. The work establishes a challenging, extensible benchmark that enables diagnosing and advancing tabular reasoning for real-world applications.

Abstract

Existing tabular reasoning benchmarks mostly test models on small, uniform tables, underrepresenting the complexity of real-world data and giving an incomplete view of Large Language Models' (LLMs) reasoning abilities. Real tables are long, heterogeneous, and domain-specific, mixing structured fields with free text and requiring multi-hop reasoning across thousands of tokens. To address this gap, we introduce RUST-BENCH, a benchmark of 7966 questions from 2031 real-world tables spanning two domains: i) RB-Science (NSF grant records) and ii) RB-Sports (NBA statistics). Unlike prior work, RUST-BENCH evaluates LLMs jointly across scale, heterogeneity, domain specificity, and reasoning complexity. Experiments with open-source and proprietary models show that LLMs struggle with heterogeneous schemas and complex multi-hop inference, revealing persistent weaknesses in current architectures and prompting strategies. RUST-BENCH establishes a challenging new testbed for advancing tabular reasoning research.

Figures (24)

• Figure 1: Illustration of a multi-step reasoning process for a complex question grounded in a sports table from RUST-BENCH. The example shows that real-world tabular reasoning often demands multiple complementary reasoning skills (temporal, arithmetic, and contextual) and the coordinated use of heterogeneous evidence across long, domain-specific tables.
• Figure 2: Overview of RUST-BENCH's dataset generation and verification pipeline. (a) Table Generation: Raw data are extracted from public web sources and reorganized into tables containing at least 30 rows each. (b) Dataset Generation: Question–Answer pairs are created through two complementary methods: (i) a symbolic approach, which uses SQL-like logical forms to construct schema-intensive, reasoning-heavy queries, and (ii) a semantic approach, which employs LLMs to generate natural, inference-oriented questions from unstructured text. (c) Dataset Verification: All generated pairs undergo human verification to ensure factual correctness and annotation quality.
• Figure 3: Accuracy comparison of LLMs across varying token count bins. The x-axis represents token length ranges, while the y-axis shows accuracy in percentage.
• Figure 4: Performance comparison of LLM backbones on RUST-BENCH and WikiTQ using EM accuracy. Unlike WikiTQ, RUST-BENCH tests LLMs with more challenging questions and tables, resulting in a reduced LLM performance.
• Figure 5: Performance comparison on structured and semi-structured variants for different LLM backbones using Program-of-Thought (PoT) prompting.