Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LongDA: Benchmarking LLM Agents for Long-Document Data Analysis

Yiyang Li, Zheyuan Zhang, Tianyi Ma, Zehong Wang, Keerthiram Murugesan, Chuxu Zhang, Yanfang Ye

TL;DR

LongDA targets the evaluation of LLM agents in documentation-intensive data analysis by grounding tasks in real, large-scale government datasets and long unstructured documentation. It introduces LongTA, a tool-augmented ReAct-style framework, and constructs 505 reproducible queries from 17 national surveys to probe retrieval, reasoning, and code execution within strict constraints. The study reveals substantial performance gaps across state-of-the-art models, with success hinging primarily on information retrieval and tool-use strategies rather than pure reasoning, underlining the challenges of long-context, document-grounded analytics. The results motivate focused advances in document navigation, content integration, and reliable tool orchestration to enable robust decision support in real-world analytical settings.

Abstract

We introduce LongDA, a data analysis benchmark for evaluating LLM-based agents under documentation-intensive analytical workflows. In contrast to existing benchmarks that assume well-specified schemas and inputs, LongDA targets real-world settings in which navigating long documentation and complex data is the primary bottleneck. To this end, we manually curate raw data files, long and heterogeneous documentation, and expert-written publications from 17 publicly available U.S. national surveys, from which we extract 505 analytical queries grounded in real analytical practice. Solving these queries requires agents to first retrieve and integrate key information from multiple unstructured documents, before performing multi-step computations and writing executable code, which remains challenging for existing data analysis agents. To support the systematic evaluation under this setting, we develop LongTA, a tool-augmented agent framework that enables document access, retrieval, and code execution, and evaluate a range of proprietary and open-source models. Our experiments reveal substantial performance gaps even among state-of-the-art models, highlighting the challenges researchers should consider before applying LLM agents for decision support in real-world, high-stakes analytical settings.

LongDA: Benchmarking LLM Agents for Long-Document Data Analysis

TL;DR

LongDA targets the evaluation of LLM agents in documentation-intensive data analysis by grounding tasks in real, large-scale government datasets and long unstructured documentation. It introduces LongTA, a tool-augmented ReAct-style framework, and constructs 505 reproducible queries from 17 national surveys to probe retrieval, reasoning, and code execution within strict constraints. The study reveals substantial performance gaps across state-of-the-art models, with success hinging primarily on information retrieval and tool-use strategies rather than pure reasoning, underlining the challenges of long-context, document-grounded analytics. The results motivate focused advances in document navigation, content integration, and reliable tool orchestration to enable robust decision support in real-world analytical settings.

Abstract

We introduce LongDA, a data analysis benchmark for evaluating LLM-based agents under documentation-intensive analytical workflows. In contrast to existing benchmarks that assume well-specified schemas and inputs, LongDA targets real-world settings in which navigating long documentation and complex data is the primary bottleneck. To this end, we manually curate raw data files, long and heterogeneous documentation, and expert-written publications from 17 publicly available U.S. national surveys, from which we extract 505 analytical queries grounded in real analytical practice. Solving these queries requires agents to first retrieve and integrate key information from multiple unstructured documents, before performing multi-step computations and writing executable code, which remains challenging for existing data analysis agents. To support the systematic evaluation under this setting, we develop LongTA, a tool-augmented agent framework that enables document access, retrieval, and code execution, and evaluate a range of proprietary and open-source models. Our experiments reveal substantial performance gaps even among state-of-the-art models, highlighting the challenges researchers should consider before applying LLM agents for decision support in real-world, high-stakes analytical settings.
Paper Structure (54 sections, 2 equations, 10 figures, 6 tables)

This paper contains 54 sections, 2 equations, 10 figures, 6 tables.

Table of Contents

  1. Introduction
  2. LongDA Benchmark
  3. Data Collection
  4. Query and Ground Truth Extraction
  5. Benchmark Statistics
  6. Evaluation Protocol
  7. LongTA Framework
  8. ReAct Coding Agent
  9. Block-level prompt.
  10. Iterative reasoning-and-action loop.
  11. Execution environment and safety.
  12. Tool Design
  13. Unified document representation.
  14. Document access and retrieval.
  15. Prompt revisiting and note-taking.
  16. ...and 39 more sections

Figures (10)

  • Figure 1: Comparison of existing data analysis benchmarks and LongDA.
  • Figure 2: Hierarchical composition of the LongDA benchmark. Numbers in parentheses indicate the number of queries associated with each entity, with a total of 505 queries. Publications share the same color as their parent survey, and surveys belonging to the same federal agency are rendered in the same color family.
  • Figure 3: Illustration of LongTA. The agent solves queries through a ReAct-style multi-turn interaction, coordinating multiple tools for long-document navigation and code execution over heterogeneous data sources.
  • Figure 4: Match rate vs. total token consumption.
  • Figure 5: Match rate sensitivity to tolerance threshold.
  • ...and 5 more figures