Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Aggregation Queries over Unstructured Text: Benchmark and Agentic Method

Haojia Zhu, Qinyuan Xu, Haoyu Li, Yuxi Liu, Hanchen Qiu, Jiaoyan Chen, Jiahui Jin

TL;DR

This work reframes aggregation queries over unstructured text as a completeness-oriented problem, introducing AGGBench to benchmark exhaustive evidence coverage and a modular agentic baseline, DFA, to operationalize find-all aggregation. DFA decomposes the task into disambiguation, completeness-aware filtering, and aggregation, exposing measurable failure modes and enabling principled analysis of coverage vs. noise. Empirical results show that DFA substantially improves evidence recall over strong RAG and agentic baselines, with up to 5× gains on core benchmarks and robust performance across multiple LLM backbones. The work provides a principled foundation for evaluating complete aggregation in large, sparse corpora and suggests directions for memory, tool orchestration, and iterative evidence discovery to advance completeness-aware retrieval.

Abstract

Aggregation query over free text is a long-standing yet underexplored problem. Unlike ordinary question answering, aggregate queries require exhaustive evidence collection and systems are required to "find all," not merely "find one." Existing paradigms such as Text-to-SQL and Retrieval-Augmented Generation fail to achieve this completeness. In this work, we formalize entity-level aggregation querying over text in a corpus-bounded setting with strict completeness requirement. To enable principled evaluation, we introduce AGGBench, a benchmark designed to evaluate completeness-oriented aggregation under realistic large-scale corpus. To accompany the benchmark, we propose DFA (Disambiguation--Filtering--Aggregation), a modular agentic baseline that decomposes aggregation querying into interpretable stages and exposes key failure modes related to ambiguity, filtering, and aggregation. Empirical results show that DFA consistently improves aggregation evidence coverage over strong RAG and agentic baselines. The data and code are available in https://anonymous.4open.science/r/DFA-A4C1.

Aggregation Queries over Unstructured Text: Benchmark and Agentic Method

TL;DR

This work reframes aggregation queries over unstructured text as a completeness-oriented problem, introducing AGGBench to benchmark exhaustive evidence coverage and a modular agentic baseline, DFA, to operationalize find-all aggregation. DFA decomposes the task into disambiguation, completeness-aware filtering, and aggregation, exposing measurable failure modes and enabling principled analysis of coverage vs. noise. Empirical results show that DFA substantially improves evidence recall over strong RAG and agentic baselines, with up to 5× gains on core benchmarks and robust performance across multiple LLM backbones. The work provides a principled foundation for evaluating complete aggregation in large, sparse corpora and suggests directions for memory, tool orchestration, and iterative evidence discovery to advance completeness-aware retrieval.

Abstract

Aggregation query over free text is a long-standing yet underexplored problem. Unlike ordinary question answering, aggregate queries require exhaustive evidence collection and systems are required to "find all," not merely "find one." Existing paradigms such as Text-to-SQL and Retrieval-Augmented Generation fail to achieve this completeness. In this work, we formalize entity-level aggregation querying over text in a corpus-bounded setting with strict completeness requirement. To enable principled evaluation, we introduce AGGBench, a benchmark designed to evaluate completeness-oriented aggregation under realistic large-scale corpus. To accompany the benchmark, we propose DFA (Disambiguation--Filtering--Aggregation), a modular agentic baseline that decomposes aggregation querying into interpretable stages and exposes key failure modes related to ambiguity, filtering, and aggregation. Empirical results show that DFA consistently improves aggregation evidence coverage over strong RAG and agentic baselines. The data and code are available in https://anonymous.4open.science/r/DFA-A4C1.
Paper Structure (59 sections, 6 equations, 3 figures, 9 tables)

This paper contains 59 sections, 6 equations, 3 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Aggregation Query Benchmarks
  4. Methods Related to Aggregation Queries
  5. Problem Statement
  6. Entity type.
  7. Conditions.
  8. Benchmark: AGGBench
  9. Pipeline Overview
  10. Query Construction on the Core Subset
  11. Annotation on the Core Subset
  12. Corpus Expansion
  13. DFA Framework
  14. Disambiguation
  15. Insight.
  16. ...and 44 more sections

Figures (3)

  • Figure 1: Examples of Aggregation Query and Standard Query over Unstructured Data.
  • Figure 2: Overview of the AGGBench construction pipeline.
  • Figure 3: Overview of the DFA framework.