Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Factoid Questions to Data Product Requests: Benchmarking Data Product Discovery over Tables and Text

Liangliang Zhang, Nandana Mihindukulasooriya, Niharika S. D'Souza, Sola Shirai, Sarthak Dash, Yao Ma, Horst Samulowitz

TL;DR

The paper tackles the challenge of discovering data products within data lakes by introducing DPBench, a benchmark that repurposes table-text QA data into data product requests (DPRs) with full provenance. It presents a four-stage pipeline—topic clustering, schema-aware refinement, multi-LLM DPR generation, and two-phase LLM validation—and validates the approach on HybridQA, TAT-QA, and ConvFinQA. DPBench provides ground-truth DPRs with associated tables and passages and offers baseline retrieval results using BM25, dense, and hybrid methods to establish feasibility and highlight challenges. This work lays groundwork for semantically aware data product discovery and automatic assembly of data products from mixed table-text corpora, with practical implications for faster insight generation in enterprises.

Abstract

Data products are reusable, self-contained assets designed for specific business use cases. Automating their discovery and generation is of great industry interest, as it enables discovery in large data lakes and supports analytical Data Product Requests (DPRs). Currently, there is no benchmark established specifically for data product discovery. Existing datasets focus on answering single factoid questions over individual tables rather than collecting multiple data assets for broader, coherent products. To address this gap, we introduce Data Product Benchmark, the first user-request-driven data product benchmark over hybrid table-text corpora. Our framework systematically repurposes existing table-text QA datasets by clustering related tables and passages into coherent data products, generating professional-level analytical requests that span both data sources, and validating benchmark quality through multi-LLM evaluation. Data Product Benchmark preserves full provenance while producing actionable, analyst-like data product requests. Baseline experiments with hybrid retrieval methods establish the feasibility of DPR evaluation, reveal current limitations, and point to new opportunities for automatic data product discovery research. Code and datasets are available at: Dataset: https://huggingface.co/datasets/ibm-research/data-product-benchmark Code: https://github.com/ibm/data-product-benchmark

From Factoid Questions to Data Product Requests: Benchmarking Data Product Discovery over Tables and Text

TL;DR

The paper tackles the challenge of discovering data products within data lakes by introducing DPBench, a benchmark that repurposes table-text QA data into data product requests (DPRs) with full provenance. It presents a four-stage pipeline—topic clustering, schema-aware refinement, multi-LLM DPR generation, and two-phase LLM validation—and validates the approach on HybridQA, TAT-QA, and ConvFinQA. DPBench provides ground-truth DPRs with associated tables and passages and offers baseline retrieval results using BM25, dense, and hybrid methods to establish feasibility and highlight challenges. This work lays groundwork for semantically aware data product discovery and automatic assembly of data products from mixed table-text corpora, with practical implications for faster insight generation in enterprises.

Abstract

Data products are reusable, self-contained assets designed for specific business use cases. Automating their discovery and generation is of great industry interest, as it enables discovery in large data lakes and supports analytical Data Product Requests (DPRs). Currently, there is no benchmark established specifically for data product discovery. Existing datasets focus on answering single factoid questions over individual tables rather than collecting multiple data assets for broader, coherent products. To address this gap, we introduce Data Product Benchmark, the first user-request-driven data product benchmark over hybrid table-text corpora. Our framework systematically repurposes existing table-text QA datasets by clustering related tables and passages into coherent data products, generating professional-level analytical requests that span both data sources, and validating benchmark quality through multi-LLM evaluation. Data Product Benchmark preserves full provenance while producing actionable, analyst-like data product requests. Baseline experiments with hybrid retrieval methods establish the feasibility of DPR evaluation, reveal current limitations, and point to new opportunities for automatic data product discovery research. Code and datasets are available at: Dataset: https://huggingface.co/datasets/ibm-research/data-product-benchmark Code: https://github.com/ibm/data-product-benchmark
Paper Structure (15 sections, 2 equations, 1 figure, 2 tables)

This paper contains 15 sections, 2 equations, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Data Product Benchmark
  4. Setup and Notation
  5. Topic Clustering
  6. Data Product Discovery
  7. DPR Generation
  8. LLM-as-a-Judge Annotations
  9. Benchmark Datasets
  10. Example (HybridQA).
  11. Experiment
  12. Experimental Setup
  13. Results and Analysis
  14. Conclusion
  15. Prompts Used for Generation and Validation

Figures (1)

  • Figure 1: Overall workflow of Data Product Benchmark construction. QA pairs are clustered and grouped based on their focal tables to produce data products. For each data product, multiple LLMs generate corresponding DPRs, validate their quality.