Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BenchPress: A Human-in-the-Loop Annotation System for Rapid Text-to-SQL Benchmark Curation

Fabian Wenz, Omar Bouattour, Devin Yang, Justin Choi, Cecil Gregg, Nesime Tatbul, Çağatay Demiralp

TL;DR

BenchPress tackles the gap between public text-to-SQL benchmarks and real-world enterprise workloads by enabling rapid, privacy-preserving benchmark curation through a human-in-the-loop system that combines retrieval-augmented generation with expert validation. It generates NL descriptions from SQL using contextual grounding from retrieved examples and schema data, with optional decomposition for nested queries and iterative refinement guided by domain experts. A controlled user study demonstrates that BenchPress improves annotation accuracy and reduces latency compared with manual or vanilla LLM approaches, especially on complex enterprise queries. The work shows practical impact by providing a scalable workflow for domain-specific, robust Text-to-SQL evaluation without exposing private data, and outlines future enhancements including SQL-to-NL validation, benchmark rephrasing for generalization, and broader enterprise integration.

Abstract

Large language models (LLMs) have been successfully applied to many tasks, including text-to-SQL generation. However, much of this work has focused on publicly available datasets, such as Fiben, Spider, and Bird. Our earlier work showed that LLMs are much less effective in querying large private enterprise data warehouses and released Beaver, the first private enterprise text-to-SQL benchmark. To create Beaver, we leveraged SQL logs, which are often readily available. However, manually annotating these logs to identify which natural language questions they answer is a daunting task. Asking database administrators, who are highly trained experts, to take on additional work to construct and validate corresponding natural language utterances is not only challenging but also quite costly. To address this challenge, we introduce BenchPress, a human-in-the-loop system designed to accelerate the creation of domain-specific text-to-SQL benchmarks. Given a SQL query, BenchPress uses retrieval-augmented generation (RAG) and LLMs to propose multiple natural language descriptions. Human experts then select, rank, or edit these drafts to ensure accuracy and domain alignment. We evaluated BenchPress on annotated enterprise SQL logs, demonstrating that LLM-assisted annotation drastically reduces the time and effort required to create high-quality benchmarks. Our results show that combining human verification with LLM-generated suggestions enhances annotation accuracy, benchmark reliability, and model evaluation robustness. By streamlining the creation of custom benchmarks, BenchPress offers researchers and practitioners a mechanism for assessing text-to-SQL models on a given domain-specific workload. BenchPress is freely available via our public GitHub repository at https://github.com/fabian-wenz/enterprise-txt2sql and is also accessible on our website at http://dsg-mcgraw.csail.mit.edu:5000.

BenchPress: A Human-in-the-Loop Annotation System for Rapid Text-to-SQL Benchmark Curation

TL;DR

BenchPress tackles the gap between public text-to-SQL benchmarks and real-world enterprise workloads by enabling rapid, privacy-preserving benchmark curation through a human-in-the-loop system that combines retrieval-augmented generation with expert validation. It generates NL descriptions from SQL using contextual grounding from retrieved examples and schema data, with optional decomposition for nested queries and iterative refinement guided by domain experts. A controlled user study demonstrates that BenchPress improves annotation accuracy and reduces latency compared with manual or vanilla LLM approaches, especially on complex enterprise queries. The work shows practical impact by providing a scalable workflow for domain-specific, robust Text-to-SQL evaluation without exposing private data, and outlines future enhancements including SQL-to-NL validation, benchmark rephrasing for generalization, and broader enterprise integration.

Abstract

Large language models (LLMs) have been successfully applied to many tasks, including text-to-SQL generation. However, much of this work has focused on publicly available datasets, such as Fiben, Spider, and Bird. Our earlier work showed that LLMs are much less effective in querying large private enterprise data warehouses and released Beaver, the first private enterprise text-to-SQL benchmark. To create Beaver, we leveraged SQL logs, which are often readily available. However, manually annotating these logs to identify which natural language questions they answer is a daunting task. Asking database administrators, who are highly trained experts, to take on additional work to construct and validate corresponding natural language utterances is not only challenging but also quite costly. To address this challenge, we introduce BenchPress, a human-in-the-loop system designed to accelerate the creation of domain-specific text-to-SQL benchmarks. Given a SQL query, BenchPress uses retrieval-augmented generation (RAG) and LLMs to propose multiple natural language descriptions. Human experts then select, rank, or edit these drafts to ensure accuracy and domain alignment. We evaluated BenchPress on annotated enterprise SQL logs, demonstrating that LLM-assisted annotation drastically reduces the time and effort required to create high-quality benchmarks. Our results show that combining human verification with LLM-generated suggestions enhances annotation accuracy, benchmark reliability, and model evaluation robustness. By streamlining the creation of custom benchmarks, BenchPress offers researchers and practitioners a mechanism for assessing text-to-SQL models on a given domain-specific workload. BenchPress is freely available via our public GitHub repository at https://github.com/fabian-wenz/enterprise-txt2sql and is also accessible on our website at http://dsg-mcgraw.csail.mit.edu:5000.
Paper Structure (27 sections, 3 figures, 2 tables)

This paper contains 27 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. SQL-to-NL generation
  3. Related Work
  4. Text-to-SQL Benchmarks
  5. LLMs for SQL Generation
  6. Annotation Systems and Tools
  7. Enterprise Data Challenges
  8. BenchPress in Context
  9. Enterprise SQL Logs: Challenges
  10. Domain-Specific Terminology:
  11. Privacy Constraints:
  12. Schema Ambiguity and Duplication:
  13. Data Sparsity and Imbalance:
  14. The BenchPress System
  15. Workflow Overview
  16. ...and 12 more sections

Figures (3)

  • Figure 1: Execution accuracy of different LLMs on public benchmarks (Spider, Bird, Fiben) and the enterprise benchmark (Beaver). Since the best-performing model varies across datasets, the specific model achieving the highest accuracy is labeled above each teal-colored bar: Spider – miniSeek miniseek2023, Bird – askData askData2025, Fiben – Athena++ sen2020athenaplusplus, and Beaverchen2024beaver – contextModel.
  • Figure 2: BenchPress workflow: initial one-time setup (orange), iterative annotation loop (green), and final review (gray).
  • Figure 3: Backtranslation fidelity: proportion of SQL outputs at each clarity level across conditions.