Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BEAVER: An Enterprise Benchmark for Text-to-SQL

Peter Baile Chen, Fabian Wenz, Yi Zhang, Devin Yang, Justin Choi, Nesime Tatbul, Michael Cafarella, Çağatay Demiralp, Michael Stonebraker

TL;DR

BEAVER introduces the first enterprise text-to-SQL benchmark derived from private data warehouses to reveal real-world challenges not captured by public datasets. It systematically evaluates retrieval-augmented generation, table retrieval, and column mapping with gold annotations, showing that off-the-shelf LLMs achieve near-zero end-to-end accuracy on BEAVER. The study identifies three core difficulties: large, complex schemas; multi-table, nested queries; and the private nature of enterprise data that limits model training. The work provides a comprehensive error analysis and actionable directions, arguing that BEAVER is a crucial groundwork for building enterprise-ready text-to-SQL systems.

Abstract

Existing text-to-SQL benchmarks have largely been constructed from web tables with human-generated question-SQL pairs. LLMs typically show strong results on these benchmarks, leading to a belief that LLMs are effective at text-to-SQL tasks. However, how these results transfer to enterprise settings is unclear because tables in enterprise databases might differ substantially from web tables in structure and content. To contend with this problem, we introduce a new dataset BEAVER, the first enterprise text-to-SQL benchmark sourced from real private enterprise data warehouses. This dataset includes natural language queries and their correct SQL statements, which we collected from actual query logs. We then benchmark off-the-shelf LLMs on this dataset. LLMs perform poorly, even when augmented with standard prompt engineering and RAG techniques. We identify three main reasons for the poor performance: (1) schemas of enterprise tables are more complex than the schemas in public data, resulting in SQL-generation tasks intrinsically harder; (2) business-oriented questions are often more complex, requiring joins over multiple tables, aggregations, and nested queries; (3) public LLMs cannot train on private enterprise data warehouses that are not publicly accessible, and therefore it is difficult for the model to learn to solve (1) and (2). We believe BEAVER will facilitate future research in building text-to-SQL systems that perform better in enterprise settings.

BEAVER: An Enterprise Benchmark for Text-to-SQL

TL;DR

BEAVER introduces the first enterprise text-to-SQL benchmark derived from private data warehouses to reveal real-world challenges not captured by public datasets. It systematically evaluates retrieval-augmented generation, table retrieval, and column mapping with gold annotations, showing that off-the-shelf LLMs achieve near-zero end-to-end accuracy on BEAVER. The study identifies three core difficulties: large, complex schemas; multi-table, nested queries; and the private nature of enterprise data that limits model training. The work provides a comprehensive error analysis and actionable directions, arguing that BEAVER is a crucial groundwork for building enterprise-ready text-to-SQL systems.

Abstract

Existing text-to-SQL benchmarks have largely been constructed from web tables with human-generated question-SQL pairs. LLMs typically show strong results on these benchmarks, leading to a belief that LLMs are effective at text-to-SQL tasks. However, how these results transfer to enterprise settings is unclear because tables in enterprise databases might differ substantially from web tables in structure and content. To contend with this problem, we introduce a new dataset BEAVER, the first enterprise text-to-SQL benchmark sourced from real private enterprise data warehouses. This dataset includes natural language queries and their correct SQL statements, which we collected from actual query logs. We then benchmark off-the-shelf LLMs on this dataset. LLMs perform poorly, even when augmented with standard prompt engineering and RAG techniques. We identify three main reasons for the poor performance: (1) schemas of enterprise tables are more complex than the schemas in public data, resulting in SQL-generation tasks intrinsically harder; (2) business-oriented questions are often more complex, requiring joins over multiple tables, aggregations, and nested queries; (3) public LLMs cannot train on private enterprise data warehouses that are not publicly accessible, and therefore it is difficult for the model to learn to solve (1) and (2). We believe BEAVER will facilitate future research in building text-to-SQL systems that perform better in enterprise settings.
Paper Structure (44 sections, 4 figures, 10 tables)

This paper contains 44 sections, 4 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Dataset
  3. Task Formulation
  4. Sources
  5. Annotation
  6. Databases.
  7. SQL statements.
  8. Natural language questions.
  9. Column mapping.
  10. Statistics
  11. Benchmark
  12. Experimental setup
  13. Datasets.
  14. Retrieval-augmented Generation (RAG).
  15. Column mapping prediction.
  16. ...and 29 more sections

Figures (4)

  • Figure 1: The mean values for the number of joins, aggregations, and nesting depth for Spider, Bird, and BEAVER.
  • Figure 2: Schema of tables to illustrate retriever models did not retrieve sufficient information. A green tick means the table was retrieved, and a red cross means the table was not retrieved. Green dotted lines represent join relationships.
  • Figure 3: Schema of tables to illustrate retriever model did not retrieve connecting tables.
  • Figure 4: Interface for annotating natural language questions.