SQLBarber: A System Leveraging Large Language Models to Generate Customized and Realistic SQL Workloads
Jiale Lao, Immanuel Trummer
TL;DR
SQLBarber tackles the challenge of creating realistic, customized SQL workloads under privacy constraints by leveraging LLMs to generate SQL templates from natural-language specifications and aligning query costs with real-world distributions derived from Redshift and Snowflake statistics. The system combines a self-correcting template generator with a cost-aware query generator that profiles templates, refines them to cover cost intervals, and uses Bayesian Optimization to instantiate queries that match target distributions. Ten open benchmarks and extensive experiments demonstrate that SQLBarber both dramatically reduces generation time and achieves precise cost-distribution alignment, outperforming prior approaches. The work enables practical, realistic benchmarking for DBMS research and development and lays the groundwork for extending workload synthesis to broader cost metrics and database types.
Abstract
Database research and development often require a large number of SQL queries for benchmarking purposes. However, acquiring real-world SQL queries is challenging due to privacy concerns, and existing SQL generation methods are limited in customization and in satisfying realistic constraints. To address this issue, we present SQLBarber, a system based on Large Language Models (LLMs) to generate customized and realistic SQL workloads. SQLBarber (i) eliminates the need for users to manually craft SQL templates in advance, while providing the flexibility to accept natural language specifications to constrain SQL templates, (ii) scales efficiently to generate large volumes of queries matching any user-defined cost distribution (e.g., cardinality and execution plan cost), and (iii) uses execution statistics from Amazon Redshift and Snowflake to derive SQL template specifications and query cost distributions that reflect real-world query characteristics. SQLBarber introduces (i) a declarative interface for users to effortlessly generate customized SQL templates, (ii) an LLM-powered pipeline augmented with a self-correction module that profiles, refines, and prunes SQL templates based on query costs, and (iii) a Bayesian Optimizer to efficiently explore different predicate values and identify a set of queries that satisfy the target cost distribution. We construct and open-source ten benchmarks of varying difficulty levels and target query cost distributions based on real-world statistics from Snowflake and Amazon Redshift. Extensive experiments on these benchmarks show that SQLBarber is the only system that can generate customized SQL templates. It reduces query generation time by one to three orders of magnitude, and significantly improves alignment with the target cost distribution, compared with existing methods.