Business Logic-Driven Text-to-SQL Data Synthesis for Business Intelligence

TL;DR

This paper tackles the challenge of evaluating Text-to-SQL in private BI environments where realistic domain data are scarce. It introduces a Business Logic-Driven Data Synthesis framework that grounds data generation in a structured hierarchy of personas, work scenarios, and workflows, and adds a business reasoning complexity control to diversify analytical tasks. The method demonstrates high business realism (around 98% in evaluation) and strong question–SQL alignment on a production-scale Salesforce database, while revealing substantial performance gaps for current state-of-the-art models on the most complex business queries. By combining structured business logic with LLM-based quality judging and execution-based refinement, the work provides a practical pipeline for generating realistic, domain-specific Text-to-SQL evaluation data, enabling more reliable benchmarking and development in BI contexts. The findings suggest significant implications for private-data evaluation, model robustness to real-world BI tasks, and directions for extending BI-focused T2S systems to interactive, multi-turn analytics.

Abstract

Evaluating Text-to-SQL agents in private business intelligence (BI) settings is challenging due to the scarcity of realistic, domain-specific data. While synthetic evaluation data offers a scalable solution, existing generation methods fail to capture business realism--whether questions reflect realistic business logic and workflows. We propose a Business Logic-Driven Data Synthesis framework that generates data grounded in business personas, work scenarios, and workflows. In addition, we improve the data quality by imposing a business reasoning complexity control strategy that diversifies the analytical reasoning steps required to answer the questions. Experiments on a production-scale Salesforce database show that our synthesized data achieves high business realism (98.44%), substantially outperforming OmniSQL (+19.5%) and SQL-Factory (+54.7%), while maintaining strong question-SQL alignment (98.59%). Our synthetic data also reveals that state-of-the-art Text-to-SQL models still have significant performance gaps, achieving only 42.86% execution accuracy on the most complex business queries.

Figures (2)

• Figure 1: Overview of the proposed Business Logic–Driven Text-to-SQL Data Synthesis framework. The framework starts with business logic modeling (see left), where business logic is modeled as personas, work scenarios, and workflows. This structured business logic guides business logic–driven schema selection (see middle), which scores and filters table schema based on business logic relevance. Conditioned on the selected schema and business logic context, the framework performs Text-to-SQL Data Generation (see right), comprising business-driven SQL generation with reasoning complexity control with execution-based refinement and SQL-to-question conversion, to synthesize executable queries across diverse business reasoning complexity levels and business-oriented natural language questions. Finally, a data quality evaluation module assesses question realism and question–SQL alignment, ensuring high-quality, business-realistic Text-to-SQL evaluation data (see top right).
• Figure 2: An example structured persona-centric business logic in the sales domain.