Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Falcon: A Comprehensive Chinese Text-to-SQL Benchmark for Enterprise-Grade Evaluation

Wenzhen Luo, Wei Guan, Yifan Yao, Yimin Pan, Feng Wang, Zhipeng Yu, Zhe Wen, Liang Chen, Yihong Zhuang

TL;DR

Falcon presents a Chinese Text-to-SQL benchmark tailored to enterprise environments, emphasizing MaxCompute/Hive dialects and wide, denormalized schemas to reflect real-world constraints. It combines public Kaggle databases with enterprise-inspired synthetic cases and introduces a dual annotation framework plus a schema-aware execution comparator to enable detailed, end-to-end evaluation. Experimental results show current large language models struggle with enterprise-scale joins and precise operator mapping, with no model exceeding 50% execution accuracy. The benchmark’s reproducible pipeline and dialect-aware evaluation offer a practical bridge between academic progress and production deployment in Chinese BI contexts.

Abstract

We introduce Falcon, a cross-domain Chinese text-to-SQL benchmark grounded in an enterprise-compatible dialect (MaxCompute/Hive). It contains 600 Chinese questions over 28 databases; 77% require multi-table reasoning and over half touch more than four tables. Each example is annotated along SQL-computation features and Chinese semantics. For evaluation, we release a robust execution comparator and an automated evaluation pipeline, under which all current state-of-the-art large-scale models (including Deepseek) achieve accuracies of at most 50%. Major errors originate from two sources: (1) schema linking in large enterprise landscapes - hundreds of tables, denormalized fields, ambiguous column names, implicit foreign-key relations and domain-specific synonyms that make correct join/column selection difficult; and (2) mapping concise, colloquial Chinese into the exact operators and predicates required for analytics - e.g., choosing the correct aggregation and group-by keys, expressing time windows and granularities, applying unit conversions, handling NULLs and data-quality rules, and formulating nested or windowed subqueries. Falcon therefore targets Chinese-specific semantics and enterprise dialects (abbreviations, business jargon, fuzzy entity references) and provides a reproducible middle ground before full production deployment by using realistic enterprise schemas, query templates, an execution comparator, and an automated evaluation pipeline for end-to-end validation.

Falcon: A Comprehensive Chinese Text-to-SQL Benchmark for Enterprise-Grade Evaluation

TL;DR

Falcon presents a Chinese Text-to-SQL benchmark tailored to enterprise environments, emphasizing MaxCompute/Hive dialects and wide, denormalized schemas to reflect real-world constraints. It combines public Kaggle databases with enterprise-inspired synthetic cases and introduces a dual annotation framework plus a schema-aware execution comparator to enable detailed, end-to-end evaluation. Experimental results show current large language models struggle with enterprise-scale joins and precise operator mapping, with no model exceeding 50% execution accuracy. The benchmark’s reproducible pipeline and dialect-aware evaluation offer a practical bridge between academic progress and production deployment in Chinese BI contexts.

Abstract

We introduce Falcon, a cross-domain Chinese text-to-SQL benchmark grounded in an enterprise-compatible dialect (MaxCompute/Hive). It contains 600 Chinese questions over 28 databases; 77% require multi-table reasoning and over half touch more than four tables. Each example is annotated along SQL-computation features and Chinese semantics. For evaluation, we release a robust execution comparator and an automated evaluation pipeline, under which all current state-of-the-art large-scale models (including Deepseek) achieve accuracies of at most 50%. Major errors originate from two sources: (1) schema linking in large enterprise landscapes - hundreds of tables, denormalized fields, ambiguous column names, implicit foreign-key relations and domain-specific synonyms that make correct join/column selection difficult; and (2) mapping concise, colloquial Chinese into the exact operators and predicates required for analytics - e.g., choosing the correct aggregation and group-by keys, expressing time windows and granularities, applying unit conversions, handling NULLs and data-quality rules, and formulating nested or windowed subqueries. Falcon therefore targets Chinese-specific semantics and enterprise dialects (abbreviations, business jargon, fuzzy entity references) and provides a reproducible middle ground before full production deployment by using realistic enterprise schemas, query templates, an execution comparator, and an automated evaluation pipeline for end-to-end validation.

Paper Structure

This paper contains 29 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Related Work and Existing Datasets
  4. Corpus Construction
  5. Data Source Composition
  6. Annotation Protocol
  7. Quality Assurance
  8. Innovative Features
  9. Dataset Statistics and Comparison
  10. Data Composition and Scale.
  11. Structural Characteristics.
  12. Linguistic Profiling.
  13. Task Definition and Evaluation Metrics
  14. Problem setting.
  15. Inputs and outputs.
  16. ...and 14 more sections

Figures (3)

  • Figure 1: Distribution showing benchmark composition (500 public + 100 enterprise-inspired questions). Ant-sourced cases concentrate in high-value domains like Finance (15/97) and E-commerce (21/131).
  • Figure 2: ExAcc by join width. The corpus mean is indicated by the dashed line.
  • Figure 3: RIGHT/ERROR composition by join width. Error share escalates with schema width.