Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Distill-C: Enhanced NL2SQL via Distilled Customization with LLMs

Cong Duy Vu Hoang, Gioacchino Tangari, Clemence Lanfranchi, Dalu Guo, Paul Cayet, Steve Siu, Don Dharmasiri, Yuan-Fang Li, Long Duong, Damien Hilloulin, Rhicheek Patra, Sungpack Hong, Hassan Chafi

TL;DR

Distill-C presents a customizable distillation framework for NL2SQL that transfers knowledge from large teacher LLMs to smaller models via targeted data synthesis, separation of NL and SQL generation, and a robust multi-stage filtering pipeline. By incorporating three enterprise-oriented customization scenarios (AddRef, LearnPrior, FixIt) and leveraging ensembles of high-capacity LLMs, the approach yields substantial performance gains on DateTime, Financial Analytics, and OracleSQL Compliance benchmarks while reducing runtime costs. The framework demonstrates strong improvements on internal customer benchmarks and includes thorough ablation studies confirming the benefit of combining all customization signals. Overall, Distill-C offers a scalable path for deploying lightweight yet powerful NL2SQL models tailored to specific business needs.

Abstract

The growing adoption of large language models (LLMs) in business applications has amplified interest in Natural Language to SQL (NL2SQL) solutions, in which there is competing demand for high performance and efficiency. Domain- and customer-specific requirements further complicate the problem. To address this conundrum, we introduce Distill-C, a distilled customization framework tailored for NL2SQL tasks. Distill-C utilizes large teacher LLMs to produce high-quality synthetic data through a robust and scalable pipeline. Finetuning smaller and open-source LLMs on this synthesized data enables them to rival or outperform teacher models an order of magnitude larger. Evaluated on multiple challenging benchmarks, Distill-C achieves an average improvement of 36% in execution accuracy compared to the base models from three distinct LLM families. Additionally, on three internal customer benchmarks, Distill-C demonstrates a 22.6% performance improvement over the base models. Our results demonstrate that Distill-C is an effective, high-performing and generalizable approach for deploying lightweight yet powerful NL2SQL models, delivering exceptional accuracies while maintaining low computational cost.

Distill-C: Enhanced NL2SQL via Distilled Customization with LLMs

TL;DR

Distill-C presents a customizable distillation framework for NL2SQL that transfers knowledge from large teacher LLMs to smaller models via targeted data synthesis, separation of NL and SQL generation, and a robust multi-stage filtering pipeline. By incorporating three enterprise-oriented customization scenarios (AddRef, LearnPrior, FixIt) and leveraging ensembles of high-capacity LLMs, the approach yields substantial performance gains on DateTime, Financial Analytics, and OracleSQL Compliance benchmarks while reducing runtime costs. The framework demonstrates strong improvements on internal customer benchmarks and includes thorough ablation studies confirming the benefit of combining all customization signals. Overall, Distill-C offers a scalable path for deploying lightweight yet powerful NL2SQL models tailored to specific business needs.

Abstract

The growing adoption of large language models (LLMs) in business applications has amplified interest in Natural Language to SQL (NL2SQL) solutions, in which there is competing demand for high performance and efficiency. Domain- and customer-specific requirements further complicate the problem. To address this conundrum, we introduce Distill-C, a distilled customization framework tailored for NL2SQL tasks. Distill-C utilizes large teacher LLMs to produce high-quality synthetic data through a robust and scalable pipeline. Finetuning smaller and open-source LLMs on this synthesized data enables them to rival or outperform teacher models an order of magnitude larger. Evaluated on multiple challenging benchmarks, Distill-C achieves an average improvement of 36% in execution accuracy compared to the base models from three distinct LLM families. Additionally, on three internal customer benchmarks, Distill-C demonstrates a 22.6% performance improvement over the base models. Our results demonstrate that Distill-C is an effective, high-performing and generalizable approach for deploying lightweight yet powerful NL2SQL models, delivering exceptional accuracies while maintaining low computational cost.

Paper Structure

This paper contains 25 sections, 10 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Customization Scenarios
  4. The Distill-C Framework
  5. Distillation Pipelines
  6. Multi-Step Filtering Pipeline
  7. Finetuning
  8. Experiments
  9. Evaluation Tasks
  10. Data and Evaluation Settings
  11. Model Settings
  12. Public Main Results
  13. Customer Impact
  14. Ablation Study
  15. Related Work
  16. ...and 10 more sections

Figures (10)

  • Figure 1: The Proposed Distill-C Framework.
  • Figure 2: The Multi-Step Filtering Pipeline in our Distill-C Framework.
  • Figure 3: FixIt Ablation Study Experiments. Performance of student models finetuned with the FixIt scenario using Distill-C on Spider (dev) sub-groups, showing results for student, finetuned, and teacher models, with sample counts per group.
  • Figure 4: Ablation study with distillation settings (Table \ref{['tab:settings']}). Notations: spd: Spider, bd: Bird, dt: DateTime, ana: Analytics, ora: OracleSQL, lite: SQLite, cpl: Compliance. Numerical results are reported in Appendix \ref{['appendix:abl']}.
  • Figure 5: The NL Synthesizer Pipeline in our Distill-C Framework.
  • ...and 5 more figures