Learning from Imperfect Data: Towards Efficient Knowledge Distillation of Autoregressive Language Models for Text-to-SQL

TL;DR

The core of KID is to efficiently mitigate the training-inference mismatch by simulating the cascading effect of inference in the imperfect training data, which effectively boosts the performance without introducing much training budget.

Abstract

Large Language Models (LLMs) have shown promising performance in text-to-SQL, which involves translating natural language questions into SQL queries. However, current text-to-SQL LLMs are computationally expensive and challenging to deploy in real-world applications, highlighting the importance of compressing them. To achieve this goal, knowledge distillation (KD) is a common approach, which aims to distill the larger teacher model into a smaller student model. While numerous KD methods for autoregressive LLMs have emerged recently, it is still under-explored whether they work well in complex text-to-SQL scenarios. To this end, we conduct a series of analyses and reveal that these KD methods generally fall short in balancing performance and efficiency. In response to this problem, we propose to improve the KD with Imperfect Data, namely KID, which effectively boosts the performance without introducing much training budget. The core of KID is to efficiently mitigate the training-inference mismatch by simulating the cascading effect of inference in the imperfect training data. Extensive experiments on 5 text-to-SQL benchmarks show that, KID can not only achieve consistent and significant performance gains (up to +5.83% average score) across all model types and sizes, but also effectively improve the training efficiency.

Figures (7)

• Figure 1: Comparisons of different KD methods for distilling the student model (QWen1.5-0.5B) from the teacher (QWen1.5-4B). The x-axis denotes the training latency relative to the SFT baseline, while the y-axis denotes the average performance of students on several popular text-to-SQL benchmarks. The evaluation details are in §\ref{['sec:experiments']}. We see that our method achieves the best trade-off between performance and efficiency.
• Figure 2: Comparisons of training latency between various KD methods. The x-axis denotes the teacher models, and the y-axis denotes the training latency relative to the SFT baseline. For ease of illustration, we only report the results of RKL divergence for GKD.
• Figure 3: Illustrations of different KD methods: (a) KD methods with ground-truth data, (b) KD methods with model-generated data and (c) our KID method with imperfect data. Additionally, we show (d) the pipeline to obtain the imperfect data, which contains three-stage processes: ❶ masking, ❷ predicting and ❸ rewriting.
• Figure 4: Analysis of different masking strategies. The y-axis denotes the EX performance on Spider-dev. For reference, we also report the results of SFT.
• Figure 5: Parameter analysis of masking ratio $\alpha$. We report the EX results of TinyLLaMA-1.1B and CodeGen-350M on the Spider-dev.