Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Thinkquel: A Model Dedicated to Text-to-dbt Using Synthetic Data and a Span-Aware Objective

Anni Li, Aria Attar, Paul Dong

TL;DR

Thinkquel addresses the difficulty of producing production-ready dbt transformations from natural language by combining a scalable synthetic NL+dbt data pipeline with a span-aware reinforcement learning objective. The approach pairs a two-stage supervised fine-tuning curriculum with Token–Sequence GRPO (TS–GRPO), which separates planning (token-level) from the production dbt/SQL output (sequence-level), to align optimization with execution-based supervision. Empirical results show strong in-domain performance on TS-SQL and Spider, with robust stability and competitive out-of-domain results on BIRD-dbt, driven by planning-aware training and span-based credit routing. These contributions advance portable, execution-validated text-to-dbt generation and suggest future work on broader warehouse portability and tool-assisted verification.

Abstract

Transforming natural-language requests into reliable, production-ready data transformations remains challenging: correctness depends on precise schema linking and warehouse-specific SQL dialects, while the strongest supervision available during training--execution success and result matching--are provided only at the sequence level. At the same time, assembling large, execution-validated corpora is costly, and token-level objectives misalign with these global signals, yielding unstable optimization and limited portability. We introduce Thinkquel, a fine-tuned model for producing robust, portable, and execution-validated database queries. Methodologies in Thinkquel integrates a novel synthetic data pipeline, TS-SQL, that leverages dbt as a portable intermediate representation with a span-aware reinforcement learning objective, and Token-Sequence GRPO (TS-GRPO), specifically designed to bridge the gap between token-level training signals and sequence-level execution rewards when finetuning LLMs. On the 500-example TS-SQL test set, Thinkquel (32B) reaches 93.2% execution success and 61.8% exact-result match with a two-stage SFT curriculum, improving over the base model by 67.2% (exec.) and 44.4% (match). In Spider (14B) experiments, TS-GRPO increases training stability and speeds convergence of the execution-match reward relative to GRPO and GSPO.

Thinkquel: A Model Dedicated to Text-to-dbt Using Synthetic Data and a Span-Aware Objective

TL;DR

Thinkquel addresses the difficulty of producing production-ready dbt transformations from natural language by combining a scalable synthetic NL+dbt data pipeline with a span-aware reinforcement learning objective. The approach pairs a two-stage supervised fine-tuning curriculum with Token–Sequence GRPO (TS–GRPO), which separates planning (token-level) from the production dbt/SQL output (sequence-level), to align optimization with execution-based supervision. Empirical results show strong in-domain performance on TS-SQL and Spider, with robust stability and competitive out-of-domain results on BIRD-dbt, driven by planning-aware training and span-based credit routing. These contributions advance portable, execution-validated text-to-dbt generation and suggest future work on broader warehouse portability and tool-assisted verification.

Abstract

Transforming natural-language requests into reliable, production-ready data transformations remains challenging: correctness depends on precise schema linking and warehouse-specific SQL dialects, while the strongest supervision available during training--execution success and result matching--are provided only at the sequence level. At the same time, assembling large, execution-validated corpora is costly, and token-level objectives misalign with these global signals, yielding unstable optimization and limited portability. We introduce Thinkquel, a fine-tuned model for producing robust, portable, and execution-validated database queries. Methodologies in Thinkquel integrates a novel synthetic data pipeline, TS-SQL, that leverages dbt as a portable intermediate representation with a span-aware reinforcement learning objective, and Token-Sequence GRPO (TS-GRPO), specifically designed to bridge the gap between token-level training signals and sequence-level execution rewards when finetuning LLMs. On the 500-example TS-SQL test set, Thinkquel (32B) reaches 93.2% execution success and 61.8% exact-result match with a two-stage SFT curriculum, improving over the base model by 67.2% (exec.) and 44.4% (match). In Spider (14B) experiments, TS-GRPO increases training stability and speeds convergence of the execution-match reward relative to GRPO and GSPO.

Paper Structure

This paper contains 45 sections, 18 equations, 8 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Why this is hard.
  3. Thinkquel: A Model Dedicated to Text-to-dbt Using Synthetic Data and a Span-Aware Objective.
  4. Contributions.
  5. Results in brief.
  6. Roadmap.
  7. Preliminaries
  8. Synthetic Dataset
  9. Group-Relative Policy Optimization (GRPO)
  10. Group-relative advantage.
  11. Importance ratios.
  12. Per-token proximal surrogate.
  13. Group Sequence Policy Optimization (GSPO)
  14. Sequence-level importance ratio.
  15. Sequence-level proximal surrogate.
  16. ...and 30 more sections

Figures (8)

  • Figure 1: TensorStax-SQL Data Synthesis Pipeline.
  • Figure 2: An Example of the Model's Completion.
  • Figure 3: Distribution of Average log-probability of SQL Token with Planning vs. No Planning
  • Figure 4: TS-GRPO vs. GRPO vs. GSPO on Spider Dataset
  • Figure 5: Learning Curves of Thinkquel on TS-SQL: Mean Match Reward vs. Steps.
  • ...and 3 more figures