AI-Assisted SQL Authoring at Industry Scale

TL;DR

This work addresses AI-assisted SQL authoring at industry scale by differentiating SQL from general programming languages through its declarative nature, schema dependence, and non-sequential coding patterns. It introduces a three-model progression—Public Llama baseline, SqlComposeSA (schema-aware fine-tuning), and SqlComposeFIM (fill-in-the-middle with bidirectional context)—evaluated both offline on the SeqComp benchmark and in large-scale production at Meta. Empirical results show substantial gains for SqlComposeSA and especially SqlComposeFIM across metrics such as Exact Match, BLEU, Containment, and Table Match, with SqlComposeFIM achieving strong multi-line performance and 75% correct table name prediction. The rollout data indicate broad adoption (tens of thousands of users) and meaningful productivity benefits, while highlighting ongoing challenges around table/column name hallucinations and UX integration, ultimately illustrating that smaller, domain-specialist models can outperform larger general-purpose LLMs in real-world SQL completion. The work provides a concrete, scalable blueprint for deploying AI-assisted SQL at scale and contributes domain-specific benchmarks and metrics to the field.

Abstract

SqlCompose brings generative AI into the data analytics domain. SQL is declarative, has formal table schemas, and is often written in a non-linear manner. We address each of these challenges and develop a set of models that shows the importance of each problem. We first develop an internal SQL benchmark to perform offline tests at Meta. We evaluate how well the Public Llama model performs. We attain a BLEU score of 53% and 24% for single- and multi-line predictions, respectively. This performance is consistent with prior works on imperative languages. We then fine-tune Llama on our internal data and database schemas. SqlComposeSA substantially outperforms Llama by 16 percentage points on BLEU score. SQL is often written with multiple sub queries and in a non-sequential manner. We develop SqlComposeFIM which is aware of the context before and after the line(s) that need to be completed. This fill-in-the-middle model outperform SqlComposeFIM by 35 percentage points. We also measure how often the models get the correct table names, and SqlComposeFIM is able to do this 75% of the time. Aside from our scientific research, we also roll out SqlComposeFIM at Meta. SqlCompose is used on a weekly basis by over 10k users including data scientists and software engineers, less than 1% of users have disabled SqlCompose. We use the feedback from users to improve SqlCompose. Interesting positive themes include completing tedious or repetitive SQL clauses, suggesting boilerplate coding, and help in eliminate the need to remember difficult SQL syntax. The most significant negative themes was table and column name hallucinations, which has been reduced with the release of SqlComposeFIM. The SqlCompose models consistently outperform public and internal LLMs, despite being smaller (7 bn and 13 bn), which provides early indications that smaller specialist models can outperform larger general purpose models.

Figures (3)

• Figure 1: Data pyramid
• Figure 2: The three screenshots demonstrate how SqlCompose has improved its accuracy with first-party fine tuning and FIM training. The first image shows a suggestion from the public Llama model, the second image from SqlComposeSA, and the third is generated by SqlComposeFIM with FIM training. Red arrows indicate the mistakes made by the model and green arrows indicate the correct predictions. One can see how these models recovered gradually from their mistakes and hallucinations with training improvements (from Llama to SqlComposeSA to SqlComposeFIM)
• Figure 3: Converting a query to fill-in-the-middle (FIM) format with SqlComposeFIM's processor