Bertrand-DR: Improving Text-to-SQL using a Discriminative Re-ranker
Amol Kelkar, Rohan Relan, Vaishali Bhardwaj, Saurabh Vaichal, Chandra Khatri, Peter Relan
TL;DR
Bertrand-DR introduces a discriminative re-ranker that uses a BERT-based binary classifier to select the correct SQL query from beam-produced candidates in text-to-SQL tasks. The re-ranker is trained without schema during inference and employs a threshold to safely promote candidates, improving performance when the gold query is in the beam but not top-ranked. Applied to GNN and EditSQL on the Spider dataset, Bertrand-DR yields substantial gains and achieves top-4 leaderboard standings, with analysis showing benefits across query hardness levels and insights into when schema information helps or hurts. The work also provides threshold-tuning guidance and discusses potential extensions, including hardness-specific re-rankers and ensembles.
Abstract
To access data stored in relational databases, users need to understand the database schema and write a query using a query language such as SQL. To simplify this task, text-to-SQL models attempt to translate a user's natural language question to corresponding SQL query. Recently, several generative text-to-SQL models have been developed. We propose a novel discriminative re-ranker to improve the performance of generative text-to-SQL models by extracting the best SQL query from the beam output predicted by the text-to-SQL generator, resulting in improved performance in the cases where the best query was in the candidate list, but not at the top of the list. We build the re-ranker as a schema agnostic BERT fine-tuned classifier. We analyze relative strengths of the text-to-SQL and re-ranker models across different query hardness levels, and suggest how to combine the two models for optimal performance. We demonstrate the effectiveness of the re-ranker by applying it to two state-of-the-art text-to-SQL models, and achieve top 4 score on the Spider leaderboard at the time of writing this article.