Enhancing Text-to-SQL Translation for Financial System Design

TL;DR

The paper tackles Text-to-SQL in finance by benchmarking a suite of LLMs, introducing input optimization via prompting and rephrasing, and proposing two non-executable similarity metrics, SQAM and TSED, to evaluate SQL generation when database access is restricted. It provides a bank-focused dataset to stress-test transaction-related queries and demonstrates that models with domain-aligned, code-rich training data (e.g., nsql-6B) can outperform larger, general models in finance tasks, while prompt design and post-processing significantly impact performance. The novel TSED metric, based on tree edit distance over ASTs, offers a reliable, execution-free benchmark that correlates highly with actual execution matching, enabling practical evaluation in regulated or inaccessible environments. Overall, the work advances Text-to-SQL in real-world banking contexts by delivering empirical guidance on model selection, dataset design, and robust evaluation methods with meaningful implications for deploying natural language interfaces to financial databases.

Abstract

Text-to-SQL, the task of translating natural language questions into SQL queries, is part of various business processes. Its automation, which is an emerging challenge, will empower software practitioners to seamlessly interact with relational databases using natural language, thereby bridging the gap between business needs and software capabilities. In this paper, we consider Large Language Models (LLMs), which have achieved state of the art for various NLP tasks. Specifically, we benchmark Text-to-SQL performance, the evaluation methodologies, as well as input optimization (e.g., prompting). In light of the empirical observations that we have made, we propose two novel metrics that were designed to adequately measure the similarity between SQL queries. Overall, we share with the community various findings, notably on how to select the right LLM on Text-to-SQL tasks. We further demonstrate that a tree-based edit distance constitutes a reliable metric for assessing the similarity between generated SQL queries and the oracle for benchmarking Text2SQL approaches. This metric is important as it relieves researchers from the need to perform computationally expensive experiments such as executing generated queries as done in prior works. Our work implements financial domain use cases and, therefore contributes to the advancement of Text2SQL systems and their practical adoption in this domain.

Figures (6)

• Figure 1: Development of Large Language Models from Year 2018-2023
• Figure 2: Pipeline of Text to SQL Evaluation Practice
• Figure 3: Financial Dataset for Text-to-SQL (green cells indicate foreign keys)
• Figure 4: Type I
• Figure 5: Type II