Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Agentic NL2SQL to Reduce Computational Costs

Dominik Jehle, Lennart Purucker, Frank Hutter

TL;DR

This work tackles the high token and cost burden of NL2SQL when operating over large, multi-database corpora by introducing the Datalake Agent, an interactive framework that employs information acquisition, iterative refinement, and query formulation to selectively retrieve only necessary metadata. By comparing against a direct prompt baseline, the authors show that the Datalake Agent can reduce token usage by up to 87% while maintaining competitive accuracy on a 100-task, 23-database benchmark, including 319 tables. The approach scales better with data volume, offering substantial cost reductions in practical enterprise settings. Limitations include potential infinite reasoning loops and evaluation restricted to GPT-4 Mini, with future work aiming to broaden datasets and LLM support and to enhance loop mitigation strategies.

Abstract

Translating natural language queries into SQL queries (NL2SQL or Text-to-SQL) has recently been empowered by large language models (LLMs). Using LLMs to perform NL2SQL methods on a large collection of SQL databases necessitates processing large quantities of meta-information about the databases, which in turn results in lengthy prompts with many tokens and high processing costs. To address this challenge, we introduce Datalake Agent, an agentic system designed to enable an LLM to solve NL2SQL tasks more efficiently. Instead of utilizing direct solvers for NL2SQL that call the LLM once with all meta-information in the prompt, the Datalake Agent employs an interactive loop to reduce the utilized meta-information. Within the loop, the LLM is used in a reasoning framework that selectively requests only the necessary information to solve a table question answering task. We evaluate the Datalake Agent on a collection of 23 databases with 100 table question answering tasks. The Datalake Agent reduces the tokens used by the LLM by up to 87\% and thus allows for substantial cost reductions while maintaining competitive performance.

Agentic NL2SQL to Reduce Computational Costs

TL;DR

This work tackles the high token and cost burden of NL2SQL when operating over large, multi-database corpora by introducing the Datalake Agent, an interactive framework that employs information acquisition, iterative refinement, and query formulation to selectively retrieve only necessary metadata. By comparing against a direct prompt baseline, the authors show that the Datalake Agent can reduce token usage by up to 87% while maintaining competitive accuracy on a 100-task, 23-database benchmark, including 319 tables. The approach scales better with data volume, offering substantial cost reductions in practical enterprise settings. Limitations include potential infinite reasoning loops and evaluation restricted to GPT-4 Mini, with future work aiming to broaden datasets and LLM support and to enhance loop mitigation strategies.

Abstract

Translating natural language queries into SQL queries (NL2SQL or Text-to-SQL) has recently been empowered by large language models (LLMs). Using LLMs to perform NL2SQL methods on a large collection of SQL databases necessitates processing large quantities of meta-information about the databases, which in turn results in lengthy prompts with many tokens and high processing costs. To address this challenge, we introduce Datalake Agent, an agentic system designed to enable an LLM to solve NL2SQL tasks more efficiently. Instead of utilizing direct solvers for NL2SQL that call the LLM once with all meta-information in the prompt, the Datalake Agent employs an interactive loop to reduce the utilized meta-information. Within the loop, the LLM is used in a reasoning framework that selectively requests only the necessary information to solve a table question answering task. We evaluate the Datalake Agent on a collection of 23 databases with 100 table question answering tasks. The Datalake Agent reduces the tokens used by the LLM by up to 87\% and thus allows for substantial cost reductions while maintaining competitive performance.
Paper Structure (11 sections, 11 figures)

This paper contains 11 sections, 11 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Experimental Setup
  5. Results
  6. Conclusion
  7. The Datalake Agent
  8. Input Tokens
  9. Performance
  10. Example Tasks for Evaluation
  11. Handling of Infinite Reasoning Loops

Figures (11)

  • Figure 1: Performance trend of Direct Solver and Datalake Agent.
  • Figure 2: Trend of average amount of tokens required for solving one task.
  • Figure 3: Comparison of costs for 1000 tasks across methods and LLMs.
  • Figure 4: Solution process of a Table Question Answering Task using the Datalake Agent
  • Figure 5: Communication loop before solving a task between model and Datalake Agent
  • ...and 6 more figures