Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Text2SQL is Not Enough: Unifying AI and Databases with TAG

Asim Biswal, Liana Patel, Siddarth Jha, Amog Kamsetty, Shu Liu, Joseph E. Gonzalez, Carlos Guestrin, Matei Zaharia

TL;DR

This work proposes Table-Augmented Generation (TAG), a unified and general-purpose paradigm for answering natural language questions over databases that represents a wide range of interactions between the LM and database that have been previously unexplored and creates exciting research opportunities.

Abstract

AI systems that serve natural language questions over databases promise to unlock tremendous value. Such systems would allow users to leverage the powerful reasoning and knowledge capabilities of language models (LMs) alongside the scalable computational power of data management systems. These combined capabilities would empower users to ask arbitrary natural language questions over custom data sources. However, existing methods and benchmarks insufficiently explore this setting. Text2SQL methods focus solely on natural language questions that can be expressed in relational algebra, representing a small subset of the questions real users wish to ask. Likewise, Retrieval-Augmented Generation (RAG) considers the limited subset of queries that can be answered with point lookups to one or a few data records within the database. We propose Table-Augmented Generation (TAG), a unified and general-purpose paradigm for answering natural language questions over databases. The TAG model represents a wide range of interactions between the LM and database that have been previously unexplored and creates exciting research opportunities for leveraging the world knowledge and reasoning capabilities of LMs over data. We systematically develop benchmarks to study the TAG problem and find that standard methods answer no more than 20% of queries correctly, confirming the need for further research in this area. We release code for the benchmark at https://github.com/TAG-Research/TAG-Bench.

Text2SQL is Not Enough: Unifying AI and Databases with TAG

TL;DR

This work proposes Table-Augmented Generation (TAG), a unified and general-purpose paradigm for answering natural language questions over databases that represents a wide range of interactions between the LM and database that have been previously unexplored and creates exciting research opportunities.

Abstract

AI systems that serve natural language questions over databases promise to unlock tremendous value. Such systems would allow users to leverage the powerful reasoning and knowledge capabilities of language models (LMs) alongside the scalable computational power of data management systems. These combined capabilities would empower users to ask arbitrary natural language questions over custom data sources. However, existing methods and benchmarks insufficiently explore this setting. Text2SQL methods focus solely on natural language questions that can be expressed in relational algebra, representing a small subset of the questions real users wish to ask. Likewise, Retrieval-Augmented Generation (RAG) considers the limited subset of queries that can be answered with point lookups to one or a few data records within the database. We propose Table-Augmented Generation (TAG), a unified and general-purpose paradigm for answering natural language questions over databases. The TAG model represents a wide range of interactions between the LM and database that have been previously unexplored and creates exciting research opportunities for leveraging the world knowledge and reasoning capabilities of LMs over data. We systematically develop benchmarks to study the TAG problem and find that standard methods answer no more than 20% of queries correctly, confirming the need for further research in this area. We release code for the benchmark at https://github.com/TAG-Research/TAG-Bench.
Paper Structure (18 sections, 1 equation, 2 figures, 2 tables)

This paper contains 18 sections, 1 equation, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. The TAG Model
  3. Query Synthesis
  4. Query Execution
  5. Answer Generation
  6. TAG Design Space
  7. Evaluation
  8. Benchmark Methodology
  9. Baselines
  10. Results
  11. Related Work
  12. Conclusion
  13. Acknowledgements
  14. Sample Queries
  15. LM Prompts
  16. ...and 3 more sections

Figures (2)

  • Figure 1: An example TAG implementation for answering the user's natural language question over a table about movies. The TAG pipeline proceeds in three stages: query synthesis, query execution, and answer generation
  • Figure 2: Example Aggregation Results: The RAG baseline provides an incomplete answer to the query while Text2SQL + LM fails to answer the question using any data from the DB. The Hand-written TAG baseline provides the most thorough answer, synthesizing data from the DB and its own world knowledge.