MURRE: Multi-Hop Table Retrieval with Removal for Open-Domain Text-to-SQL

TL;DR

The paper tackles open-domain text-to-SQL, where the task requires retrieving relevant database tables and generating SQL queries from natural language. It identifies that conventional multi-hop retrieval designed for open-domain QA is ill-suited for this setting, because questions typically contain all necessary information; thus Murre introduces a removal-based multi-hop retrieval, where information from previously retrieved tables is removed from the question to guide the next hop toward unretrieved relevant tables. Murre combines beam-search retrieval with a removal step, using an embedding-based cosine probability for retrieval and LLM-driven removal to maintain focus on novel relevant tables, followed by a two-part scoring mechanism to select the top-N tables for SQL generation. Empirical results on SpiderUnion and BirdUnion show Murre achieves an average 5.7% improvement over prior SOTA, with larger gains on BirdUnion and with controlled trade-offs as the number of retrieved tables grows. The work advances open-domain text-to-SQL by enabling robust multi-hop retrieval that prioritizes question-relevant, non-redundant tables, improving execution accuracy and providing insights into retrieval strategy and efficiency trade-offs in practical applications.

Abstract

The open-domain text-to-SQL task aims to retrieve question-relevant tables from massive databases and generate SQL. However, the performance of current methods is constrained by single-hop retrieval, and existing multi-hop retrieval of open-domain question answering is not directly applicable due to the tendency to retrieve tables similar to the retrieved ones but irrelevant to the question. Since the questions in text-to-SQL usually contain all required information, while previous multi-hop retrieval supplements the questions with retrieved documents. Therefore, we propose the multi-hop table retrieval with removal (MURRE), which removes previously retrieved information from the question to guide the retriever towards unretrieved relevant tables. Our experiments on two open-domain text-to-SQL datasets demonstrate an average improvement of 5.7% over the previous state-of-the-art results.

Figures (7)

• Figure 1: Comparison of multi-hop retrieval with supplementary and removal. The same shapes denote similar tables. Multi-hop retrieval with supplementary retrieves tables similar to those already retrieved, even if they are irrelevant to the question. In contrast, we employ multi-hop retrieval with removal which can successfully retrieve other relevant tables in the second hop.
• Figure 2: An overview of Murre. Each hop consists of: (i) Retrieval: retrieving tables similar to the question; (ii) Removal: removing the retrieved information from the user question and representing the unretrieved information in the tabular format with LLM. We employ the beam search paradigm maintaining multiple retrievals at each hop. The color depth represents the probability that the table is relevant to the question of the hop, and ✓ denotes the relevant table. We demonstrate an example that early stops at the second hop for brevity.
• Figure 3: The proportion of performance improvements achieved by Murre in addressing various limitations, compared to Single-hop. The numbers in parentheses in the legend represent the number of improved examples.
• Figure 4: The complete recall with different beam sizes on SpiderUnion with SGPT-125M.
• Figure 5: The left part is the average rank of relevant tables with different numbers of max hops on the SpiderUnion using Murre. The right part is the proportion of questions that are not early stopped with different hops on the SpiderUnion using Murre.