TARGET: Benchmarking Table Retrieval for Generative Tasks

TL;DR

TARGET addresses the challenge of retrieving relevant structured data tables to ground generative models in open-domain queries over tabular data. The paper proposes TARGET, a benchmark examining both retrieval quality and downstream generation across QA, fact verification, and Text-to-SQL tasks, using dense and sparse representations and various metadata inputs. Key findings show that dense embeddings of tables or metadata significantly outperform sparse lexical methods, that metadata quality matters, and that retrieval performance is sensitive to corpus size and table positioning in prompts, with a strong negative link between ground-truth table rank and downstream accuracy. The work demonstrates that retrieval-augmented generation yields substantial improvements over no-context baselines and highlights the need for scalable, robust table retrievers and better evaluation metrics for long-form tabular reasoning.

Abstract

The data landscape is rich with structured data, often of high value to organizations, driving important applications in data analysis and machine learning. Recent progress in representation learning and generative models for such data has led to the development of natural language interfaces to structured data, including those leveraging text-to-SQL. Contextualizing interactions, either through conversational interfaces or agentic components, in structured data through retrieval-augmented generation can provide substantial benefits in the form of freshness, accuracy, and comprehensiveness of answers. The key question is: how do we retrieve the right table(s) for the analytical query or task at hand? To this end, we introduce TARGET: a benchmark for evaluating TAble Retrieval for GEnerative Tasks. With TARGET we analyze the retrieval performance of different retrievers in isolation, as well as their impact on downstream tasks. We find that dense embedding-based retrievers far outperform a BM25 baseline which is less effective than it is for retrieval over unstructured text. We also surface the sensitivity of retrievers across various metadata (e.g., missing table titles), and demonstrate a stark variation of retrieval performance across datasets and tasks. TARGET is available at https://target-benchmark.github.io.

Figures (6)

• Figure 1: Pipeline of "open domain" question answering over tabular data, in which no tables containing "evidence" for the question are provided. Most research considers the "closed domain" setting, and focuses on query interpretation and augmentation, or table reasoning or query generation (e.g. text-to-SQL). With target we intend to stimulate and facilitate research on the critical retrieval step in the "open domain" setting.
• Figure 2: Overview of the target benchmark for evaluating table retrieval methods and downstream generation for various datasets across three downstream tasks: tabular question answering, fact verification, and Text-to-SQL.
• Figure 3: Influence of $k$ on retrieval performance with various baselines on the FeTaQA dataset, confirming the expectation that performance gradually increases with $k$, most significantly for dense embedding approaches.
• Figure 4: Influence of corpus size on retrieval, illustrating the sensitivity in retrieval performance of dense retrievers when the corpus reaches a large scale.
• Figure 5: As the rank of the ground-truth table increases (i.e. the table is deeper in the prompt), the performance on downstream tasks tends to decrease. "N.A." indicates that the ground-truth table was not retrieved.