T$^2$-RAGBench: Text-and-Table Benchmark for Evaluating Retrieval-Augmented Generation

TL;DR

This work introduces T$^2$-RAGBench, a context-independent benchmark for evaluating retrieval-augmented generation on text-and-table data, drawn from FinQA, ConvFinQA, and TAT-DQA. By reformulating questions to be answerable without access to the exact context, the benchmark enables rigorous assessment of both retrieval and numerical reasoning in unknown-context settings. Across three finance-focused subsets, the study shows that Hybrid BM25 offers the strongest text-and-table retrieval performance, but even state-of-the-art RAG systems exhibit substantial gaps relative to Oracle-context baselines. Ablation studies reveal sensitivities to embedding models and corpus size, and human validation confirms the reformulated questions are largely context-independent. The dataset and code provide a rigorous resource to drive future advances in real-world RAG for multimodal documents.

Abstract

Since many real-world documents combine textual and tabular data, robust Retrieval Augmented Generation (RAG) systems are essential for effectively accessing and analyzing such content to support complex reasoning tasks. Therefore, this paper introduces $\textbf{$T^2$-RAGBench}$, a benchmark comprising $\textbf{23,088}$ question-context-answer triples, designed to evaluate RAG methods on real-world text-and-table data. Unlike typical QA datasets that operate under $\textit{Oracle Context}$ settings, $\textbf{$T^2$-RAGBench}$ challenges models to first retrieve the correct context before conducting numerical reasoning. Existing QA datasets containing text-and-table data typically contain context-dependent questions, which may yield multiple correct answers depending on the provided context. To address this, we transform SOTA datasets into a context-independent format, validated by experts as 91.3% context-independent questions, enabling reliable RAG evaluation. Our comprehensive evaluation identifies $\textit{Hybrid BM25}$ , a technique that combines dense and sparse vectors, as the most effective approach for text-and-table data. However, results demonstrate that $\textbf{$T^2$-RAGBench}$ remains challenging even for SOTA LLMs and RAG methods. Further ablation studies examine the impact of embedding models and corpus size on retrieval performance. $\textbf{$T^2$-RAGBench}$ provides a realistic and rigorous benchmark for existing RAG methods on text-and-table data. Code and dataset are available online: https://github.com/uhh-hcds/g4kmu-paper

Figures (13)

• Figure 1: Overview of current SOTA approaches and dataset example. a) Most benchmarks test models in an Oracle Context setting, chen_finqa_2021chen_convfinqa_2022. Our task (b) targets the unknown-context setting, requiring retrieval from mixed text-tables before answering.
• Figure 2: Number Match comparison per subset and weighted average all between original and reformulated questions from our new benchmark.
• Figure 3: Percentage of context-independent questions (100 per subset, weighted avg overall). $\kappa$ indicates inter-annotator agreement.
• Figure 4: MRR@3 comparison for FinQA, ConvFinQA, and TAT-DQA across five evenly split document subsets.
• Figure 5: Results of the manual error analysis. Percentage of each error category per subset.