Tabular Embedding Model (TEM): Finetuning Embedding Models For Tabular RAG Applications

TL;DR

The paper tackles the challenge of applying retrieval-augmented generation to large-scale tabular data by finetuning a lightweight embedding model tailored for tabular fed contexts. It introduces the Tabular Embedding Model (TEM) and a tabular-aware RAG workflow that uses a data analysis agent to select relevant CSVs/SQL tables, aided by a semi-automated, GPT-4–generated training dataset. A two-step finetuning process—new word embedding initialization plus MNR loss optimization on a compact base model—yields substantial performance gains over SOTA text-focused embeddings on financial tabular tasks, even with limited compute (e.g., a local MacBook). The results demonstrate the practicality and scalability of domain-specific tabular RAG, offering a blueprint for improving numeric and structured-data reasoning in real-world applications. This approach advances tabular data analysis by enabling efficient, accurate retrieval and reasoning without requiring large-scale retraining of heavyweight models.

Abstract

In recent times Large Language Models have exhibited tremendous capabilities, especially in the areas of mathematics, code generation and general-purpose reasoning. However for specialized domains especially in applications that require parsing and analyzing large chunks of numeric or tabular data even state-of-the-art (SOTA) models struggle. In this paper, we introduce a new approach to solving domain-specific tabular data analysis tasks by presenting a unique RAG workflow that mitigates the scalability issues of existing tabular LLM solutions. Specifically, we present Tabular Embedding Model (TEM), a novel approach to fine-tune embedding models for tabular Retrieval-Augmentation Generation (RAG) applications. Embedding models form a crucial component in the RAG workflow and even current SOTA embedding models struggle as they are predominantly trained on textual datasets and thus underperform in scenarios involving complex tabular data. The evaluation results showcase that our approach not only outperforms current SOTA embedding models in this domain but also does so with a notably smaller and more efficient model structure.

Figures (5)

• Figure 1: Generic indexing approach vs Tabular Embedding Approach
• Figure 2: Proposed Tabular RAG Workflow
• Figure 3: Custom finetuning dataset
• Figure 4: Semi-automated data generation process via role-play
• Figure 5: Sample Questions generated via role-playing framework