Language Model Representations for Efficient Few-Shot Tabular Classification

TL;DR

This work investigates a lightweight paradigm for few-shot tabular classification that directly utilizes semantic embeddings of individual table rows, and demonstrates the viability of reusing existing LLM infrastructure for efficient semantics-driven pathway to reuse existing LLM infrastructure for Web table understanding.

Abstract

The Web is a rich source of structured data in the form of tables, from product catalogs and knowledge bases to scientific datasets. However, the heterogeneity of the structure and semantics of these tables makes it challenging to build a unified method that can effectively leverage the information they contain. Meanwhile, Large language models (LLMs) are becoming an increasingly integral component of web infrastructure for tasks like semantic search. This raises a crucial question: can we leverage these already-deployed LLMs to classify structured data in web-native tables (e.g., product catalogs, knowledge base exports, scientific data portals), avoiding the need for specialized models or extensive retraining? This work investigates a lightweight paradigm, $\textbf{Ta}$ble $\textbf{R}$epresentation with $\textbf{L}$anguage Model~($\textbf{TaRL}$), for few-shot tabular classification that directly utilizes semantic embeddings of individual table rows. We first show that naive application of these embeddings underperforms compared to specialized tabular models. We then demonstrate that their potentials can be unlocked with two key techniques: removing the common component from all embeddings and calibrating the softmax temperature. We show that a simple meta-learner, trained on handcrafted features, can learn to predict an appropriate temperature. This approach achieves performance comparable to state-of-the-art models in low-data regimes ($k \leq 32$) of semantically-rich tables. Our findings demonstrate the viability of reusing existing LLM infrastructure for efficient semantics-driven pathway to reuse existing LLM infrastructure for Web table understanding.

Figures (5)

• Figure 1: Comparison between KNN on raw data and on Llama 3.1 8B's embeddings for $k$-shot classification on CARTE-CLF datasets. The results show that substituting raw features with LLM-derived embeddings alone yields a clear performance gain.
• Figure 2: Comparison of mTaRL against SotA baselines on CARTE-CLF benchmark (11 datasets). mTaRL shows competitive performance against SotA models in lower $k$ settings. Gray line indicates the context limit for TabuLa, and is able to scale much further than the 32-shot limit imposed for TabuLa.
• Figure 3: Balanced accuracy of classifiers built from LLM embeddings as more components are added.
• Figure 4: Comparison of mTaRL against non-semantic baselines on (a) CARTE-CLF and (b) TabArena-CLF benchmarks. Higher is better for balanced accuracy (left) and lower is better for rank (right).
• Figure 5: Comparison of runtime (in seconds) for different models across various $k$ values on CARTE-CLF benchmark. TabuLa-8B is excluded because its runtimes are significantly higher (up to 1,000 times) than the other models.