Latte: Transfering LLMs` Latent-level Knowledge for Few-shot Tabular Learning

TL;DR

Latte addresses the challenge of few-shot tabular learning by transferring latent-level knowledge from large language models (LLMs) during training. It introduces a semantic-aware tabular encoder to inject feature semantics, and a knowledge adapter (GTransformer) to distill task-relevant LLM knowledge into the downstream model, with an unsupervised pre-training stage to leverage unlabeled data. The approach demonstrates state-of-the-art performance across nine real-world datasets, with ablations confirming the superiority of latent-level knowledge over text-level rules and the critical role of each component. Latte also reduces LLM invocation costs by performing a single preprocessing pass to gather task knowledge, enabling scalable, knowledge-guided learning under label scarcity.

Abstract

Few-shot tabular learning, in which machine learning models are trained with a limited amount of labeled data, provides a cost-effective approach to addressing real-world challenges. The advent of Large Language Models (LLMs) has sparked interest in leveraging their pre-trained knowledge for few-shot tabular learning. Despite promising results, existing approaches either rely on test-time knowledge extraction, which introduces undesirable latency, or text-level knowledge, which leads to unreliable feature engineering. To overcome these limitations, we propose Latte, a training-time knowledge extraction framework that transfers the latent prior knowledge within LLMs to optimize a more generalized downstream model. Latte enables general knowledge-guided downstream tabular learning, facilitating the weighted fusion of information across different feature values while reducing the risk of overfitting to limited labeled data. Furthermore, Latte is compatible with existing unsupervised pre-training paradigms and effectively utilizes available unlabeled samples to overcome the performance limitations imposed by an extremely small labeled dataset. Extensive experiments on various few-shot tabular learning benchmarks demonstrate the superior performance of Latte, establishing it as a state-of-the-art approach in this domain

Figures (3)

• Figure 1: Overview of our proposed Latte. This framework begins with the extraction of task-relevant knowledge from LLMs. Knowledge adapters are then employed to guide pre-training on unlabeled data, followed by semantic-aware fine-tuning using few-shot labeled samples. Here we refer to the unsupervised pre-training and the semantic-aware fine-tuning as StageI and StageII, respectively.
• Figure 2: Latte's parameter experiments and representation visualizations. (a-b) Impact of the number of labeled samples and LLM activation layers on Latte's performance in classification and regression tasks. (c-e) Visualization of Latte's learned representations: (c) original representations, and Latte representations trained on (d) 4 and (e) 64 labeled samples.
• Figure 3: The heatmap visualizes the task-related semantic information learned by the model during the meta-learning stage. Higher values represent a greater semantic similarity between the model's learned representations and the semantic knowledge associated with the LLM task.