Tabular Transfer Learning via Prompting LLMs

TL;DR

This work tackles the problem of transferring knowledge for tabular tasks when labeled data is scarce. It introduces Prompt to Transfer (P2T), a prompting-based framework that uses LLMs to extract transferable knowledge from unlabeled or heterogeneous tabular sources by generating pseudo-demonstrations that mirror the target task. Through extensive zero-shot and few-shot experiments on diverse OpenML and Kaggle datasets, P2T consistently outperforms self-supervised, unsupervised meta-learning, and LIFT-ICL baselines, with further gains when using advanced models like GPT-4. The approach demonstrates a practical, parameter-free transfer mechanism that leverages in-context learning to bridge heterogeneous tabular domains and handle missing values robustly, offering a promising direction for data-efficient tabular learning.

Abstract

Learning with a limited number of labeled data is a central problem in real-world applications of machine learning, as it is often expensive to obtain annotations. To deal with the scarcity of labeled data, transfer learning is a conventional approach; it suggests to learn a transferable knowledge by training a neural network from multiple other sources. In this paper, we investigate transfer learning of tabular tasks, which has been less studied and successful in the literature, compared to other domains, e.g., vision and language. This is because tables are inherently heterogeneous, i.e., they contain different columns and feature spaces, making transfer learning difficult. On the other hand, recent advances in natural language processing suggest that the label scarcity issue can be mitigated by utilizing in-context learning capability of large language models (LLMs). Inspired by this and the fact that LLMs can also process tables within a unified language space, we ask whether LLMs can be effective for tabular transfer learning, in particular, under the scenarios where the source and target datasets are of different format. As a positive answer, we propose a novel tabular transfer learning framework, coined Prompt to Transfer (P2T), that utilizes unlabeled (or heterogeneous) source data with LLMs. Specifically, P2T identifies a column feature in a source dataset that is strongly correlated with a target task feature to create examples relevant to the target task, thus creating pseudo-demonstrations for prompts. Experimental results demonstrate that P2T outperforms previous methods on various tabular learning benchmarks, showing good promise for the important, yet underexplored tabular transfer learning problem. Code is available at https://github.com/jaehyun513/P2T.

Figures (5)

• Figure 1: Overview. P2T creates pseudo-demonstrations to effectively transfer knowledge of source data in an in-context manner. 'Insulin' is used as a prediction target for pseudo-demonstrations because it has the highest correlation with the task feature 'Diabetes.'
• Figure 2: Example prompt for correlation identification on the Diabetes dataset.
• Figure 3: Ablation study that varies the column features used as targets for pseudo-demonstrations.
• Figure 4: Transfer learning with unlabeled data without language descriptions.
• Figure 5: 10-shot regression of P2T using unlabeled samples as transfer source. We report the average test mean squared errors (MSEs) and standard deviation over 5 different seeds. The bold denotes the lowest average score.