Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Supervised to Generative: A Novel Paradigm for Tabular Deep Learning with Large Language Models

Xumeng Wen, Han Zhang, Shun Zheng, Wei Xu, Jiang Bian

TL;DR

Generative Tabular Learning (GTL) introduces an instruction-oriented, generative paradigm to extend large language models (LLMs) to tabular deep learning. By continuing pre-training on tabular data encoded as text prompts with multiple templates, GTL enables zero-shot and in-context learning on diverse classification and regression tasks without fine-tuning. Evaluated on 384 Kaggle datasets with LLaMA-2 backbones, GTL achieves strong few-shot performance and often rivals or surpasses state-of-the-art baselines, including GPT-4 in several classification tasks, while expanding coverage to regression. The work provides a reproducible data-construction pipeline and insights into template effects and scaling, suggesting GTL as a practical path toward universal, instruction-following tabular models and a benchmark resource for future research.

Abstract

Tabular data is foundational to predictive modeling in various crucial industries, including healthcare, finance, retail, sustainability, etc. Despite the progress made in specialized models, there is an increasing demand for universal models that can transfer knowledge, generalize from limited data, and follow human instructions. These are challenges that current tabular deep learning approaches have not fully tackled. Here we introduce Generative Tabular Learning (GTL), a novel framework that integrates the advanced functionalities of large language models (LLMs)-such as prompt-based zero-shot generalization and in-context learning-into tabular deep learning. GTL capitalizes on the pre-training of LLMs on diverse tabular data, enhancing their understanding of domain-specific knowledge, numerical sequences, and statistical dependencies critical for accurate predictions. Our empirical study spans 384 public datasets, rigorously analyzing GTL's convergence and scaling behaviors and assessing the impact of varied data templates. The GTL-enhanced LLaMA-2 model demonstrates superior zero-shot and in-context learning capabilities across numerous classification and regression tasks. Notably, it achieves this without fine-tuning, outperforming traditional methods and rivaling state-of-the-art models like GPT-4 in certain cases. Through GTL, we not only foster a deeper integration of LLMs' sophisticated abilities into tabular data comprehension and application but also offer a new training resource and a test bed for LLMs to enhance their ability to comprehend tabular data. To facilitate reproducible research, we release our code, data, and model checkpoints at https://github.com/microsoft/Industrial-Foundation-Models.

From Supervised to Generative: A Novel Paradigm for Tabular Deep Learning with Large Language Models

TL;DR

Generative Tabular Learning (GTL) introduces an instruction-oriented, generative paradigm to extend large language models (LLMs) to tabular deep learning. By continuing pre-training on tabular data encoded as text prompts with multiple templates, GTL enables zero-shot and in-context learning on diverse classification and regression tasks without fine-tuning. Evaluated on 384 Kaggle datasets with LLaMA-2 backbones, GTL achieves strong few-shot performance and often rivals or surpasses state-of-the-art baselines, including GPT-4 in several classification tasks, while expanding coverage to regression. The work provides a reproducible data-construction pipeline and insights into template effects and scaling, suggesting GTL as a practical path toward universal, instruction-following tabular models and a benchmark resource for future research.

Abstract

Tabular data is foundational to predictive modeling in various crucial industries, including healthcare, finance, retail, sustainability, etc. Despite the progress made in specialized models, there is an increasing demand for universal models that can transfer knowledge, generalize from limited data, and follow human instructions. These are challenges that current tabular deep learning approaches have not fully tackled. Here we introduce Generative Tabular Learning (GTL), a novel framework that integrates the advanced functionalities of large language models (LLMs)-such as prompt-based zero-shot generalization and in-context learning-into tabular deep learning. GTL capitalizes on the pre-training of LLMs on diverse tabular data, enhancing their understanding of domain-specific knowledge, numerical sequences, and statistical dependencies critical for accurate predictions. Our empirical study spans 384 public datasets, rigorously analyzing GTL's convergence and scaling behaviors and assessing the impact of varied data templates. The GTL-enhanced LLaMA-2 model demonstrates superior zero-shot and in-context learning capabilities across numerous classification and regression tasks. Notably, it achieves this without fine-tuning, outperforming traditional methods and rivaling state-of-the-art models like GPT-4 in certain cases. Through GTL, we not only foster a deeper integration of LLMs' sophisticated abilities into tabular data comprehension and application but also offer a new training resource and a test bed for LLMs to enhance their ability to comprehend tabular data. To facilitate reproducible research, we release our code, data, and model checkpoints at https://github.com/microsoft/Industrial-Foundation-Models.
Paper Structure (40 sections, 2 equations, 5 figures, 2 tables)

This paper contains 40 sections, 2 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Predictive Modeling on Tabular Data
  4. When LLMs meet Tabular Deep Learning
  5. Augmented LLMs
  6. Generative Tabular Learning for Large Language Models
  7. Problem Formulation
  8. Notations
  9. Zero-shot Learning
  10. In-context Learning (Few-shot Learning)
  11. Construction of Tabular Data in an Instruction-Oriented Language Format
  12. Learning and Adaptation
  13. Notations for Tokenized Instruction-oriented Language Data
  14. The Objective of GTL
  15. Adaptation to Downstream Tasks.
  16. ...and 25 more sections

Figures (5)

  • Figure 1: An overview of generative tabuar learning for LLMs.
  • Figure 2: An examination of in-domain (on pre-train test sets) and out-of-domain (on holdout data) generalization as GTL optimization progresses, highlighting the zero-shot learning scenario (left), the in-context learning scenario with few in-context examples (middle), and with many examples (right).
  • Figure 3: A study of scaling laws, focusing on the number of data samples per pre-training task (left), the percentage of pre-training tasks (middle), and the model size (right). Both AUROC and NMAE metrics, aggregated from all holdout datasets across different context examples, are represented. A dotted line connects a LLaMA model with its GTL variant for comparison.
  • Figure 4: The influence of template variability on dataset coverage, given a maximum sequence length, and on generalization performance, considering different numbers of in-context examples on holdout datasets.
  • Figure 5: Domain distribution of pre-training and holdout datasets.