Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Transfer Learning of Tabular Data by Finetuning Large Language Models

Shourav B. Rabbani, Ibna Kowsar, Manar D. Samad

TL;DR

This work tackles the challenge of applying deep learning to tabular data by comparing in-context LLM prompting with end-to-end finetuning of a pretrained language model. The authors implement an end-to-end finetuning framework using DistilGPT2, converting each tabular sample into a textual representation and training a classifier head to produce logits, achieving cross-dataset transfer learning on ten benchmarks. They show that end-to-end LLM finetuning often yields superior or competitive accuracy with significantly lower computational cost than API-based or deep-learning baselines, particularly when the number of features is small (less than ten). The results suggest that text-to-tabular transfer learning is a viable path for data-efficient learning on tabular data, and point to future directions for domain-specific LLMs tailored to tabular knowledge.

Abstract

Despite the artificial intelligence (AI) revolution, deep learning has yet to achieve much success with tabular data due to heterogeneous feature space and limited sample sizes without viable transfer learning. The new era of generative AI, powered by large language models (LLM), brings unprecedented learning opportunities to diverse data and domains. This paper investigates the effectiveness of an LLM application programming interface (API) and transfer learning of LLM in tabular data classification. LLM APIs respond to input text prompts with tokenized data and instructions, whereas transfer learning finetunes an LLM for a target classification task. This paper proposes an end-to-end finetuning of LLM to demonstrate cross-data transfer learning on ten benchmark data sets when large pre-trained tabular data models do not exist to facilitate transfer learning. The proposed LLM finetuning method outperforms state-of-the-art machine and deep learning methods on tabular data with less than ten features - a standard feature size for tabular data sets. The transfer learning approach uses a fraction of the computational cost of other deep learning or API-based solutions while ensuring competitive or superior classification performance.

Transfer Learning of Tabular Data by Finetuning Large Language Models

TL;DR

This work tackles the challenge of applying deep learning to tabular data by comparing in-context LLM prompting with end-to-end finetuning of a pretrained language model. The authors implement an end-to-end finetuning framework using DistilGPT2, converting each tabular sample into a textual representation and training a classifier head to produce logits, achieving cross-dataset transfer learning on ten benchmarks. They show that end-to-end LLM finetuning often yields superior or competitive accuracy with significantly lower computational cost than API-based or deep-learning baselines, particularly when the number of features is small (less than ten). The results suggest that text-to-tabular transfer learning is a viable path for data-efficient learning on tabular data, and point to future directions for domain-specific LLMs tailored to tabular knowledge.

Abstract

Despite the artificial intelligence (AI) revolution, deep learning has yet to achieve much success with tabular data due to heterogeneous feature space and limited sample sizes without viable transfer learning. The new era of generative AI, powered by large language models (LLM), brings unprecedented learning opportunities to diverse data and domains. This paper investigates the effectiveness of an LLM application programming interface (API) and transfer learning of LLM in tabular data classification. LLM APIs respond to input text prompts with tokenized data and instructions, whereas transfer learning finetunes an LLM for a target classification task. This paper proposes an end-to-end finetuning of LLM to demonstrate cross-data transfer learning on ten benchmark data sets when large pre-trained tabular data models do not exist to facilitate transfer learning. The proposed LLM finetuning method outperforms state-of-the-art machine and deep learning methods on tabular data with less than ten features - a standard feature size for tabular data sets. The transfer learning approach uses a fraction of the computational cost of other deep learning or API-based solutions while ensuring competitive or superior classification performance.
Paper Structure (18 sections, 1 equation, 4 figures, 3 tables)

This paper contains 18 sections, 1 equation, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Methods
  4. LLM for tabular data classification
  5. Data serialization
  6. Data tokenization
  7. Large language models
  8. Modeling for classification
  9. Baseline models
  10. Experimental setup
  11. Results
  12. Tabular data sets
  13. Prompt engineering
  14. Model training
  15. Model performance
  16. ...and 3 more sections

Figures (4)

  • Figure 1: Tabular data transfer learning framework using a large language model (LLM). In-context learning using LLM includes a detailed text prompt and is without a classifier head or finetuning step.
  • Figure 2: Text prompt template for in-context learning using FeatLLM Han2024FeatLLM.
  • Figure 3: Text prompt template for transfer learning using DistilGPT2 distill_gpt2.
  • Figure 4: Convergence plots of proposed transfer learning of tabular data using a large language model (LLM)