Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring Fine-Tuning for Tabular Foundation Models

Aditya Tanna, Pratinav Seth, Mohamed Bouadi, Vinay Kumar Sankarapu

TL;DR

This paper investigates the value of fine-tuning Tabular Foundation Models (TFMs) for structured data by conducting a unified evaluation across six TFMs and three benchmark suites (TALENT, OpenML-CC18, TabZilla) and comparing four adaptation strategies: zero-shot, meta-learning, supervised fine-tuning (SFT), and parameter-efficient fine-tuning (PEFT). It measures performance, calibration, and fairness, revealing that zero-shot TFMs are already strong, while fine-tuning yields limited, model- and data-dependent gains; meta-learning and PEFT offer moderate improvements, whereas SFT frequently degrades accuracy and calibration. The study analyzes how dataset size, imbalance, and dimensionality shape outcomes and provides practical guidelines for when adaptation is advantageous. Overall, the work highlights the importance of aligning adaptation strategy with dataset properties and safety-critical metrics in tabular learning, suggesting zero-shot as a robust default in many scenarios and recommending selective fine-tuning when conditions favor it.

Abstract

Tabular Foundation Models (TFMs) have recently shown strong in-context learning capabilities on structured data, achieving zero-shot performance comparable to traditional machine learning methods. We find that zero-shot TFMs already achieve strong performance, while the benefits of fine-tuning are highly model and data-dependent. Meta-learning and PEFT provide moderate gains under specific conditions, whereas full supervised fine-tuning (SFT) often reduces accuracy or calibration quality. This work presents the first comprehensive study of fine-tuning in TFMs across benchmarks including TALENT, OpenML-CC18, and TabZilla. We compare Zero-Shot, Meta-Learning, Supervised (SFT), and parameter-efficient (PEFT) approaches, analyzing how dataset factors such as imbalance, size, and dimensionality affect outcomes. Our findings cover performance, calibration, and fairness, offering practical guidelines on when fine-tuning is most beneficial and its limitations.

Exploring Fine-Tuning for Tabular Foundation Models

TL;DR

This paper investigates the value of fine-tuning Tabular Foundation Models (TFMs) for structured data by conducting a unified evaluation across six TFMs and three benchmark suites (TALENT, OpenML-CC18, TabZilla) and comparing four adaptation strategies: zero-shot, meta-learning, supervised fine-tuning (SFT), and parameter-efficient fine-tuning (PEFT). It measures performance, calibration, and fairness, revealing that zero-shot TFMs are already strong, while fine-tuning yields limited, model- and data-dependent gains; meta-learning and PEFT offer moderate improvements, whereas SFT frequently degrades accuracy and calibration. The study analyzes how dataset size, imbalance, and dimensionality shape outcomes and provides practical guidelines for when adaptation is advantageous. Overall, the work highlights the importance of aligning adaptation strategy with dataset properties and safety-critical metrics in tabular learning, suggesting zero-shot as a robust default in many scenarios and recommending selective fine-tuning when conditions favor it.

Abstract

Tabular Foundation Models (TFMs) have recently shown strong in-context learning capabilities on structured data, achieving zero-shot performance comparable to traditional machine learning methods. We find that zero-shot TFMs already achieve strong performance, while the benefits of fine-tuning are highly model and data-dependent. Meta-learning and PEFT provide moderate gains under specific conditions, whereas full supervised fine-tuning (SFT) often reduces accuracy or calibration quality. This work presents the first comprehensive study of fine-tuning in TFMs across benchmarks including TALENT, OpenML-CC18, and TabZilla. We compare Zero-Shot, Meta-Learning, Supervised (SFT), and parameter-efficient (PEFT) approaches, analyzing how dataset factors such as imbalance, size, and dimensionality affect outcomes. Our findings cover performance, calibration, and fairness, offering practical guidelines on when fine-tuning is most beneficial and its limitations.
Paper Structure (13 sections, 1 figure, 3 tables)

This paper contains 13 sections, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Models and Benchmarks
  4. Adaptation Strategies
  5. Evaluation Metrics
  6. Research Questions
  7. Experimental Results
  8. Performance Across Dataset Characteristics.
  9. Calibration Evaluation :
  10. Fairness Evaluation :
  11. Key Findings :
  12. Limitations :
  13. Conclusion

Figures (1)

  • Figure 1: High-level overview of models and adaptation strategies. We benchmark classical tabular baselines and multiple tabular foundation models under three regimes—zero-shot inference, parameter-efficient fine-tuning (PEFT), and full supervised fine-tuning—using both supervised fine-tuning and meta-learning variants