Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robust Tabular Foundation Models

Matthew Peroni, Franck Le, Vadim Sheinin

TL;DR

This work tackles the gap between tabular data methods and deep learning by enabling robust tabular foundation models (TFMs) trained entirely on synthetic data. It formalizes an adversarial training paradigm over the space of structural causal models (SCMs) that generate synthetic datasets, using an optimality-gap objective relative to strong baselines and a distributionally robust optimization (DRO) approach. The authors introduce Robust Tabular Foundation Models (RTFM), a two-stage algorithm that first searches for high-gap SCM configurations and then trains the TFM on data drawn from a softmax-weighted mix of these configurations. Applied to TabPFN V2, RTFM achieves up to a 6% increase in mean normalized AUC with roughly 90k–100k additional synthetic datasets, demonstrating the practicality and effectiveness of targeted adversarial training with synthetic data in tabular settings.

Abstract

The development of tabular foundation models (TFMs) has accelerated in recent years, showing strong potential to outperform traditional ML methods for structured data. A key finding is that TFMs can be pretrained entirely on synthetic datasets, opening opportunities to design data generators that encourage desirable model properties. Prior work has mainly focused on crafting high-quality priors over generators to improve overall pretraining performance. Our insight is that parameterizing the generator distribution enables an adversarial robustness perspective: during training, we can adapt the generator to emphasize datasets that are particularly challenging for the model. We formalize this by introducing an optimality gap measure, given by the difference between TFM performance and the best achievable performance as estimated by strong baselines such as XGBoost, CatBoost, and Random Forests. Building on this idea, we propose Robust Tabular Foundation Models (RTFM), a model-agnostic adversarial training framework. Applied to the TabPFN V2 classifier, RTFM improves benchmark performance, with up to a 6% increase in mean normalized AUC over the original TabPFN and other baseline algorithms, while requiring less than 100k additional synthetic datasets. These results highlight a promising new direction for targeted adversarial training and fine-tuning of TFMs using synthetic data alone.

Robust Tabular Foundation Models

TL;DR

This work tackles the gap between tabular data methods and deep learning by enabling robust tabular foundation models (TFMs) trained entirely on synthetic data. It formalizes an adversarial training paradigm over the space of structural causal models (SCMs) that generate synthetic datasets, using an optimality-gap objective relative to strong baselines and a distributionally robust optimization (DRO) approach. The authors introduce Robust Tabular Foundation Models (RTFM), a two-stage algorithm that first searches for high-gap SCM configurations and then trains the TFM on data drawn from a softmax-weighted mix of these configurations. Applied to TabPFN V2, RTFM achieves up to a 6% increase in mean normalized AUC with roughly 90k–100k additional synthetic datasets, demonstrating the practicality and effectiveness of targeted adversarial training with synthetic data in tabular settings.

Abstract

The development of tabular foundation models (TFMs) has accelerated in recent years, showing strong potential to outperform traditional ML methods for structured data. A key finding is that TFMs can be pretrained entirely on synthetic datasets, opening opportunities to design data generators that encourage desirable model properties. Prior work has mainly focused on crafting high-quality priors over generators to improve overall pretraining performance. Our insight is that parameterizing the generator distribution enables an adversarial robustness perspective: during training, we can adapt the generator to emphasize datasets that are particularly challenging for the model. We formalize this by introducing an optimality gap measure, given by the difference between TFM performance and the best achievable performance as estimated by strong baselines such as XGBoost, CatBoost, and Random Forests. Building on this idea, we propose Robust Tabular Foundation Models (RTFM), a model-agnostic adversarial training framework. Applied to the TabPFN V2 classifier, RTFM improves benchmark performance, with up to a 6% increase in mean normalized AUC over the original TabPFN and other baseline algorithms, while requiring less than 100k additional synthetic datasets. These results highlight a promising new direction for targeted adversarial training and fine-tuning of TFMs using synthetic data alone.

Paper Structure

This paper contains 9 sections, 1 theorem, 22 equations, 3 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Problem Definition
  3. Robust Tabular Foundation Models
  4. Experiments
  5. Conclusion
  6. RTFM Algorithm
  7. SCM Parameters
  8. Proof of Optimal Maximizer Distribution
  9. Full Experiment Results

Key Result

Proposition 1

Let $\delta =(\delta_{\theta_1}, \dots, \delta_{\theta_n})\in\mathbb{R}^n$ and consider where $Q = (q_1,\dots, q_n)$, $\Delta_n=\{Q\in\mathbb{R}^n:\ q_i\ge0,\ \sum_i q_i=1\}$ and $H(Q)=-\sum_i q_i\log q_i$. If $0<H_{\min}<\log n$ and $\exists i,j$ such that $\delta_{\theta_i} \neq \delta_{\theta_j}$, the optimizer $q^*$ is strictly positive and there exists a unique $\lambda>0$ such th where $\la

Figures (3)

  • Figure 1: Overview of Robust Tabular Foundation Models (RTFM).
  • Figure 2: Maximum estimated optimality gap after each parameter search during model training. After epoch 5, the original TFM is introduced as an additional baseline model.
  • Figure 3: Anatomy of an MLP-based SCM.

Theorems & Definitions (2)

  • Proposition 1
  • proof