Causal Pre-training Under the Fairness Lens: An Empirical Study of TabPFN

TL;DR

This paper investigates the fairness of TabPFN, a tabular foundation model pretrained on millions of synthetic tasks generated from structural causal models and deployed with in-context learning. It compares TabPFN and its fine-tuned variant FT-TabPFN against classic baselines across four standard tabular fairness benchmarks (Heart, Bank, Law, Adult), using DP and EO as fairness metrics and robust evaluation under distribution shifts. The study introduces spurious-correlation stress tests and MNAR covariate-shift scenarios to probe reliance on non-causal shortcuts. Results show TabPFN variants deliver strong predictive accuracy and robustness to spurious correlations, but fairness gains are moderate and inconsistent, particularly under MNAR shifts, signaling that causal pre-training alone is insufficient for reliable algorithmic fairness. The work highlights practical implications for deploying causally-informed tabular models and suggests directions like causal imputation and dynamic fairness regularization to close the fairness gap.

Abstract

Foundation models for tabular data, such as the Tabular Prior-data Fitted Network (TabPFN), are pre-trained on a massive number of synthetic datasets generated by structural causal models (SCM). They leverage in-context learning to offer high predictive accuracy in real-world tasks. However, the fairness properties of these foundational models, which incorporate ideas from causal reasoning during pre-training, remain underexplored. In this work, we conduct a comprehensive empirical evaluation of TabPFN and its fine-tuned variants, assessing predictive performance, fairness, and robustness across varying dataset sizes and distributional shifts. Our results reveal that while TabPFN achieves stronger predictive accuracy compared to baselines and exhibits robustness to spurious correlations, improvements in fairness are moderate and inconsistent, particularly under missing-not-at-random (MNAR) covariate shifts. These findings suggest that the causal pre-training in TabPFN is helpful but insufficient for algorithmic fairness, highlighting implications for deploying TabPFN (and similar) models in practice and the need for further fairness interventions.

Figures (1)

• Figure 1: Performance of different models with varying dataset scale on Adult and Law datasets.