Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

What exactly has TabPFN learned to do?

Calvin McCarter

TL;DR

The paper investigates what TabPFN, a Transformer pretrained for in-context learning on tabular data, actually learns by treating it as a function-approximation generator $f_{\mathcal{D}, \\theta}: x \\rightarrow y$ and exploring its behavior across simple 1d and 2d tasks, high-dimensional cancer data, and computer-vision-as-tabular problems. It documents nontrivial inductive biases revealed by the model, such as non-monotone 1d posteriors and Voronoi-like partitioning when ensembles are used, and compares TabPFN against standard baselines on challenging, out-of-domain tasks (e.g., BladderBatch, MNIST/CIFAR-10). The study also analyzes TabPFN-v2, showing potential parity-learning under certain settings but with memory constraints and domain-dependent performance. Taken together, the work argues for modality- and task-specific PFNs, granular evaluation, and further empirical study of how in-context learning in TabPFN encodes priors for small-sample tabular learning.

Abstract

TabPFN [Hollmann et al., 2023], a Transformer model pretrained to perform in-context learning on fresh tabular classification problems, was presented at the last ICLR conference. To better understand its behavior, we treat it as a black-box function approximator generator and observe its generated function approximations on a varied selection of training datasets. Exploring its learned inductive biases in this manner, we observe behavior that is at turns either brilliant or baffling. We conclude this post with thoughts on how these results might inform the development, evaluation, and application of prior-data fitted networks (PFNs) in the future.

What exactly has TabPFN learned to do?

TL;DR

The paper investigates what TabPFN, a Transformer pretrained for in-context learning on tabular data, actually learns by treating it as a function-approximation generator and exploring its behavior across simple 1d and 2d tasks, high-dimensional cancer data, and computer-vision-as-tabular problems. It documents nontrivial inductive biases revealed by the model, such as non-monotone 1d posteriors and Voronoi-like partitioning when ensembles are used, and compares TabPFN against standard baselines on challenging, out-of-domain tasks (e.g., BladderBatch, MNIST/CIFAR-10). The study also analyzes TabPFN-v2, showing potential parity-learning under certain settings but with memory constraints and domain-dependent performance. Taken together, the work argues for modality- and task-specific PFNs, granular evaluation, and further empirical study of how in-context learning in TabPFN encodes priors for small-sample tabular learning.

Abstract

TabPFN [Hollmann et al., 2023], a Transformer model pretrained to perform in-context learning on fresh tabular classification problems, was presented at the last ICLR conference. To better understand its behavior, we treat it as a black-box function approximator generator and observe its generated function approximations on a varied selection of training datasets. Exploring its learned inductive biases in this manner, we observe behavior that is at turns either brilliant or baffling. We conclude this post with thoughts on how these results might inform the development, evaluation, and application of prior-data fitted networks (PFNs) in the future.

Paper Structure

This paper contains 9 sections, 20 figures.

Table of Contents

  1. Introduction
  2. 1d binary classification
  3. 2d multiclass classification
  4. Cancer status classification from high-dimensional gene expression
  5. Computer vision as a tabular classification problem
  6. Closing Thoughts
  7. Analysis of TabPFN-v2
  8. Re-analysis of previous experimental settings
  9. TabPFN-v2 approximately learns the parity function

Figures (20)

  • Figure 1: TabPFN predicted probabilities for simple 1d scenario, with data in red and green.
  • Figure 2: TabPFN predicted probabilities for simple 1d scenario, for varying number of ensembles. Also shown are the predicted probabilities from using inverse-square-root of Euclidean distance within softmax, in orange and lime-green.
  • Figure 3: TabPFN predicted probabilities for simple 1d scenario, but with repeated features.
  • Figure 4: TabPFN predicted probabilities for simple 1d scenario, but when both red and green samples are duplicated.
  • Figure 5: TabPFN predicted probabilities for simple 1d scenario, but when the red sample is duplicated.
  • ...and 15 more figures