nanoTabPFN: A Lightweight and Educational Reimplementation of TabPFN

TL;DR

nanoTabPFN introduces a compact, educational reimplementation of TabPFN v2 to address the steep complexity barrier. It preserves core transformer-based ideas (FeatureEncoder, TargetEncoder, TransformerEncoderLayers, Decoder) while using a minimal training loop with pre-generated data to enable rapid, hands-on learning on small datasets. Empirical results show that nanoTabPFN can reach competitive ROC AUC with minutes of pretraining on a single GPU, vastly faster than TabPFN v2 and comparable to traditional baselines in this setting. This work aims to democratize access to tabular foundation models and support fast experimental iteration in teaching and research.

Abstract

Tabular foundation models such as TabPFN have revolutionized predictive machine learning for tabular data. At the same time, the driving factors of this revolution are hard to understand. Existing open-source tabular foundation models are implemented in complicated pipelines boasting over 10,000 lines of code, lack architecture documentation or code quality. In short, the implementations are hard to understand, not beginner-friendly, and complicated to adapt for new experiments. We introduce nanoTabPFN, a simplified and lightweight implementation of the TabPFN v2 architecture and a corresponding training loop that uses pre-generated training data. nanoTabPFN makes tabular foundation models more accessible to students and researchers alike. For example, restricted to a small data setting it achieves a performance comparable to traditional machine learning baselines within one minute of pre-training on a single GPU (160,000x faster than TabPFN v2 pretraining). This eliminated requirement of large computational resources makes pre-training tabular foundation models accessible for educational purposes. Our code is available at https://github.com/automl/nanoTabPFN.

Figures (5)

• Figure 1: nanoTabPFN Architecture. The architecture consists of the FeatureEncoder, which normalizes and embeds the features, the TargetEncoder, which pads up the labels to the full length of rows and embeds the Targets, followed by a repeated TransformerEncoderStack, and the Decoder, which maps the high-dimensional embeddings to our predictions. Adapted from Figure 1 of hollmann2025accurate.
• Figure 2: Transformer Layer. The Transformer Layer consists of Feature Attention, Datapoint Attention and a 2-layer MLP at the end. We have skip connections around each of the attention blocks and the MLP. A Layer Norm follows each skip connection.
• Figure 3: Code example showing how to train nanoTabPFN.
• Figure 4: Within 60 seconds of pretraining on one consumer GPU, nanoTabPFN achieves average ROC AUC on a subset of subsampled datasets from TabArena comparable to traditional machine learning baselines.
• Figure 5: Per dataset results