TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problems

TL;DR

This work tackles scalable tabular classification by enhancing TabPFN through Beta, a method that jointly reduces bias and variance. Beta employs encoder-based fine-tuning to align downstream data with the pre-trained TabPFN, uses multiple lightweight encoders with Batch Ensemble to diversify representations and mitigate variance, and applies bootstrapped sampling during inference for robustness, supplemented by ECOC for multiclass handling. Empirical results on over 200 tabular benchmarks demonstrate state-of-the-art or competitive performance, with strong gains on high-dimensional and large-scale datasets and in multiclass settings, all while preserving inference efficiency. The approach offers a practical, parameter-efficient pathway to adapt TabPFN to real-world, complex tabular tasks.

Abstract

TabPFN has emerged as a promising in-context learning model for tabular data, capable of directly predicting the labels of test samples given labeled training examples. It has demonstrated competitive performance, particularly on small-scale classification tasks. However, despite its effectiveness, TabPFN still requires further refinement in several areas, including handling high-dimensional features, aligning with downstream datasets, and scaling to larger datasets. In this paper, we revisit existing variants of TabPFN and observe that most approaches focus either on reducing bias or variance, often neglecting the need to address the other side, while also increasing inference overhead. To fill this gap, we propose Beta (Bagging and Encoder-based Fine-tuning for TabPFN Adaptation), a novel and effective method designed to minimize both bias and variance. To reduce bias, we introduce a lightweight encoder to better align downstream tasks with the pre-trained TabPFN. By increasing the number of encoders in a lightweight manner, Beta mitigate variance, thereby further improving the model's performance. Additionally, bootstrapped sampling is employed to further reduce the impact of data perturbations on the model, all while maintaining computational efficiency during inference. Our approach enhances TabPFN's ability to handle high-dimensional data and scale to larger datasets. Experimental results on over 200 benchmark classification datasets demonstrate that Beta either outperforms or matches state-of-the-art methods.

Figures (9)

• Figure 1: The prediction process of TabPFN. The training set and test set are concatenated, with the training labels added to the input sequence. TabPFN then performs a single forward pass to generate predictions for all test samples.
• Figure 2: Generalization Error, Bias, and Variance for different methods on the Adult and Bank datasets. The methods shown include TabPFN-1000 (subsample size = 1000), TabPFN, TabPFN-en-16, TabPFN-Bagging, KNN-PFN, TabPFN-finetune, and Ours (Beta). The legend is located in the top-right plot for clarity.
• Figure 3: Overview of the proposed method, Beta, which consists of the fine-tuning stage and the inference stage.
• Figure 4: The critical difference diagrams based on the Wilcoxon-Holm test with a significance level of 0.05 to detect pairwise significance for TALENT datasets with fewer than 10 classes.
• Figure 5: The relative improvement of LocalPFN LocalPFN, MixturePFN MixturePFN, and Beta (Ours) over TabPFN across 173 tabular datasets sorted by dataset size (number of rows). The curves represent smoothed results to better illustrate trends in performance improvement as dataset size increases.