Realistic Evaluation of TabPFN v2 in Open Environments

TL;DR

The paper tackles the gap between closed-environment success and open-world reliability by evaluating TabPFN v2 on four open-environment challenges using a unified framework. It demonstrates that TabPFN v2, though strong on small datasets and covariate shift detection, struggles with feature decrements and distribution shifts, whereas tree-based methods remain more robust overall. The work contributes a principled open-environment evaluation framework, empirical insights across Emerging New Classes, Decremental/Incremental Features, Changing Data Distributions, and Varied Learning Objectives, and practical recommendations for benchmarks and universal robustness modules. This advances practical tabular learning by highlighting when TabPFN v2 is advantageous and outlining steps to improve robustness in real-world, dynamic settings.

Abstract

Tabular data, owing to its ubiquitous presence in real-world domains, has garnered significant attention in machine learning research. While tree-based models have long dominated tabular machine learning tasks, the recently proposed deep learning model TabPFN v2 has emerged, demonstrating unparalleled performance and scalability potential. Although extensive research has been conducted on TabPFN v2 to further improve performance, the majority of this research remains confined to closed environments, neglecting the challenges that frequently arise in open environments. This raises the question: Can TabPFN v2 maintain good performance in open environments? To this end, we conduct the first comprehensive evaluation of TabPFN v2's adaptability in open environments. We construct a unified evaluation framework covering various real-world challenges and assess the robustness of TabPFN v2 under open environments scenarios using this framework. Empirical results demonstrate that TabPFN v2 shows significant limitations in open environments but is suitable for small-scale, covariate-shifted, and class-balanced tasks. Tree-based models remain the optimal choice for general tabular tasks in open environments. To facilitate future research on open environments challenges, we advocate for open environments tabular benchmarks, multi-metric evaluation, and universal modules to strengthen model robustness. We publicly release our evaluation framework at https://anonymous.4open.science/r/tabpfn-ood-4E65.

Figures (7)

• Figure 1: Open environments challenges in tabular data learning, including emerging new classes, decremental/incremental features, changing data distributions and varied learning objectives.
• Figure 2: Model performance comparison on accuracy between TabPFN v2 and XGBoost on changing data distributions task.
• Figure 3: Model performance on four learning objectives of classification task.
• Figure 4: Mean Balance Accuracy Across 9 Datasets with Distribution Shifts
• Figure 5: Mean AUC Across 9 Datasets with Distribution Shift