Noise Immunity in In-Context Tabular Learning: An Empirical Robustness Analysis of TabPFN's Attention Mechanisms

Abstract

Tabular foundation models (TFMs) such as TabPFN (Tabular Prior-Data Fitted Network) are designed to generalize across heterogeneous tabular datasets through in-context learning (ICL). They perform prediction in a single forward pass conditioned on labeled examples without dataset-specific parameter updates. This paradigm is particularly attractive in industrial domains (e.g., finance and healthcare) where tabular prediction is pervasive. Retraining a bespoke model for each new table can be costly or infeasible in these settings, while data quality issues such as irrelevant predictors, correlated feature groups, and label noise are common. In this paper, we provide strong empirical evidence that TabPFN is highly robust under these sub-optimal conditions. We study TabPFN and its attention mechanisms for binary classification problems with controlled synthetic perturbations that vary: (i) dataset width by injecting random uncorrelated features and by introducing nonlinearly correlated features, (ii) dataset size by increasing the number of training rows, and (iii) label quality by increasing the fraction of mislabeled targets. Beyond predictive performance, we analyze internal signals including attention concentration and attention-based feature ranking metrics. Across these parametric tests, TabPFN is remarkably resilient: ROC-AUC remains high, attention stays structured and sharp, and informative features are highly ranked by attention-based metrics. Qualitative visualizations with attention heatmaps, feature-token embeddings, and SHAP plots further support a consistent pattern across layers in which TabPFN increasingly concentrates on useful features while separating their signals from noise. Together, these findings suggest that TabPFN is a robust TFM capable of maintaining both predictive performance and coherent internal behavior under various scenarios of data imperfections.

Figures (11)

• Figure 1: Feature-wise attention weights heatmaps in layer {3, 6, 9, 12} of TabPFN for the baseline case. In each plot, the first 2 indices represent informative features and last index is the target label.
• Figure 2: Feature-token embeddings in layer {3, 6, 9, 12} of TabPFN for the baseline case. In each plot, the green points represent informative features and gray points are random features.
• Figure 3: SHAP plots of TabPFN for the baseline case. First two indices represent informative features, and the rest are random features. Left: aggregate SHAP values across samples for all features; right: per-sample SHAP values distribution for all features.
• Figure 4: Performance and attention metrics of TabPFN with respect to increasing number of random features. Top left: ROC-AUC; top right: attention proportions and attention ratio; bottom left: KL-divergence with respect to different distributions; bottom right: average attention rank of informative features and their proportion in top-2 attended features.
• Figure 5: Feature-wise attention weights heatmaps in layer 12 of TabPFN. In each plot, the first 2 indices represent informative features and last index is the target label. Left: total number of features = 16; right: total number of features = 256, the first 16 features and the target label are shown.