GAMformer: Bridging Tabular Foundation Models and Interpretable Machine Learning

TL;DR

GAMformer addresses the interpretability gap in tabular foundation models by delivering Generalized Additive Model (GAM) shape functions in a single forward pass via in-context learning. Trained exclusively on synthetic priors, it estimates discrete, non-parametric shape functions and sums them to produce predictions, enabling additive interpretability without iterative fitting. Empirically, GAMformer achieves competitive accuracy with leading GAMs across classification benchmarks and demonstrates meaningful, domain-relevant interpretability in ICU mortality and housing datasets, while offering potential for extension to higher-order interactions and causal modeling. This work advances safe, transparent tabular modeling with foundation-model-inspired efficiency, particularly relevant for safety-critical applications and regulatory contexts.

Abstract

While interpretability is crucial for machine learning applications in safety-critical domains and for regulatory compliance, existing tabular foundation models like TabPFN lack transparency. Generalized Additive Models (GAMs) provide the needed interpretability through their additive structure, but traditional GAM methods rely on iterative learning algorithms (such as splines, boosted trees, or neural networks) that are fundamentally incompatible with the in-context learning paradigm of foundation models. In this paper, we introduce GAMformer, the first tabular foundation model for GAMs that bridges the gap between the power of foundation models and the interpretability requirements of critical real-world applications. GAMformer estimates GAM shape functions in a single forward pass using in-context learning, representing a significant departure from conventional iterative approaches. Building on previous research on tabular foundation models, we train GAMformer exclusively on synthetically generated tables to prevent data leakage. Our experiments demonstrate that GAMformer performs comparably to other leading GAMs across various classification benchmarks.

Figures (15)

• Figure 1: GAMformer's forward pass on a new dataset with three features ($x_1$, $x_2$, $x_3$) and label $y$ and two training and one test-data point: (1) For each training data point, we bin all features, one-hot encode them, embed the resulting vectors and add the label of the data point. (2) We alternate between applying attention across the features and the data points, allowing us to handle varying numbers of each. (3) We decode per-feature shape functions using a shared MLP decoder. (4) We infer the prediction for test data points by looking up and adding each feature's shape function value (red bins) forming the GAM prediction. (5) Finally, we compute the loss based on the prediction allowing the end-to-end training of the shape function estimation based on (in our case, synthetic) training datasets.
• Figure 2: Shape functions derived from GAMformer and EBMs applied to the linear, binary classification problem $f(x_1, x_2, x_3) = \mathbb{I}((-1) x_1 + 0 x_2 + x_3 > 0)$. We use a twin y axis with GAMformer and EBM on left and right, respectively. All models shown result from a 30-fold cross-validation over 1500 data points.
• Figure 3: Robustness analysis (linear, binary classification): GAMformer consistently outperforms or matches EBM across various sample sizes and feature counts, showcasing its efficiency.
• Figure 4: (a) Shape functions derived from GAMformer and EBMs applied to the polynomial, binary classification problem $f(x_1, x_2) = \mathbb{I}(x_1 + x_2^2 > 0)$. All models result from a 30-fold cross-validation over 1500 data points are shown.
• Figure 5: Visualization of classification boundaries for various baseline classifiers and GAMformer on scikit-learn dataset examples scikit-learn, in the lower right corner we show the ROC-AUC on a validation split. Due to the absence of higher-order feature interaction terms in both GAMformer and EBM (main effects), the 'XOR' dataset (bottom row) is not accurately modeled by them. Incorporating second-order effects solves the problem (EBM$^*$ and GAMformer$^*$).