Interpretable Tabular Foundation Models via In-Context Kernel Regression

TL;DR

This work tackles the opacity of tabular foundation models that rely on in-context learning by introducing KernelICL, which replaces the final prediction head with explicit kernel functions and uses symmetric in-context embeddings. The method yields predictions as transparent weighted averages over training labels, quantified by a perplexity-based inspectability measure, and is supported by a two-dimensional taxonomy that unifies standard kernel methods, neighbor-based approaches, and attention. KernelICL demonstrates competitive accuracy on 55 TALENT datasets (e.g., around 82.88% accuracy, within 0.2% of TabICL) while enabling explicit, interpretable weights over training samples. The combination of distance-based kernels, symmetric embeddings, and tunable sparsity provides a practical accuracy–inspectability trade-off for real-world tabular decision-making tasks.

Abstract

Tabular foundation models like TabPFN and TabICL achieve state-of-the-art performance through in-context learning, yet their architectures remain fundamentally opaque. We introduce KernelICL, a framework to enhance tabular foundation models with quantifiable sample-based interpretability. Building on the insight that in-context learning is akin to kernel regression, we make this mechanism explicit by replacing the final prediction layer with kernel functions (Gaussian, dot-product, kNN) so that every prediction is a transparent weighted average of training labels. We introduce a two-dimensional taxonomy that formally unifies standard kernel methods, modern neighbor-based approaches, and attention mechanisms under a single framework, and quantify inspectability via the perplexity of the weight distribution over training samples. On 55 TALENT benchmark datasets, KernelICL achieves performance on par with existing tabular foundation models, demonstrating that explicit kernel constraints on the final layer enable inspectable predictions without sacrificing performance.

Figures (7)

• Figure 1: Three approaches to tabular prediction (see Section \ref{['sec:method']} for notation). Standard Kernel Regression (dotted): Kernel functions applied to inputs yield transparent predictions but lack learned representations. TabICL or similar (dashed): Foundation model learns powerful embeddings but uses an opaque MLP head. KernelICL (ours, solid): Combines both by fine-tuning the foundation model with explicit kernel form, producing transparent weighted averages with inspectable coefficients.
• Figure 2: Illustration of the KernelICL approach with Gaussian kernel on a 2D synthetic dataset. Left: Input space $x$ showing concentric circles of different classes. Middle: 2D UMAP projection of 512D ICL embedding $h_D(x)$ showing class separation. Right: 1D "sample space", x-axis representing the training samples sorted by first UMAP dimension. Top Row: Training samples colored by class with decision boundary in input space. Other Rows: Weight $w_i$ (circle size for input and embedding space, height for sample space) of each training sample $i$ for 3 example test points (red crosses). Relative perplexity quantifies weight inspectability.
• Figure 3: Decision boundaries on synthetic datasets with 18 added noise features. Standard kernels operate in input space; KernelICL kernels operate in learned embedding space and show noise robustness with test accuracies (bottom right) on par with TabICL.
• Figure 4: Critical difference diagram comparing 14 methods on 55 TALENT binary classification datasets. Methods connected by horizontal bars show no statistically significant difference in accuracy. KernelICL variants form a tight cluster with TabICL and TabPFN, demonstrating that explicit kernel constraints preserve competitive performance.
• Figure 5: Overhead of symmetric embeddings on the embedding time measured on synthetic datasets, approaching a 2x factor in the large sample limit due to duplication of the training samples as both context and queries. Setups running out of memory are skipped.