Towards Understanding Layer Contributions in Tabular In-Context Learning Models

TL;DR

This work investigates how individual layers contribute to predictions in tabular ICL models (TabPFN-v1, TabPFN-v2, TabICL) using the Layers as Painters framework. By conducting layer swapping, repeating, skipping, and probing across 15 TabArena tasks, the authors reveal that later layers do not always add new information and that early layers often dominate, with notable model-specific differences—especially for TabPFN-v2. The results highlight partial redundancy and opportunities for compression and interpretability, while confirming that layer order and inter-layer dynamics differ from those observed in LLMs. These findings motivate the development of lightweight, more interpretable tabular ICL architectures and invite further study into stability across tasks and initializations.

Abstract

Despite the architectural similarities between tabular in-context learning (ICL) models and large language models (LLMs), little is known about how individual layers contribute to tabular prediction. In this paper, we investigate how the latent spaces evolve across layers in tabular ICL models, identify potential redundant layers, and compare these dynamics with those observed in LLMs. We analyze TabPFN and TabICL through the "layers as painters" perspective, finding that only subsets of layers share a common representational language, suggesting structural redundancy and offering opportunities for model compression and improved interpretability.

Figures (13)

• Figure 1: Layer reorganization in tabular ICL models following the "Layers as Painters" framework.
• Figure 2: Average AUC for probing classifiers (logistic regression) trained on embeddings at different layers of the models.
• Figure 3: Impact on average AUC when swapping two layers in the forward pass of the models.
• Figure 4: Impact of repeating the layers of the models.
• Figure 5: Impact on average AUC when skipping a layer during the forward pass.