Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TabPFN Through The Looking Glass: An interpretability study of TabPFN and its internal representations

Aviral Gupta, Armaan Sethi, Dhruv Kumar

TL;DR

The paper addresses the interpretability gap in tabular foundational models by applying targeted linear probing to TabPFN v2, revealing that hidden representations encode linear coefficients and intermediate quantities in a decodable, layer-dependent manner. It demonstrates a multi-step, hierarchical computation where intermediate terms like $a\cdot b$ are recoverable in middle layers, while the final answer emerges early but is then transformed for the output head in later layers using a Logit Lens. The findings argue for structured internal algorithms within TabPFN rather than a mere end-to-end mapping, contributing to more transparent and trustworthy tabular foundation models. This work advances interpretability in tabular FMs and lays groundwork for mechanistic analyses that could improve debugging, trust, and possible steering of such models in practical applications.

Abstract

Tabular foundational models are pre-trained models designed for a wide range of tabular data tasks. They have shown strong performance across domains, yet their internal representations and learned concepts remain poorly understood. This lack of interpretability makes it important to study how these models process and transform input features. In this work, we analyze the information encoded inside the model's hidden representations and examine how these representations evolve across layers. We run a set of probing experiments that test for the presence of linear regression coefficients, intermediate values from complex expressions, and the final answer in early layers. These experiments allow us to reason about the computations the model performs internally. Our results provide evidence that meaningful and structured information is stored inside the representations of tabular foundational models. We observe clear signals that correspond to both intermediate and final quantities involved in the model's prediction process. This gives insight into how the model refines its inputs and how the final output emerges. Our findings contribute to a deeper understanding of the internal mechanics of tabular foundational models. They show that these models encode concrete and interpretable information, which moves us closer to making their decision processes more transparent and trustworthy.

TabPFN Through The Looking Glass: An interpretability study of TabPFN and its internal representations

TL;DR

The paper addresses the interpretability gap in tabular foundational models by applying targeted linear probing to TabPFN v2, revealing that hidden representations encode linear coefficients and intermediate quantities in a decodable, layer-dependent manner. It demonstrates a multi-step, hierarchical computation where intermediate terms like are recoverable in middle layers, while the final answer emerges early but is then transformed for the output head in later layers using a Logit Lens. The findings argue for structured internal algorithms within TabPFN rather than a mere end-to-end mapping, contributing to more transparent and trustworthy tabular foundation models. This work advances interpretability in tabular FMs and lays groundwork for mechanistic analyses that could improve debugging, trust, and possible steering of such models in practical applications.

Abstract

Tabular foundational models are pre-trained models designed for a wide range of tabular data tasks. They have shown strong performance across domains, yet their internal representations and learned concepts remain poorly understood. This lack of interpretability makes it important to study how these models process and transform input features. In this work, we analyze the information encoded inside the model's hidden representations and examine how these representations evolve across layers. We run a set of probing experiments that test for the presence of linear regression coefficients, intermediate values from complex expressions, and the final answer in early layers. These experiments allow us to reason about the computations the model performs internally. Our results provide evidence that meaningful and structured information is stored inside the representations of tabular foundational models. We observe clear signals that correspond to both intermediate and final quantities involved in the model's prediction process. This gives insight into how the model refines its inputs and how the final output emerges. Our findings contribute to a deeper understanding of the internal mechanics of tabular foundational models. They show that these models encode concrete and interpretable information, which moves us closer to making their decision processes more transparent and trustworthy.
Paper Structure (10 sections, 11 equations, 4 figures)

This paper contains 10 sections, 11 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Experiments and Results
  4. Preliminaries
  5. TabPFN v2.
  6. Probing for coefficients
  7. Probing for Intermediaries
  8. Probing for the answer and Logit Lens
  9. Probing for inputs in answer token
  10. Discussion and Future Work

Figures (4)

  • Figure 1: Coefficient probe across layers. Probing $R^2$ and $\mathrm{MSE}$ for coefficient of the linear relationship plotted across different layer activations. The probe $R^2$ sees a sharp increase at layer 6 and drops off at the last layer accompanied by the inverse behaviour in the $\mathrm{MSE}$. High $R^2$ and low $\mathrm{MSE}$ values indicate a better performing probe.
  • Figure 3: Intermediary probe across layers. Probing $R^2$ and $\mathrm{MSE}$ for the intermediary of the expression plotted across different layer activations. Two different expressions and intermediaries displayed. The probe $R^2$ sees a sharp increase at layer 6 and drops off at the last layer accompanied by the inverse behaviour in the $\mathrm{MSE}$. High $R^2$ values and low $\mathrm{MSE}$ indicate a better performing probe.
  • Figure 5: Answer probe across layers. Probing $R^2$ and $\mathrm{MSE}$ for the answer of the expression in the answer token activations plotted across different layer activations. The probe reaches a very high $R^2$ at layer 5 and stays consistent, till an expected drop at the last layer accompanied by the inverse behaviour in the $\mathrm{MSE}$. This indicates the probes effectiveness at extracting the answer earlier in the model through its activations.
  • Figure 7: Input probe across layers. Probing $R^2$ for the inputs and their linear combination in the answer token activations plotted across different layer activations. The probe reaches a very high $R^2$ at layer 5 and stays consistent, till an expected drop at the last layer. This indicates the probes effectiveness at extracting the inputs purely from the answer tokens indicating some kind of copying behavior in the earlier parts of the model.