Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

In Search of Grandmother Cells: Tracing Interpretable Neurons in Tabular Representations

Ricardo Knauer, Erik Rodner

TL;DR

The paper tackles the opacity of foundation models by introducing two information-theoretic metrics to quantify neuronal saliency and selectivity for single concepts, operationalized via the empirical mutual information $\hat{I}(\mathbf{a}_i, \mathbf{b}_j)$ and surprisal. It applies these metrics to TabPFN representations, deriving null distributions under an exchangeability assumption and conducting a Bonferroni-corrected hypothesis test to identify significant neuron-concept pairs. Across ICD-coded concepts in four ED triage tasks, the study finds first evidence that some neurons exhibit moderate saliency and selectivity for high-level concepts, identifiable through knee-point searches on surprisal-selectivity Pareto fronts; baselines like SHAP and optimal probing are Pareto-dominated. Overall, the framework provides a practical, scalable approach to mechanistic interpretability in tabular foundation models and motivates extending these methods to other architectures and modalities.

Abstract

Foundation models are powerful yet often opaque in their decision-making. A topic of continued interest in both neuroscience and artificial intelligence is whether some neurons behave like grandmother cells, i.e., neurons that are inherently interpretable because they exclusively respond to single concepts. In this work, we propose two information-theoretic measures that quantify the neuronal saliency and selectivity for single concepts. We apply these metrics to the representations of TabPFN, a tabular foundation model, and perform a simple search across neuron-concept pairs to find the most salient and selective pair. Our analysis provides the first evidence that some neurons in such models show moderate, statistically significant saliency and selectivity for high-level concepts. These findings suggest that interpretable neurons can emerge naturally and that they can, in some cases, be identified without resorting to more complex interpretability techniques.

In Search of Grandmother Cells: Tracing Interpretable Neurons in Tabular Representations

TL;DR

The paper tackles the opacity of foundation models by introducing two information-theoretic metrics to quantify neuronal saliency and selectivity for single concepts, operationalized via the empirical mutual information and surprisal. It applies these metrics to TabPFN representations, deriving null distributions under an exchangeability assumption and conducting a Bonferroni-corrected hypothesis test to identify significant neuron-concept pairs. Across ICD-coded concepts in four ED triage tasks, the study finds first evidence that some neurons exhibit moderate saliency and selectivity for high-level concepts, identifiable through knee-point searches on surprisal-selectivity Pareto fronts; baselines like SHAP and optimal probing are Pareto-dominated. Overall, the framework provides a practical, scalable approach to mechanistic interpretability in tabular foundation models and motivates extending these methods to other architectures and modalities.

Abstract

Foundation models are powerful yet often opaque in their decision-making. A topic of continued interest in both neuroscience and artificial intelligence is whether some neurons behave like grandmother cells, i.e., neurons that are inherently interpretable because they exclusively respond to single concepts. In this work, we propose two information-theoretic measures that quantify the neuronal saliency and selectivity for single concepts. We apply these metrics to the representations of TabPFN, a tabular foundation model, and perform a simple search across neuron-concept pairs to find the most salient and selective pair. Our analysis provides the first evidence that some neurons in such models show moderate, statistically significant saliency and selectivity for high-level concepts. These findings suggest that interpretable neurons can emerge naturally and that they can, in some cases, be identified without resorting to more complex interpretability techniques.
Paper Structure (13 sections, 6 equations, 2 figures, 1 table)

This paper contains 13 sections, 6 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related work
  3. Quantifying neuronal interpretability
  4. Metric definitions
  5. Significance testing
  6. Experiments
  7. Experimental setup
  8. Experimental results
  9. Limitations
  10. Conclusion
  11. Datasets and preprocessing
  12. Baselines
  13. Pareto front visualization

Figures (2)

  • Figure 1: Surprisal-selectivity Pareto fronts for datasets in which the surprisal and selectivity reached statistical significance ($p < 0.05$). The Pareto fronts obtained from our search procedure, sparse probes via SHAP values covert2021explaininglundberg2017unified, and optimal probing bertsimas2021sparsegurnee2023finding are shown in green, orange, and red, respectively. Larger markers indicate knee points. Both sparse probes via SHAP values and optimal probing are Pareto-dominated by our method. Values $< 0.01$ are omitted for improved readability.
  • Figure 2: Surprisal-selectivity Pareto fronts for datasets in which the surprisal and selectivity did not reach statistical significance ($p \geq 0.05$). The Pareto fronts obtained from our search procedure, sparse probes via SHAP values covert2021explaininglundberg2017unified, and optimal probing bertsimas2021sparsegurnee2023finding are shown in green, orange, and red, respectively. Larger markers indicate knee points. Both sparse probes via SHAP values and optimal probing are Pareto-dominated by our method. Values $< 0.01$ are omitted for improved readability.