Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Making Sense Of Distributed Representations With Activation Spectroscopy

Kyle Reing, Greg Ver Steeg, Aram Galstyan

TL;DR

This work tackles interpretability under distributed representations by introducing Activation Spectroscopy (ActSpec), which casts a layer-to-output subnetwork as a pseudo-Boolean function and analyzes its Fourier spectrum to identify high-value, non-redundant neuron subsets. It combines a constrained Goldreich-Levin-style search with in-distribution sampling and an explicit redundancy filter to handle the combinatorial search over subsets. The approach is validated on synthetic benchmarks, MNIST, and a transformer-based sentiment model, showing recovery of influential joint-activation patterns and providing interpretable feature sets that traditional attribution methods may miss. ActSpec offers a principled, scalable pathway to quantify distribution in neural representations and to guide interventions, with promising implications for mechanistic interpretability and debugging of complex models.

Abstract

In the study of neural network interpretability, there is growing evidence to suggest that relevant features are encoded across many neurons in a distributed fashion. Making sense of these distributed representations without knowledge of the network's encoding strategy is a combinatorial task that is not guaranteed to be tractable. This work explores one feasible path to both detecting and tracing the joint influence of neurons in a distributed representation. We term this approach Activation Spectroscopy (ActSpec), owing to its analysis of the pseudo-Boolean Fourier spectrum defined over the activation patterns of a network layer. The sub-network defined between a given layer and an output logit is cast as a special class of pseudo-Boolean function. The contributions of each subset of neurons in the specified layer can be quantified through the function's Fourier coefficients. We propose a combinatorial optimization procedure to search for Fourier coefficients that are simultaneously high-valued, and non-redundant. This procedure can be viewed as an extension of the Goldreich-Levin algorithm which incorporates additional problem-specific constraints. The resulting coefficients specify a collection of subsets, which are used to test the degree to which a representation is distributed. We verify our approach in a number of synthetic settings and compare against existing interpretability benchmarks. We conclude with a number of experimental evaluations on an MNIST classifier, and a transformer-based network for sentiment analysis.

Making Sense Of Distributed Representations With Activation Spectroscopy

TL;DR

This work tackles interpretability under distributed representations by introducing Activation Spectroscopy (ActSpec), which casts a layer-to-output subnetwork as a pseudo-Boolean function and analyzes its Fourier spectrum to identify high-value, non-redundant neuron subsets. It combines a constrained Goldreich-Levin-style search with in-distribution sampling and an explicit redundancy filter to handle the combinatorial search over subsets. The approach is validated on synthetic benchmarks, MNIST, and a transformer-based sentiment model, showing recovery of influential joint-activation patterns and providing interpretable feature sets that traditional attribution methods may miss. ActSpec offers a principled, scalable pathway to quantify distribution in neural representations and to guide interventions, with promising implications for mechanistic interpretability and debugging of complex models.

Abstract

In the study of neural network interpretability, there is growing evidence to suggest that relevant features are encoded across many neurons in a distributed fashion. Making sense of these distributed representations without knowledge of the network's encoding strategy is a combinatorial task that is not guaranteed to be tractable. This work explores one feasible path to both detecting and tracing the joint influence of neurons in a distributed representation. We term this approach Activation Spectroscopy (ActSpec), owing to its analysis of the pseudo-Boolean Fourier spectrum defined over the activation patterns of a network layer. The sub-network defined between a given layer and an output logit is cast as a special class of pseudo-Boolean function. The contributions of each subset of neurons in the specified layer can be quantified through the function's Fourier coefficients. We propose a combinatorial optimization procedure to search for Fourier coefficients that are simultaneously high-valued, and non-redundant. This procedure can be viewed as an extension of the Goldreich-Levin algorithm which incorporates additional problem-specific constraints. The resulting coefficients specify a collection of subsets, which are used to test the degree to which a representation is distributed. We verify our approach in a number of synthetic settings and compare against existing interpretability benchmarks. We conclude with a number of experimental evaluations on an MNIST classifier, and a transformer-based network for sentiment analysis.
Paper Structure (16 sections, 2 theorems, 4 equations, 4 figures, 2 tables)

This paper contains 16 sections, 2 theorems, 4 equations, 4 figures, 2 tables.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. PREREQUISITES
  4. Pseudo-Boolean Functions
  5. Goldreich-Levin Algorithm
  6. METHODS
  7. Out-of-distribution Sampling
  8. Fourier Coefficient Redundancy
  9. EXPERIMENTS
  10. Multi-tiered Function
  11. Spurious Constant Variables
  12. Spurious Noise Variables
  13. MNIST Input Layer
  14. MNIST Intermediate Layer
  15. Sentiment Analysis
  16. ...and 1 more sections

Key Result

Theorem 3.1

Every function $f:\{-1,1\}^n \rightarrow \mathbb{R}$ can be uniquely expressed as a multilinear polynomial, This expression is called the Fourier expansion of $f$, and the real number $\hat{f}(S)$ is called the Fourier coefficient of $f$ on $S$.

Figures (4)

  • Figure 1: Showcasing redundant variables, where a large inner product between variables leads to identical Fourier coefficients
  • Figure 2: Top Row: Experiments on samples from two classes. Bottom Row: Experiments on all of MNIST. Left: ActSpec (red pixels) alongside redundancy (orange pixels). Right: Attribution methods for the same task.
  • Figure 3: Left: Size of Subsets returned by ActSpec as a function of Dropout regularization. Right: Amount of Redundancy present as a function of Dropout regularization.
  • Figure 4: Hypergraph representations of sets output by ActSpec for the intermediate representations of an MNIST network with and without Dropout regularization

Theorems & Definitions (2)

  • Theorem 3.1: Fourier Expansion Theorem
  • Theorem 3.2: Goldreich-Levin Theorem odonnell