Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Spectroscopy: Susceptibility Clusters in Language Models

Andrew Gordon, Garrett Baker, George Wang, William Snell, Stan van Wingerden, Daniel Murfet

TL;DR

This work introduces a spectroscopy-inspired framework for neural language model interpretability based on perturbing the data distribution and measuring susceptibilities $\chi_{xy}$ under a localized Gibbs posterior via SGLD. By decomposing susceptibilities into mode contributions, the authors link token-context continuations to underlying data modes and identify clusters as spectral lines that reflect these modes. Applying a conductance-based clustering on 780k susceptibility vectors from Pythia-14M yields 510 interpretable clusters spanning syntax, mathematics, and code patterns, with about 50.8% matching sparse autoencoder features from a larger model, validating cross-method structure. The results persist across model scales (up to 1.4B parameters) and reveal networks of linked clusters, suggesting that the approach captures genuine, scalable internal structure and offers a path toward targeted interventions in training data distributions.

Abstract

Spectroscopy infers the internal structure of physical systems by measuring their response to perturbations. We apply this principle to neural networks: perturbing the data distribution by upweighting a token $y$ in context $x$, we measure the model's response via susceptibilities $χ_{xy}$, which are covariances between component-level observables and the perturbation computed over a localized Gibbs posterior via stochastic gradient Langevin dynamics (SGLD). Theoretically, we show that susceptibilities decompose as a sum over modes of the data distribution, explaining why tokens that follow their contexts "for similar reasons" cluster together in susceptibility space. Empirically, we apply this methodology to Pythia-14M, developing a conductance-based clustering algorithm that identifies 510 interpretable clusters ranging from grammatical patterns to code structure to mathematical notation. Comparing to sparse autoencoders, 50% of our clusters match SAE features, validating that both methods recover similar structure.

Towards Spectroscopy: Susceptibility Clusters in Language Models

TL;DR

This work introduces a spectroscopy-inspired framework for neural language model interpretability based on perturbing the data distribution and measuring susceptibilities under a localized Gibbs posterior via SGLD. By decomposing susceptibilities into mode contributions, the authors link token-context continuations to underlying data modes and identify clusters as spectral lines that reflect these modes. Applying a conductance-based clustering on 780k susceptibility vectors from Pythia-14M yields 510 interpretable clusters spanning syntax, mathematics, and code patterns, with about 50.8% matching sparse autoencoder features from a larger model, validating cross-method structure. The results persist across model scales (up to 1.4B parameters) and reveal networks of linked clusters, suggesting that the approach captures genuine, scalable internal structure and offers a path toward targeted interventions in training data distributions.

Abstract

Spectroscopy infers the internal structure of physical systems by measuring their response to perturbations. We apply this principle to neural networks: perturbing the data distribution by upweighting a token in context , we measure the model's response via susceptibilities , which are covariances between component-level observables and the perturbation computed over a localized Gibbs posterior via stochastic gradient Langevin dynamics (SGLD). Theoretically, we show that susceptibilities decompose as a sum over modes of the data distribution, explaining why tokens that follow their contexts "for similar reasons" cluster together in susceptibility space. Empirically, we apply this methodology to Pythia-14M, developing a conductance-based clustering algorithm that identifies 510 interpretable clusters ranging from grammatical patterns to code structure to mathematical notation. Comparing to sparse autoencoders, 50% of our clusters match SAE features, validating that both methods recover similar structure.
Paper Structure (70 sections, 2 theorems, 42 equations, 13 figures, 8 tables, 1 algorithm)

This paper contains 70 sections, 2 theorems, 42 equations, 13 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Tokens and patterns
  4. Susceptibility Space
  5. Visualization.
  6. The spectrum of a language model
  7. Transform from tokens to modes
  8. Similar sequences have similar susceptibilities
  9. Clusters as spectral lines.
  10. Methodology
  11. Data and susceptibilities
  12. Clustering
  13. Preprocessing.
  14. Graph construction.
  15. Local clustering via conductance.
  16. ...and 55 more sections

Key Result

Lemma B.2

As functions of $w$ we have where $\Phi(w)(x) = \sum_y \ell_{xy}(w) y$ and $\Phi_{\alpha\beta}(w) = \langle \Phi(w), e_{\alpha\beta} \rangle_{\mathscr{H}}$.

Figures (13)

  • Figure 1: The spectrum of Pythia-14M. Representation using UMAP of $780{,}000$ susceptibility vectors $\{ \chi_{x_iy_i} \}_i$ computed for a 14M parameter language model. Shown is one view on a 3D point cloud. Each point represents a token $y$ in context $x$, colored by pattern type (legend above, see \ref{['tab:token-patterns']}). Marked are some external bodies for token sequences where $y$ is a particular token and $x$ varies (e.g. there is a green body for $y = \hbox{\textbackslash n}$). Descriptions of clusters (A)-(L) can be found in Table \ref{['tab:umap-clusters']}. Bottom left inset: zoomed in view of the body of tokens .,,.
  • Figure 2: Double quote token clusters.Left: Full low-dimensional representation of susceptibility vectors computed from Pythia-14M, with points colored by pattern type. Right: Filtered view showing only tokens which, when decoded, contain double quotes (e.g. ", ."). (A) Tokens containing double quotes as closing a quotation (C69, C182, C372, C469). (B) Tokens containing double quotes as opening a quotation (C168, C174).
  • Figure 3: Examples of clusters. For each, the final token is highlighted in a selection of contexts. These clusters were selected for either having high entropy in final or penultimate tokens (blue), completely at random (green), as examples of semantic/meaning based clusters (purple), or as examples of syntactic/grammatical clusters (orange). The size of the cluster is shown in the top right.
  • Figure 4: Percentages of tokens in each dataset which follow a given pattern. Note that not all patterns are mutually exclusive.
  • Figure 5: Conductance of Pythia-14M clusters in larger models. Each point represents one of the 510 clusters, with its conductance measured on the susceptibility distance graph of a larger Pythia model (y-axis) versus Pythia-14M (x-axis). Clusters with low conductance in both models correspond to patterns that persist across scale.
  • ...and 8 more figures

Theorems & Definitions (4)

  • Definition B.1
  • Lemma B.2
  • proof
  • Lemma B.3