Concept-Guided Interpretability via Neural Chunking

TL;DR

This work reframes interpretability by proposing the Reflection Hypothesis: neural networks reflect the regularities of their training data in their internal dynamics. It introduces three architecture-agnostic methods—DSC, PA, and UCD—to extract recurring, concept-encoding neural chunks from high-dimensional population activity, enabling both concrete and abstract concept discovery. Empirical studies in simple RNNs and various large language models show that these chunks align with recurring words, narrative schemas, and syntactic structure, and that causal grafting/freezing of chunks can controllably bias model behavior. The approach offers a cognitively grounded, scalable lens on the inner computations of deep networks, with implications for interpretability, transfer learning, and safety-focused interventions.

Abstract

Neural networks are often described as black boxes, reflecting the significant challenge of understanding their internal workings and interactions. We propose a different perspective that challenges the prevailing view: rather than being inscrutable, neural networks exhibit patterns in their raw population activity that mirror regularities in the training data. We refer to this as the Reflection Hypothesis and provide evidence for this phenomenon in both simple recurrent neural networks (RNNs) and complex large language models (LLMs). Building on this insight, we propose to leverage our cognitive tendency of chunking to segment high-dimensional neural population dynamics into interpretable units that reflect underlying concepts. We propose three methods to extract recurring chunks on a neural population level, complementing each other based on label availability and neural data dimensionality. Discrete sequence chunking (DSC) learns a dictionary of entities in a lower-dimensional neural space; population averaging (PA) extracts recurring entities that correspond to known labels; and unsupervised chunk discovery (UCD) can be used when labels are absent. We demonstrate the effectiveness of these methods in extracting concept-encoding entities agnostic to model architectures. These concepts can be both concrete (words), abstract (POS tags), or structural (narrative schema). Additionally, we show that extracted chunks play a causal role in network behavior, as grafting them leads to controlled and predictable changes in the model's behavior. Our work points to a new direction for interpretability, one that harnesses both cognitive principles and the structure of naturalistic data to reveal the hidden computations of complex learning systems, gradually transforming them from black boxes into systems we can begin to understand.

Figures (32)

• Figure 1: a. Naturalistic data are highly redundant and compositional, e.g. in language sequences, cognitive systems learn a dictionary of recurring concrete and abstract entities. b - e. In simple networks that contain a small number of neurons, chunking methods can be used to learn a dictionary of frequently recurring population trajectories. This discrete representation can reliably predict the input in the sequence and the network's predictions on the next character.
• Figure 2: a. Hidden states can be grafted to causally change network memory and prediction. b. Embedding grafting enables faster transfer learning of a compositional vocabulary. c. When RNNs learn from sequences that contain context-dependent predictions, training creates extra chunks inside the embedding space. d. Left: The neural population trajectory can be parsed by bigger chunks, which translates to smaller sequence parsing length with an increasing level of hierarchy in the training sequence. The number of extracted chunks increases with the level of hierarchy inside sequences.
• Figure 3: The reflection hypothesis: ANNs' neural activities can be parsed into entities that reflect the structured regularities in training sequences. a-c. Population activity of the initial 5 neurons (unsorted) from an RNN trained to predict the next sequence character, from a sequence that contains repeating ABCD patterns and another that contains ABCD embedding in a default background noise (random E, F, and G); d. Raw neural activity of the first 50 neurons of LLaMA-3 (unsorted) across all layers (32) processing the prompt up to the last token of each highlighted word.
• Figure 4: The identifiability of the presence of extracted chunks evaluated by signal detection measures, and a comparison between SAE. Grafting and freezing word-related population chunks alter the network’s sequence generation.
• Figure 5: Average decoding performance of the top 100 words in English, applying population averaging on a variety of models. Error bar denotes the standard error of the mean.