Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Observable Propagation: Uncovering Feature Vectors in Transformers

Jacob Dunefsky, Arman Cohan

TL;DR

Observable propagation (ObProp) offers a data-free framework to uncover linear feature vectors in transformer activations by treating task signals as linear functionals on logits. Observables $n$ map logits via measurements $n^T y$, enabling feature vectors to be derived along OV circuits and extended through attention, MLPs, and LayerNorms with first-order linearizations. Theoretical contributions include a LayerNorm invariance result and a coupling coefficient $C(y_1,y_2)$, along with data-free norms and coupling analyses, and empirical demonstrations on gender pronoun prediction and occupational bias that ObProp can outperform data-heavy probes in low-data regimes. The work reveals shared internal features driving different bias-related tasks, discusses debiasing implications, and outlines limitations related to QK circuits, while highlighting the practical impact of a reproducible, data-efficient interpretability toolkit. Overall, ObProp advances mechanistic interpretability by linking task observables to internal feature vectors with minimal data, enabling scalable insights into bias mechanisms and model behavior.

Abstract

A key goal of current mechanistic interpretability research in NLP is to find linear features (also called "feature vectors") for transformers: directions in activation space corresponding to concepts that are used by a given model in its computation. Present state-of-the-art methods for finding linear features require large amounts of labelled data -- both laborious to acquire and computationally expensive to utilize. In this work, we introduce a novel method, called "observable propagation" (in short: ObProp), for finding linear features used by transformer language models in computing a given task -- using almost no data. Our paradigm centers on the concept of "observables", linear functionals corresponding to given tasks. We then introduce a mathematical theory for the analysis of feature vectors, including a similarity metric between feature vectors called the coupling coefficient which estimates the degree to which one feature's output correlates with another's. We use ObProp to perform extensive qualitative investigations into several tasks, including gendered occupational bias, political party prediction, and programming language detection. Our results suggest that ObProp surpasses traditional approaches for finding feature vectors in the low-data regime, and that ObProp can be used to better understand the mechanisms responsible for bias in large language models.

Observable Propagation: Uncovering Feature Vectors in Transformers

TL;DR

Observable propagation (ObProp) offers a data-free framework to uncover linear feature vectors in transformer activations by treating task signals as linear functionals on logits. Observables map logits via measurements , enabling feature vectors to be derived along OV circuits and extended through attention, MLPs, and LayerNorms with first-order linearizations. Theoretical contributions include a LayerNorm invariance result and a coupling coefficient , along with data-free norms and coupling analyses, and empirical demonstrations on gender pronoun prediction and occupational bias that ObProp can outperform data-heavy probes in low-data regimes. The work reveals shared internal features driving different bias-related tasks, discusses debiasing implications, and outlines limitations related to QK circuits, while highlighting the practical impact of a reproducible, data-efficient interpretability toolkit. Overall, ObProp advances mechanistic interpretability by linking task observables to internal feature vectors with minimal data, enabling scalable insights into bias mechanisms and model behavior.

Abstract

A key goal of current mechanistic interpretability research in NLP is to find linear features (also called "feature vectors") for transformers: directions in activation space corresponding to concepts that are used by a given model in its computation. Present state-of-the-art methods for finding linear features require large amounts of labelled data -- both laborious to acquire and computationally expensive to utilize. In this work, we introduce a novel method, called "observable propagation" (in short: ObProp), for finding linear features used by transformer language models in computing a given task -- using almost no data. Our paradigm centers on the concept of "observables", linear functionals corresponding to given tasks. We then introduce a mathematical theory for the analysis of feature vectors, including a similarity metric between feature vectors called the coupling coefficient which estimates the degree to which one feature's output correlates with another's. We use ObProp to perform extensive qualitative investigations into several tasks, including gendered occupational bias, political party prediction, and programming language detection. Our results suggest that ObProp surpasses traditional approaches for finding feature vectors in the low-data regime, and that ObProp can be used to better understand the mechanisms responsible for bias in large language models.
Paper Structure (42 sections, 4 theorems, 28 equations, 3 figures, 6 tables, 1 algorithm)

This paper contains 42 sections, 4 theorems, 28 equations, 3 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Observable Propagation: From Tasks to Feature Vectors
  4. Our Paradigm: Observables
  5. Observable Propagation for Attention Sublayers
  6. General Form: Addressing MLPs and LayerNorms
  7. The Effect of LayerNorms on Feature Vectors
  8. Data-free Analysis of Feature Vectors
  9. Feature vector norms
  10. Coupling coefficients
  11. Experiments
  12. Gendered Pronouns Prediction
  13. Problem setting
  14. Feature vector norms for single attention heads
  15. Cosine similarities and coupling coefficients
  16. ...and 27 more sections

Key Result

Theorem 1

Define $\operatorname{LayerNorm}(x) = \frac{x-(\vec{1}^Tx)\vec{1}}{\left\Vert x-(\vec{1}^Tx)\vec{1} \right\Vert}$, where $\vec{1}$ is the vector of all ones. For a feature vector $n$, define $f(x; n) = n \cdot \operatorname{LayerNorm}(x)$. Define -- that is, $\theta(x;n)$ is the angle between $n$ and $\nabla_x f(x; n)$. Then if $n \sim \mathcal{N}(0,I)$ in $\mathbb{R}^d$, and $d\ge8$ then

Figures (3)

  • Figure 1: The dot product of model activations with (normalized) feature vectors, compared to the model's output for observables. (a) Dot products with the $n_{\text{bias}}$ feature vector for 6::6$\to$9::1$\to$..., versus the model's output with respect to $n_{\text{bias}}$. (b) Dot products with the $n_{\text{subj}}$ feature vector for 6::6$\to$9::1$\to$13::11, versus the model's output w.r.t $n_{\text{subj}}$.
  • Figure 2: Mean norms of activations before each LayerNorm
  • Figure 3: Ratio between LayerNorm weight matrix variances and mean absolute entries of each layer's embeddings

Theorems & Definitions (9)

  • Definition 2.1
  • Theorem 1
  • Theorem 2
  • Lemma 1
  • proof
  • proof
  • Lemma 2
  • proof
  • proof