DePass: Unified Feature Attributing by Simple Decomposed Forward Pass

TL;DR

DePass introduces a decomposed forward pass to attribute Transformer behavior in a faithful, additive manner. By freezing attention scores and MLP activations, it enables exact reconstruction of hidden states from decomposed components, enabling multi-granular attribution from tokens to heads, neurons, and subspaces. The framework supports token-wise, component-wise, and subspace-level analyses, with extensive experiments showing improved comprehensiveness and competitive sufficiency over baselines. It provides a practical, training-free tool for mechanistic interpretability with potential applications in subspace analysis and model steering.

Abstract

Attributing the behavior of Transformer models to internal computations is a central challenge in mechanistic interpretability. We introduce DePass, a unified framework for feature attribution based on a single decomposed forward pass. DePass decomposes hidden states into customized additive components, then propagates them with attention scores and MLP's activations fixed. It achieves faithful, fine-grained attribution without requiring auxiliary training. We validate DePass across token-level, model component-level, and subspace-level attribution tasks, demonstrating its effectiveness and fidelity. Our experiments highlight its potential to attribute information flow between arbitrary components of a Transformer model. We hope DePass serves as a foundational tool for broader applications in interpretability.

Figures (27)

• Figure 1: Faithfulness evaluation on Known_1000 and IOI using Llama-2-13b-chat-hf. Our method yields better comprehensiveness and competitive sufficiency.
• Figure 2: Two input-level attribution examples. The top case illustrates contributions to the model prediction "Germany", with higher scores indicating greater influence. The bottom case shows token-wise contributions to the classifier's prediction of the "truthful" label, where more negative scores support the untruthful classification (label 0).
• Figure 3: Performance of our method compared to baseline attribution techniques on Llama-2-7b-chat-hf under various masking strategies. Our approach more accurately identifies critical components, as reflected by the sharper drop in Top-$k$ Masking and stronger performance in Bottom-$k$ Masking.
• Figure 4: t-SNE visualization of token-wise projections onto the language subspace. Distinct clusters indicate strong language-specific structure.
• Figure 5: Faithfulness evaluation using Llama-2-7b-chat-hf. DePass achieves a sharper drop in comprehensiveness and higher sufficiency retention, indicating better identification of faithful tokens.