Concept Component Analysis: A Principled Approach for Concept Extraction in LLMs

TL;DR

The paper tackles the interpretability challenge of LLMs by providing a principled theory that links model representations to human-interpretable concepts. It introduces Concept Component Analysis (ConCA), which inverts a theoretically motivated linear mixture of latent-concept log-posteriors to recover $\log p(z_i|\mathbf{x})$ in an unsupervised way, and its sparse variant to handle ill-posedness. Empirically, Sparse ConCA outperforms Sparse Autoencoders across multiple model families and scales, showing stronger concept recovery (higher Pearson correlations) and better downstream transfer (few-shot AUC and OOD robustness), with ablations highlighting the importance of LayerNorm and SoftPlus activations. The work provides a theoretically grounded, practical framework for mechanistic interpretability and sets the stage for principled analysis, manipulation, and evaluation of LLM representations, including exploring alternative regularizers and larger benchmark suites.

Abstract

Developing human understandable interpretation of large language models (LLMs) becomes increasingly critical for their deployment in essential domains. Mechanistic interpretability seeks to mitigate the issues through extracts human-interpretable process and concepts from LLMs' activations. Sparse autoencoders (SAEs) have emerged as a popular approach for extracting interpretable and monosemantic concepts by decomposing the LLM internal representations into a dictionary. Despite their empirical progress, SAEs suffer from a fundamental theoretical ambiguity: the well-defined correspondence between LLM representations and human-interpretable concepts remains unclear. This lack of theoretical grounding gives rise to several methodological challenges, including difficulties in principled method design and evaluation criteria. In this work, we show that, under mild assumptions, LLM representations can be approximated as a {linear mixture} of the log-posteriors over concepts given the input context, through the lens of a latent variable model where concepts are treated as latent variables. This motivates a principled framework for concept extraction, namely Concept Component Analysis (ConCA), which aims to recover the log-posterior of each concept from LLM representations through a {unsupervised} linear unmixing process. We explore a specific variant, termed sparse ConCA, which leverages a sparsity prior to address the inherent ill-posedness of the unmixing problem. We implement 12 sparse ConCA variants and demonstrate their ability to extract meaningful concepts across multiple LLMs, offering theory-backed advantages over SAEs.

Figures (18)

• Figure 1: We introduce a latent variable generative model in which observed the input context $\mathbf{x}$ and next token ${y}$, arises from an unknown underlying process over latent concepts $\mathbf{z}$ (Sec. \ref{['sec: setup']}). Under this model, we show that LLM representations $\mathbf{f}_{\mathbf{x}}({\mathbf{x}})$, learned by next-token prediction, can be approximated as a linear mixture of the column vector obtained by stacking log-posteriors of individual latent concepts $\log p(z_i|\mathbf{x})$, conditioned on the input, i.e., $\mathbf{f}_{\mathbf{x}}({\mathbf{x}}) \approx \mathbf{A} \bigl[[\log p(z_1 \mid \mathbf x)]_{z_1};\dots; [\log p(z_\ell \mid \mathbf x)]_{z_\ell} \bigr] + \mathbf{b}$, where $\mathbf{A}$ is a mixing matrix and $\mathbf{b}$ is a constant (Sec. \ref{['sec: linear']}). Motivated by this, we propose Concept Component Analysis (ConCA), a method for linearly unmixing LLM representations $\mathbf{f}_{\mathbf{x}}({\mathbf{x}})$ to recover the log-posteriors over individual latent concepts $\log p(z_i|\mathbf{x})$ (Sec. \ref{['sec: cca']}).
• Figure 2: Ablation study of 4 different normalization methods and 3 activation functions. For GroupNorm, the number of groups (num_groups) is set to 4, 8, 16, or 32. Left axis shows Mean of Pearson Correlation (MPC), right axis shows MSE. Each subplot corresponds to one combination of normalization method and activation function, with each configuration run three times. Across all configurations, ConCA exhibits a remarkably stable correlation regime (MPC $\approx$ 0.72–0.74, excluding BatchNorm), with performance largely insensitive to the sparsity level. Notably, the MSE is affected by the number of groups used in GroupNorm, i.e., fewer groups lead to lower MSE. Overall, LayerNorm emerges as a strong choice, offering consistently good performance in both MPC and MSE.
• Figure 3: Comparison of SAE variants and the proposed ConCA variant across different scales and architectures. The left two shows the results for Pythia family with varying sizes (70m, 1.4b, 2.8b), while the right compares different architectures across multiple models (Gemma-3-1b, Pythia-1.4b, Qwen3-1.7b). Pearson correlation (left axis) and MSE (right axis) are reported for each method. ConCA variants (BatchNorm, GroupNorm, LayerNorm), overall, achieve higher MPC than SAE baselines (Top-k, Batch-Top-k, Panneal), with LayerNorm-ConCA performing best across all settings (approximately 0.70–0.80). SAE methods remain in a lower band (approximately 0.60–0.70) and show weaker gains with model scale. Reconstruction error (MSE) varies substantially across methods: only Panneal (SAE) obtain lower MSE, whereas ConCA maintains strong both MPC and MSE. Overall, the figure highlights that ConCA more reliably extracts concepts, robust across model size and architecture. Full numerical mean and std values, see Sec. \ref{['sec:detailsvalue']}.
• Figure 4: Rank-based fraction of features exhibiting significant changes between counterfactual pairs for SAE and ConCA variants. ConCA shows smaller feature variations, indicating more stable feature responses under counterfactual pairs.
• Figure 5: Test AUC of SAE variants and the proposed ConCA variants under different few-shot settings across 113 datasets (Top), and out-of-distribution tasks across 8 datasets (Bottom), respectively. An example of visualization can be found in Sec. \ref{['app: visual_class']}.

Theorems & Definitions (13)

• Definition 2.1
• Theorem 2.2
• Theorem 3.1: liu2025predict
• Lemma 4.1: Factorization of the Posterior as Conditional Entropy Vanishes
• proof
• Lemma 4.2: Exact Linear Representation of Joint Log Posterior via Full Marginals
• proof
• Lemma 4.3: Expectation difference vanishes under posterior concentration
• proof
• Theorem 2.1