CASL: Concept-Aligned Sparse Latents for Interpreting Diffusion Models

TL;DR

This work introduces CASL (Concept-Aligned Sparse Latents), a supervised framework that aligns sparse latent dimensions of diffusion models with semantic concepts and introduces the Editing Precision Ratio (EPR), a metric that jointly measures concept specificity and the preservation of unrelated attributes.

Abstract

Internal activations of diffusion models encode rich semantic information, but interpreting such representations remains challenging. While Sparse Autoencoders (SAEs) have shown promise in disentangling latent representations, existing SAE-based methods for diffusion model understanding rely on unsupervised approaches that fail to align sparse features with human-understandable concepts. This limits their ability to provide reliable semantic control over generated images. We introduce CASL (Concept-Aligned Sparse Latents), a supervised framework that aligns sparse latent dimensions of diffusion models with semantic concepts. CASL first trains an SAE on frozen U-Net activations to obtain disentangled latent representations, and then learns a lightweight linear mapping that associates each concept with a small set of relevant latent dimensions. To validate the semantic meaning of these aligned directions, we propose CASL-Steer, a controlled latent intervention that shifts activations along the learned concept axis. Unlike editing methods, CASL-Steer is used solely as a causal probe to reveal how concept-aligned latents influence generated content. We further introduce the Editing Precision Ratio (EPR), a metric that jointly measures concept specificity and the preservation of unrelated attributes. Experiments show that our method achieves superior editing precision and interpretability compared to existing approaches. To the best of our knowledge, this is the first work to achieve supervised alignment between latent representations and semantic concepts in diffusion models.

Figures (23)

• Figure 1: Demonstration of semantic directions in the activation space. Left: Geometric illustration of ideal, entangled, and learned semantic directions in the activation space. Right: scaling the Asyrp direction $\Delta h^{(\text{smile})}$ from $\gamma = -2$ to $\gamma = 2$ increases smiling intensity, but also introduces entangled changes in identity, hairstyle, and gender.
• Figure 2: Overview of our proposed CASL framework. Stage 1 (Concept Disentanglement): A Sparse Autoencoder is trained on U-Net activations to obtain a structured sparse latent representation. Stage 2 (Concept Alignment): A lightweight linear mapping aligns selected latent dimensions with human-defined semantic concepts, producing concept-aligned directions. Stage 3 (CASL-Steer): A controlled latent intervention is applied along the aligned direction to verify its semantic effect, serving as a probing mechanism.
• Figure 3: Attribute editing results of concept-aligned editing. Our method enables concept-aligned editing across diverse attributes and domains. Each column shows a specific attribute (Smiling, Young, Puppy, etc.) with consistent edits across different identities. Results are obtained by traversing sparse latent dimensions learned through supervised alignment.
• Figure 4: Hyperparameter analysis of editing intensity $\alpha$ and top-$k$ dimension selection. (a) Single-dimension editing shows stable EPR across concepts. (b) Multi-dimension editing reduces precision with increasing $\alpha$. (c) Target and non-target changes scale proportionally for $k=1$. (d) Increasing $k$ degrades EPR, confirming sparse editing maximizes interpretability.
• Figure 5: EPR vs. $\alpha$ for top-$k=1$ (all concepts).