Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Emergence and Evolution of Interpretable Concepts in Diffusion Models

Berk Tinaz, Zalan Fabian, Mahdi Soltanolkotabi

TL;DR

This work tackles the interpretability of text-to-image diffusion models by applying sparse autoencoders to model activations, uncovering a set of human-interpretable concepts and a scalable concept dictionary. By analyzing time-resolved activations across diffusion steps and three U-Net blocks, the authors demonstrate that coarse image composition emerges very early in the reverse diffusion process, even before coherent visuals appear, and that the composition becomes largely finalized by the middle stages. They extend this understanding with causal interventions—both spatially targeted and global—showing that composition can be steered in early stages while style can be edited in middle stages, with final stages exhibiting high editing resistance. The study also introduces a robust vision-based labeling pipeline to map SAE concepts to semantic objects, enabling segmentation-based evaluation and paving the way for time-adaptive editing techniques. Overall, the results illuminate the temporal evolution of visual representations in diffusion models and suggest practical, stage-aware editing strategies for controllable image generation.

Abstract

Diffusion models have become the go-to method for text-to-image generation, producing high-quality images from pure noise. However, the inner workings of diffusion models is still largely a mystery due to their black-box nature and complex, multi-step generation process. Mechanistic interpretability techniques, such as Sparse Autoencoders (SAEs), have been successful in understanding and steering the behavior of large language models at scale. However, the great potential of SAEs has not yet been applied toward gaining insight into the intricate generative process of diffusion models. In this work, we leverage the SAE framework to probe the inner workings of a popular text-to-image diffusion model, and uncover a variety of human-interpretable concepts in its activations. Interestingly, we find that even before the first reverse diffusion step is completed, the final composition of the scene can be predicted surprisingly well by looking at the spatial distribution of activated concepts. Moreover, going beyond correlational analysis, we design intervention techniques aimed at manipulating image composition and style, and demonstrate that (1) in early stages of diffusion image composition can be effectively controlled, (2) in the middle stages image composition is finalized, however stylistic interventions are effective, and (3) in the final stages only minor textural details are subject to change.

Emergence and Evolution of Interpretable Concepts in Diffusion Models

TL;DR

This work tackles the interpretability of text-to-image diffusion models by applying sparse autoencoders to model activations, uncovering a set of human-interpretable concepts and a scalable concept dictionary. By analyzing time-resolved activations across diffusion steps and three U-Net blocks, the authors demonstrate that coarse image composition emerges very early in the reverse diffusion process, even before coherent visuals appear, and that the composition becomes largely finalized by the middle stages. They extend this understanding with causal interventions—both spatially targeted and global—showing that composition can be steered in early stages while style can be edited in middle stages, with final stages exhibiting high editing resistance. The study also introduces a robust vision-based labeling pipeline to map SAE concepts to semantic objects, enabling segmentation-based evaluation and paving the way for time-adaptive editing techniques. Overall, the results illuminate the temporal evolution of visual representations in diffusion models and suggest practical, stage-aware editing strategies for controllable image generation.

Abstract

Diffusion models have become the go-to method for text-to-image generation, producing high-quality images from pure noise. However, the inner workings of diffusion models is still largely a mystery due to their black-box nature and complex, multi-step generation process. Mechanistic interpretability techniques, such as Sparse Autoencoders (SAEs), have been successful in understanding and steering the behavior of large language models at scale. However, the great potential of SAEs has not yet been applied toward gaining insight into the intricate generative process of diffusion models. In this work, we leverage the SAE framework to probe the inner workings of a popular text-to-image diffusion model, and uncover a variety of human-interpretable concepts in its activations. Interestingly, we find that even before the first reverse diffusion step is completed, the final composition of the scene can be predicted surprisingly well by looking at the spatial distribution of activated concepts. Moreover, going beyond correlational analysis, we design intervention techniques aimed at manipulating image composition and style, and demonstrate that (1) in early stages of diffusion image composition can be effectively controlled, (2) in the middle stages image composition is finalized, however stylistic interventions are effective, and (3) in the final stages only minor textural details are subject to change.

Paper Structure

This paper contains 32 sections, 7 equations, 38 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Method
  4. SAE Architecture and Loss
  5. Collecting Model Activations
  6. Extracting Interpretations from SAE Features
  7. Predicting Image Composition from SAE Features
  8. Causal Intervention Techniques
  9. Experiments
  10. Building the Concept Dictionary
  11. Emergence of Image Composition
  12. Effectiveness of Interventions Across Diffusion Time
  13. Early-stage Interventions
  14. Middle-stage Interventions
  15. Final-stage Interventions
  16. ...and 17 more sections

Figures (38)

  • Figure 1: General scene layout emerges during the very first generation step in diffusion models. We generate an image with the prompt Several men walking on the dirt with palm trees in the background. Our interpretability framework can predict segmentation masks for each object mentioned in the input prompt, solely relying on model activations cached during the first diffusion step. At this early stage, the posterior mean predicted by the diffusion model does not contain any visual clues about the final generated image.
  • Figure 2: An overview of our SAE intervention technique. The prompt "An apple in a basket" specifies the necessary concepts but is vague in terms of spatial composition. We intercept activations of the denoising model and edit the latents after encoding them with the SAE. For the features that are spatially located in the bottom-right quadrant, we increase the coefficient corresponding to "apple" concept, while setting it to $0$ for all other features. After the intervention, generated image satisfies the specified layout where all the apples are located in the bottom-right quadrant.
  • Figure 3: Evolution of predicted image composition during the reverse diffusion process, shown through segmentation accuracy (left) and visualizations (right). Features from later time steps become progressively more accurate at predicting the final layout of the image. However, the general image composition emerges as early as the first time step.
  • Figure 4: Early-stage intervention.
  • Figure 5: Middle-stage intervention
  • ...and 33 more figures