Self-Attention Decomposition For Training Free Diffusion Editing

TL;DR

This work presents a training-free method for semantic editing of diffusion models by performing eigen-decomposition on self-attention weight matrices. By assuming latent whitening and analyzing perturbations in the input latents, the authors reduce the editing direction problem to a Rayleigh quotient whose solution is the principal eigenvector of $C = W_Q^{\top}W_Q + W_K^{\top}W_K + W_V W_V^{\top}$. The resulting editing directions are sample-independent, demonstrate linear, disentangled edits, and significantly reduce editing latency while preserving image fidelity across multiple datasets. This approach leverages pretrained parameters to yield robust semantic control without data generation or fine-tuning, with potential extensions to multi-modal diffusion and cross-attention editing.

Abstract

Diffusion models achieve remarkable fidelity in image synthesis, yet precise control over their outputs for targeted editing remains challenging. A key step toward controllability is to identify interpretable directions in the model's latent representations that correspond to semantic attributes. Existing approaches for finding interpretable directions typically rely on sampling large sets of images or training auxiliary networks, which limits efficiency. We propose an analytical method that derives semantic editing directions directly from the pretrained parameters of diffusion models, requiring neither additional data nor fine-tuning. Our insight is that self-attention weight matrices encode rich structural information about the data distribution learned during training. By computing the eigenvectors of these weight matrices, we obtain robust and interpretable editing directions. Experiments demonstrate that our method produces high-quality edits across multiple datasets while reducing editing time significantly by 60% over current benchmarks.

Figures (4)

• Figure 1: Illustration of editing capabilities with our method. (a) The first row demonstrates the linear editing property as the perturbation strength $\alpha$ is varied. (b) The second row highlights the ability of our method to perform precise edits to facial attributes.
• Figure 2: Our method supports a diverse range of edits with improved disentanglement, enabling precise single-step manipulation of the target attribute beyond existing benchmarks.
• Figure 3: Illustration of how the edit strength parameter $\alpha$ controls the linear change in attribute intensity.
• Figure 4: Effect of our proposed editing method on pretrained models across multiple datasets.