Interpreting the Weight Space of Customized Diffusion Models

TL;DR

It is found that inverting a single image into this space encodes a realistic identity into a model, even if the input image is out of distribution (e.g., a painting), and these linear properties of the diffusion model weight space extend to other visual concepts.

Abstract

We investigate the space of weights spanned by a large collection of customized diffusion models. We populate this space by creating a dataset of over 60,000 models, each of which is a base model fine-tuned to insert a different person's visual identity. We model the underlying manifold of these weights as a subspace, which we term weights2weights. We demonstrate three immediate applications of this space that result in new diffusion models -- sampling, editing, and inversion. First, sampling a set of weights from this space results in a new model encoding a novel identity. Next, we find linear directions in this space corresponding to semantic edits of the identity (e.g., adding a beard), resulting in a new model with the original identity edited. Finally, we show that inverting a single image into this space encodes a realistic identity into a model, even if the input image is out of distribution (e.g., a painting). We further find that these linear properties of the diffusion model weight space extend to other visual concepts. Our results indicate that the weight space of fine-tuned diffusion models can behave as an interpretable meta-latent space producing new models.

Figures (34)

• Figure 1: weights2weights (w2w) space enables controllable creation of new customized diffusion models. We model a manifold of customized diffusion models as a subspace of weights that encodes different instances of a broad visual concept (e.g., human identities, dog breeds, etc.). This forms a space that supports inverting the subject (e.g., identity) from a single image into a model, editing the subject encoded in the model, and sampling new models that encode new instances of the visual concept. Each of these operations results in a new model that can consistently generate the subject.
• Figure 2: The weights2weights space operates as a meta-latent space. Unlike a traditional generative latent space, w2w space controls the model itself rather than single image instances. New identity-encoding models can be sampled from the space and edited by linearly traversing along semantic directions in weight space. Additionally, a single image can be inverted into the space to produce a model that consistently generates that identity.
• Figure 3: Building weights2weights (w2w) space. We create a dataset of model weights where each model is personalized to a specific identity using low-rank updates (LoRA). These model weights lie on a weights manifold that we further project into a lower-dimensional subspace spanned by its principal components. We train linear classifiers to find disentangled edit directions in this space.
• Figure 4: Identity samples from w2w space. We show the samples from w2w space do not overfit to nearest-neighbor identities, although they incorporate facial attributes from them. The identities are diverse and consistent across generations.
• Figure 5: Qualitative comparison.w2w edits preserve identity while being disentangled and semantically aligned. Concept Sliders conceptsliders tends to exaggerate effects which induces artifacts and degrades identity, while prompting the subject with the desired edit has unexpected effects.