Compositional Inversion for Stable Diffusion Models

TL;DR

This work addresses overfitting in personalized image synthesis via inversion by identifying that inverted concepts tend to occupy out-of-distribution regions in embedding space, hindering compositionality. It introduces a dual-module compositional inversion with semantic inversion that guides embeddings toward a core distribution using anchor attractors and sparse regularization, and spatial inversion that learns layout distributions to regularize cross-attention via location-based penalties. The approach is a post-training technique compatible with existing inversion methods and demonstrates improved text alignment and concept inclusion while maintaining image alignment, validated through quantitative metrics and a user study. The method yields more diverse and balanced concept compositions, enabling robust personalization without extensive retraining, and is complemented by open-source code for broader adoption.

Abstract

Inversion methods, such as Textual Inversion, generate personalized images by incorporating concepts of interest provided by user images. However, existing methods often suffer from overfitting issues, where the dominant presence of inverted concepts leads to the absence of other desired concepts. It stems from the fact that during inversion, the irrelevant semantics in the user images are also encoded, forcing the inverted concepts to occupy locations far from the core distribution in the embedding space. To address this issue, we propose a method that guides the inversion process towards the core distribution for compositional embeddings. Additionally, we introduce a spatial regularization approach to balance the attention on the concepts being composed. Our method is designed as a post-training approach and can be seamlessly integrated with other inversion methods. Experimental results demonstrate the effectiveness of our proposed approach in mitigating the overfitting problem and generating more diverse and balanced compositions of concepts in the synthesized images. The source code is available at https://github.com/zhangxulu1996/Compositional-Inversion.

Figures (14)

• Figure 1: Image synthesis using traditional inversion methods and the proposed compositional inversion: concepts of ${butter}\space{flies}$, $street$, and $spaceship$ are absent when composed with concepts inverted with traditional methods.
• Figure 2: Visualization of compositionality in the embedding space with the evident core distribution and the OOD.
• Figure 3: Development of the relative attention similarity and attention maps of various types of concepts.
• Figure 4: The framework of the proposed method consisting of semantic and spatial inversion components.
• Figure 5: Examples of composing inverted concepts cat* and dog* with pretrained concepts backpack and book.