DreamMatcher: Appearance Matching Self-Attention for Semantically-Consistent Text-to-Image Personalization

TL;DR

DreamMatcher tackles semantically consistent text-to-image personalization by preserving the pre-trained model's structure while injecting reference appearance through appearance matching self-attention (AMA). It introduces semantic matching, a cycle-consistency mask, and semantic matching guidance to align and enrich subject details without retraining, making it a plug-in compatible with existing T2I personalization models. Empirical results show state-of-the-art performance among tuning-free methods and competitive results versus learnable approaches, especially in challenging non-rigid personalization scenarios. The approach offers robust, flexible customization with practical impact for personalized content generation without additional training overhead.

Abstract

The objective of text-to-image (T2I) personalization is to customize a diffusion model to a user-provided reference concept, generating diverse images of the concept aligned with the target prompts. Conventional methods representing the reference concepts using unique text embeddings often fail to accurately mimic the appearance of the reference. To address this, one solution may be explicitly conditioning the reference images into the target denoising process, known as key-value replacement. However, prior works are constrained to local editing since they disrupt the structure path of the pre-trained T2I model. To overcome this, we propose a novel plug-in method, called DreamMatcher, which reformulates T2I personalization as semantic matching. Specifically, DreamMatcher replaces the target values with reference values aligned by semantic matching, while leaving the structure path unchanged to preserve the versatile capability of pre-trained T2I models for generating diverse structures. We also introduce a semantic-consistent masking strategy to isolate the personalized concept from irrelevant regions introduced by the target prompts. Compatible with existing T2I models, DreamMatcher shows significant improvements in complex scenarios. Intensive analyses demonstrate the effectiveness of our approach.

Figures (29)

• Figure 1: DreamMatcher enables semantically-consistent text-to-image (T2I) personalization. Our DreamMatcher is designed to be compatible with any existing T2I personalization models, without requiring additional training or fine-tuning. When integrated with them, DreamMatcher significantly enhances subject appearance, including colors, textures, and shapes, while accurately preserving the target structure as guided by the target prompt.
• Figure 2: Intuition of DreamMatcher: (a) reference image, (b) disrupted target structure path by key-value replacement cao2023masactrlmou2023dragondiffusionchen2023anydoorchen2023fechuang2023kvkhandelwal2023infusion, (c) generated image by (b), (d) target structure path in pre-trained T2I model ruiz2023dreambooth, and (e) generated image by DreamMatcher. For visualization, principal component analysis (PCA) pearson1901liii is applied to the structure path. Key-value replacement disrupts the target structure, yielding sub-optimal personalized results, whereas DreamMatcher better preserves the target structure, producing high-fidelity subject images aligned with target prompts.
• Figure 3: Overall architecture: Given a reference image $I^{X}$, appearance matching self-attention (AMA) aligns the reference appearance into the fixed target structure in self-attention module of pre-trained personalized model $\epsilon_{\theta}$. This is achieved by explictly leveraging reliable semantic matching from reference to target. Furthermore, semantic matching guidance enhances the fine-grained details of the subject in the generated images.
• Figure 4: Comparison between (a) key-value replacement cao2023masactrlmou2023dragondiffusionchen2023anydoorchen2023fechuang2023kvkhandelwal2023infusion and (b) appearance matching self-attention (AMA): AMA aligns the reference appearance path toward the fixed target structure path through explicit semantic matching and consistency modeling.
• Figure 5: Semantic matching and consistency modeling: We leverage internal diffusion features at each time step to find semantic matching $F_{t}^{\mathrm{X\rightarrow Y}}$ between reference and target. Additionally, we compute the confidence map of the predicted matches $U_{t}$ through cycle-consistency.