DVI: Disentangling Semantic and Visual Identity for Training-Free Personalized Generation

TL;DR

<3-5 sentence high-level summary> DVI addresses Semantic-Visual Dissonance in tuning-free personalized generation by disentangling identity into semantic and coarse-grained visual streams. It leverages VAE latent statistics to capture the reference image's visual atmosphere and uses a Parameter-Free Feature Modulation to fuse this with semantic embeddings, guided by a Dynamic Temporal Granularity Scheduler. The framework is training-free and demonstrates improved visual consistency and identity preservation, outperforming state-of-the-art methods on IBench benchmarks. This approach enables atmosphere-aware, per-subject generation without fine-tuning, enhancing realism and integration with complex backgrounds.</_3-5 sentence high-level summary>

Abstract

Recent tuning-free identity customization methods achieve high facial fidelity but often overlook visual context, such as lighting, skin texture, and environmental tone. This limitation leads to ``Semantic-Visual Dissonance,'' where accurate facial geometry clashes with the input's unique atmosphere, causing an unnatural ``sticker-like'' effect. We propose **DVI (Disentangled Visual-Identity)**, a zero-shot framework that orthogonally disentangles identity into fine-grained semantic and coarse-grained visual streams. Unlike methods relying solely on semantic vectors, DVI exploits the inherent statistical properties of the VAE latent space, utilizing mean and variance as lightweight descriptors for global visual atmosphere. We introduce a **Parameter-Free Feature Modulation** mechanism that adaptively modulates semantic embeddings with these visual statistics, effectively injecting the reference's ``visual soul'' without training. Furthermore, a **Dynamic Temporal Granularity Scheduler** aligns with the diffusion process, prioritizing visual atmosphere in early denoising stages while refining semantic details later. Extensive experiments demonstrate that DVI significantly enhances visual consistency and atmospheric fidelity without parameter fine-tuning, maintaining robust identity preservation and outperforming state-of-the-art methods in IBench evaluations.

Figures (4)

• Figure 1: Overview of the DVI Framework. We disentangle identity customization into a person ID branch (bottom) for extracting semantic structure and an implicit visual statistics branch (top) leveraging VAE latent variables to extract atmospheric context. These heterogeneous features are synergistically fused via parameter-free feature modulation and regulated by a dynamic scheduler within the MM-DiT backbone.
• Figure 2: Qualitative Comparison: DVI achieves higher editability while ensuring ID consistency. DVI accurately realizes character consistency and complete presentation of atmospheric visual concepts in complex narrative scenes.
• Figure 3: Comparison between removing the visual stream (Left) and the full DVI model (Right). The left image retains ID but the face is too independent and clean, showing obvious Foreground-Background Detachment; the right image successfully injects atmospheric elements like film grain and low-key lighting via the visual stream.
• Figure 4: Comparison between simple feature Concatenation (Left) and DVI Parameter-Free Feature Modulation (PFFM, Right) under Double Exposure style. Concatenation leads to muddy tones and rigid edges; PFFM achieves smooth natural fusion.