XVerse: Consistent Multi-Subject Control of Identity and Semantic Attributes via DiT Modulation

TL;DR

XVerse tackles the challenge of consistent multi-subject control in diffusion-based text-to-image generation. It introduces a tuning-free approach that learns subject-specific offsets in the text-stream modulation of Diffusion Transformers, conditioned by reference images, and augments details with VAE features. The method reduces artifacts and attribute entanglement while enabling per-subject identity and semantic control across poses, lighting, and style, demonstrated on XVerseBench with strong quantitative and qualitative results. The framework combines a cross-image conditioning mechanism with regularizations to disentangle subjects and preserve editability, offering robust generalization to complex scenes.

Abstract

Achieving fine-grained control over subject identity and semantic attributes (pose, style, lighting) in text-to-image generation, particularly for multiple subjects, often undermines the editability and coherence of Diffusion Transformers (DiTs). Many approaches introduce artifacts or suffer from attribute entanglement. To overcome these challenges, we propose a novel multi-subject controlled generation model XVerse. By transforming reference images into offsets for token-specific text-stream modulation, XVerse allows for precise and independent control for specific subject without disrupting image latents or features. Consequently, XVerse offers high-fidelity, editable multi-subject image synthesis with robust control over individual subject characteristics and semantic attributes. This advancement significantly improves personalized and complex scene generation capabilities.

Figures (11)

• Figure 1: XVerse enables single/multi-subject personalization and the additional control of semantic attributes such as pose, style, and lighting. Input conditions are highlighted with red dots.
• Figure 2: Overview of the XVerse framework. The reference images are processed by a T-Mod Resampler and subsequently injected into the per-token modulation adapter. Additionally, to supplement image details, the VAE-encoded features of the reference image are also utilized as input to the single block of DiTs.
• Figure 3: Training Data Construction Pipeline.
• Figure 4: Data distribution and samples for XVerseBench. XVerseBench includes evaluations of single-subject, dual-subject, and triple-subject controlled image generation. The figure also illustrates the number of test samples allocated to each category.
• Figure 5: Qualitative comparison with different methods on XVerseBench.