Hierarchical Concept-to-Appearance Guidance for Multi-Subject Image Generation

TL;DR

This work tackles multi-subject image generation by introducing Hierarchical Concept-to-Appearance Guidance (CAG), which couples high-level VLM semantic grounding with low-level VAE appearance cues. It adds a VAE dropout training strategy to strengthen reliance on semantic signals and a correspondence-aware masked attention mechanism in the Diffusion Transformer to enforce precise word–region bindings to reference subjects. Through extensive experiments, CAG achieves state-of-the-art results in prompt following and subject consistency, validating the efficacy of explicit concept-to-appearance supervision. The approach demonstrates that hierarchical, interpretable grounding can significantly improve controllability and fidelity in complex, multi-reference image synthesis.

Abstract

Multi-subject image generation aims to synthesize images that faithfully preserve the identities of multiple reference subjects while following textual instructions. However, existing methods often suffer from identity inconsistency and limited compositional control, as they rely on diffusion models to implicitly associate text prompts with reference images. In this work, we propose Hierarchical Concept-to-Appearance Guidance (CAG), a framework that provides explicit, structured supervision from high-level concepts to fine-grained appearances. At the conceptual level, we introduce a VAE dropout training strategy that randomly omits reference VAE features, encouraging the model to rely more on robust semantic signals from a Visual Language Model (VLM) and thereby promoting consistent concept-level generation in the absence of complete appearance cues. At the appearance level, we integrate the VLM-derived correspondences into a correspondence-aware masked attention module within the Diffusion Transformer (DiT). This module restricts each text token to attend only to its matched reference regions, ensuring precise attribute binding and reliable multi-subject composition. Extensive experiments demonstrate that our method achieves state-of-the-art performance on the multi-subject image generation, substantially improving prompt following and subject consistency.

Figures (13)

• Figure 1: Our multi-character, scene-referenced generation examples show that the outputs not only preserve consistency across characters and scenes, but also faithfully follow the specified action prompts (in bold).
• Figure 2: Qualitative comparison of OmniGen2 OmniGen2 under inference with and without using VAE features from reference images.
• Figure 3: Overview of CAG.Top: The overall training pipeline of our CAG framework, which integrates the proposed VAE Dropout strategy and the Correspondence-Aware Masked Attention mechanism. Bottom: Illustration of the proposed Correspondence-Aware Masked Attention, where text tokens are explicitly matched to their associated regions in the reference images, enabling fine-grained and identity-consistent multi-subject generation.
• Figure 4: Qualitative comparison with different methods on multi-subject driven image generation.
• Figure 5: Qualitative comparison under inference without using VAE features from reference images. w/o VAE dropout denotes the model trained without the VAE dropout strategy and evaluated without reference-image VAE features, while w/ VAE dropout denotes the model trained with the VAE dropout strategy under the same inference setting.