HCMA: Hierarchical Cross-model Alignment for Grounded Text-to-Image Generation

TL;DR

HCMA tackles grounded text-to-image generation by enforcing semantic fidelity at both global (caption-to-image) and local (object-to-region) levels within every diffusion step. It introduces dual alignment modules—global C2IA and local O2RA—together with losses $\Omega_t^G$ and $\Omega_t^L$ that iteratively refine latent representations during sampling. Empirical results on MS-COCO 2014 show HCMA achieving substantial gains in FID and CLIP scores over strong baselines, including a $\approx 0.69$ FID improvement over SD-v1.5 and a $\approx 0.029$–$0.032$ CLIP uplift, validating improved global semantics and precise spatial grounding. The method demonstrates practical impact for multi-object scenes with layout constraints and offers code at the provided GitHub repository for reproducibility and further exploration.

Abstract

Text-to-image synthesis has progressed to the point where models can generate visually compelling images from natural language prompts. Yet, existing methods often fail to reconcile high-level semantic fidelity with explicit spatial control, particularly in scenes involving multiple objects, nuanced relations, or complex layouts. To bridge this gap, we propose a Hierarchical Cross-Modal Alignment (HCMA) framework for grounded text-to-image generation. HCMA integrates two alignment modules into each diffusion sampling step: a global module that continuously aligns latent representations with textual descriptions to ensure scene-level coherence, and a local module that employs bounding-box layouts to anchor objects at specified locations, enabling fine-grained spatial control. Extensive experiments on the MS-COCO 2014 validation set show that HCMA surpasses state-of-the-art baselines, achieving a 0.69 improvement in Frechet Inception Distance (FID) and a 0.0295 gain in CLIP Score. These results demonstrate HCMA's effectiveness in faithfully capturing intricate textual semantics while adhering to user-defined spatial constraints, offering a robust solution for semantically grounded image generation. Our code is available at https://github.com/hwang-cs-ime/HCMA.

Figures (4)

• Figure 1: Overview of the HCMA Approach. In each diffusion step, HCMA’s alignment ensures textual coherence globally while enforcing bounding-box layouts locally, thus providing a more robust and controllable framework for grounded text-to-image generation.
• Figure 2: Training Framework of HCMA. Given a text prompt and bounding-box layouts, a global (caption-to-image) module fosters textual coherence while a local (object-to-region) module enforces bounding-box adherence. They guide the diffusion-based training so each step reflects both high-level semantics and region-specific placement.
• Figure 3: Sampling Process of HCMA. During inference, each diffusion step systematically alternates between refining global semantics via the caption-to-image module and strictly enforcing local bounding-box fidelity via the object-to-region module. The final image thus comprehensively unifies the textual description with explicitly defined spatial constraints.
• Figure 4: Visualization comparisons of different methods on the MS-COCO 2014 validation set.