Region-Aware Text-to-Image Generation via Hard Binding and Soft Refinement

TL;DR

Region-Aware generation (RAG) tackles the challenge of fine-grained spatial control in text-to-image synthesis without model-specific training. It introduces a two-stage, tuning-free framework: Regional Hard Binding ensures accurate regional content early in denoising, and Regional Soft Refinement promotes harmonious interactions between adjacent regions through cross-attention-based fusion. A novel image repainting capability allows editing of specific regions in a prior generation without extra inpainting. Empirical results show that RAG surpasses tuning-free baselines in attributes, object relationships, and complex layout adherence, with strong qualitative and user-study support, albeit with higher inference time that can be mitigated with acceleration.

Abstract

Regional prompting, or compositional generation, which enables fine-grained spatial control, has gained increasing attention for its practicality in real-world applications. However, previous methods either introduce additional trainable modules, thus only applicable to specific models, or manipulate on score maps within cross-attention layers using attention masks, resulting in limited control strength when the number of regions increases. To handle these limitations, we present RAG, a Regional-Aware text-to-image Generation method conditioned on regional descriptions for precise layout composition. RAG decouple the multi-region generation into two sub-tasks, the construction of individual region (Regional Hard Binding) that ensures the regional prompt is properly executed, and the overall detail refinement (Regional Soft Refinement) over regions that dismiss the visual boundaries and enhance adjacent interactions. Furthermore, RAG novelly makes repainting feasible, where users can modify specific unsatisfied regions in the last generation while keeping all other regions unchanged, without relying on additional inpainting models. Our approach is tuning-free and applicable to other frameworks as an enhancement to the prompt following property. Quantitative and qualitative experiments demonstrate that RAG achieves superior performance over attribute binding and object relationship than previous tuning-free methods.

Figures (7)

• Figure 1: The overall framework of RAG, which divides regional-aware generation into two stages: (1) Regional Hard Binding ensures the proper response of regional prompts by processing each region individually with its fundamental description, and bound at the first $r$ steps to ensure accurate attribute representation and entity localization. (2) Regional Soft Refinement improves the harmony of adjacent region via enabling the interaction of regional local conditions with global image latent within the cross-attention layers at the later $T-r$ steps. The lower left corner shows the definition of spatial region in regional hard binding and regional soft refinement.
• Figure 2: Illustration of Re-painting. Different from regular image-to-image inpainting, repainting inherits from last generation with only the target area re-initialized (upper). Given the parameters in previous generation, users are allowed to specify a target area with a new prompt and repaint the image, without relying on additional inpainting models (bottom).
• Figure 3: Qualitative comparisons on compositional text-to-image generation. From top to bottom, we show 6 examples of different prompts and regions. Compared with previous methods, we demonstrate excellent regional control capabilities.
• Figure 4: Qualitative comparisons on image repainting between our RAG and the state-of-the-art inpainting model BrushNet. Our results are more region-aware with harmonious effect with the surrounding, revealing diverse potential for applications.
• Figure 5: Qualitative analysis of Hard Binding and Soft Refinement. The former ensures the proper responses of each region, while the latter enhances the coherence among regions.