Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Check, Locate, Rectify: A Training-Free Layout Calibration System for Text-to-Image Generation

Biao Gong, Siteng Huang, Yutong Feng, Shiwei Zhang, Yuyuan Li, Yu Liu

TL;DR

This paper tackles the challenge of aligning text-to-image diffusion outputs with complex layout prompts without additional training. It introduces SimM, a training-free, inference-time system that check-locate-rectifies cross-attention activations to correct misplaced objects, guided by target layouts derived from the prompt via dependency parsing and heuristic rules. Activation transfer and multi-map refinement enable precise repositioning with minimal overhead, and SimMBench provides a benchmark for superlative layout descriptions. Experiments show significant gains in layout fidelity over strong baselines while preserving image quality and achieving near real-time latency, highlighting the practical utility of inference-time layout calibration for diffusion models.

Abstract

Diffusion models have recently achieved remarkable progress in generating realistic images. However, challenges remain in accurately understanding and synthesizing the layout requirements in the textual prompts. To align the generated image with layout instructions, we present a training-free layout calibration system SimM that intervenes in the generative process on the fly during inference time. Specifically, following a "check-locate-rectify" pipeline, the system first analyses the prompt to generate the target layout and compares it with the intermediate outputs to automatically detect errors. Then, by moving the located activations and making intra- and inter-map adjustments, the rectification process can be performed with negligible computational overhead. To evaluate SimM over a range of layout requirements, we present a benchmark SimMBench that compensates for the lack of superlative spatial relations in existing datasets. And both quantitative and qualitative results demonstrate the effectiveness of the proposed SimM in calibrating the layout inconsistencies. Our project page is at https://simm-t2i.github.io/SimM.

Check, Locate, Rectify: A Training-Free Layout Calibration System for Text-to-Image Generation

TL;DR

This paper tackles the challenge of aligning text-to-image diffusion outputs with complex layout prompts without additional training. It introduces SimM, a training-free, inference-time system that check-locate-rectifies cross-attention activations to correct misplaced objects, guided by target layouts derived from the prompt via dependency parsing and heuristic rules. Activation transfer and multi-map refinement enable precise repositioning with minimal overhead, and SimMBench provides a benchmark for superlative layout descriptions. Experiments show significant gains in layout fidelity over strong baselines while preserving image quality and achieving near real-time latency, highlighting the practical utility of inference-time layout calibration for diffusion models.

Abstract

Diffusion models have recently achieved remarkable progress in generating realistic images. However, challenges remain in accurately understanding and synthesizing the layout requirements in the textual prompts. To align the generated image with layout instructions, we present a training-free layout calibration system SimM that intervenes in the generative process on the fly during inference time. Specifically, following a "check-locate-rectify" pipeline, the system first analyses the prompt to generate the target layout and compares it with the intermediate outputs to automatically detect errors. Then, by moving the located activations and making intra- and inter-map adjustments, the rectification process can be performed with negligible computational overhead. To evaluate SimM over a range of layout requirements, we present a benchmark SimMBench that compensates for the lack of superlative spatial relations in existing datasets. And both quantitative and qualitative results demonstrate the effectiveness of the proposed SimM in calibrating the layout inconsistencies. Our project page is at https://simm-t2i.github.io/SimM.
Paper Structure (24 sections, 7 equations, 15 figures, 2 tables)

This paper contains 24 sections, 7 equations, 15 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Preliminaries
  4. Check
  5. Locate
  6. Rectify
  7. Experiments
  8. Main Results
  9. Ablation Study
  10. Further Analysis
  11. Related Work
  12. Conclusion
  13. Relation Vocabulary for Checking
  14. Superlative Predefined Positions
  15. An Example of Target Layout Generation
  16. ...and 9 more sections

Figures (15)

  • Figure 1: Given only the input textual prompt, our system can autonomously detect and rectify the layout inconsistencies across various position requirements (a-d), object quantities (e-g), and resolutions (h-i).
  • Figure 2: The "check-locate-rectify" pipeline of SimM, intervening in the generative process on the fly during inference.
  • Figure 3: A detailed illustration of our SimM system.R means repeating, C means concatenating.
  • Figure 4: Examples of multi-resolution image generated by SimM.
  • Figure 5: Qualitative comparisons on DrawBench and SimMBench. Textual prompts require to generate multiple objects with relative and superlative spatial relations.
  • ...and 10 more figures