Layered Rendering Diffusion Model for Controllable Zero-Shot Image Synthesis

TL;DR

This work tackles spatial controllability in text-to-image diffusion by addressing multi-object layout with zero-shot capability. It introduces Layered Rendering Diffusion (LRDiff), a two-stage framework that uses vision guidance to bias per-object denoising in separate layers and fuses results via a global caption. Key contributions include the vision guidance mechanism (constant and dynamic $\delta$ constructions) and the two-stage rendering that avoids fine-tuning while achieving accurate layout adherence. Empirical results on bounding box-to-image and instance mask-to-image tasks show improved alignment and competitive fidelity against state-of-the-art baselines, with extension to controllable image editing.

Abstract

This paper introduces innovative solutions to enhance spatial controllability in diffusion models reliant on text queries. We first introduce vision guidance as a foundational spatial cue within the perturbed distribution. This significantly refines the search space in a zero-shot paradigm to focus on the image sampling process adhering to the spatial layout conditions. To precisely control the spatial layouts of multiple visual concepts with the employment of vision guidance, we propose a universal framework, Layered Rendering Diffusion (LRDiff), which constructs an image-rendering process with multiple layers, each of which applies the vision guidance to instructively estimate the denoising direction for a single object. Such a layered rendering strategy effectively prevents issues like unintended conceptual blending or mismatches while allowing for more coherent and contextually accurate image synthesis. The proposed method offers a more efficient and accurate means of synthesising images that align with specific layout and contextual requirements. Through experiments, we demonstrate that our method outperforms existing techniques, both quantitatively and qualitatively, in two specific layout-to-image tasks: bounding box-to-image and instance maskto-image. Furthermore, we extend the proposed framework to enable spatially controllable editing

Figures (13)

• Figure 1: Overview of our framework. (a) For synthesising a sense, the user provides the global caption, the layered caption, as well as the spatial layout entities which are used to construct the vision guidance. LRDiff divides the reverse-time diffusion process into two sections: (b) When $t\geq t_0$, each vision guidance is employed into separate layers to alter the denoising direction, ensuring each object contour generates within specific regions. (c) When $t < t_0$, we perform the general reverse diffusion process to generate texture details that are consistent with the global caption.
• Figure 2: Qualitative comparisons of methods that use bounding box entities as the spatial condition. Our results show better spatial alignments than other methods.
• Figure 3: Qualitative comparisons of methods that use instance mask entities as the spatial condition. Our results show better spatial alignments than other methods.
• Figure 4: The bounding boxes condition. The first row shows the results using setting #1 and the second row shows the results using setting #2.
• Figure 5: The instant mask condition. The different results of using setting #1 and setting #2.