POCI-Diff: Position Objects Consistently and Interactively with 3D-Layout Guided Diffusion

TL;DR

POCI-Diff addresses the challenge of precise 3D-aware spatial control and consistent object identity in diffusion-based T2I generation. It unifies depth-conditioned 3D layout guidance with per-object semantic binding via Blended Latent Diffusion, enabling one-shot scene generation and warping-free editing, further stabilized by IP-Adapter conditioning for identity preservation. The framework demonstrates state-of-the-art performance in 3D layout adherence, text–image alignment, and visual realism while supporting efficient editing operations such as object insertion, removal, and replacement. This approach offers a scalable, coherent solution for complex multi-object scene synthesis and interactive 3D editing, with practical implications for content creation and 3D-aware diffusion modeling.

Abstract

We propose a diffusion-based approach for Text-to-Image (T2I) generation with consistent and interactive 3D layout control and editing. While prior methods improve spatial adherence using 2D cues or iterative copy-warp-paste strategies, they often distort object geometry and fail to preserve consistency across edits. To address these limitations, we introduce a framework for Positioning Objects Consistently and Interactively (POCI-Diff), a novel formulation for jointly enforcing 3D geometric constraints and instance-level semantic binding within a unified diffusion process. Our method enables explicit per-object semantic control by binding individual text descriptions to specific 3D bounding boxes through Blended Latent Diffusion, allowing one-shot synthesis of complex multi-object scenes. We further propose a warping-free generative editing pipeline that supports object insertion, removal, and transformation via regeneration rather than pixel deformation. To preserve object identity and consistency across edits, we condition the diffusion process on reference images using IP-Adapter, enabling coherent object appearance throughout interactive 3D editing while maintaining global scene coherence. Experimental results demonstrate that POCI-Diff produces high-quality images consistent with the specified 3D layouts and edits, outperforming state-of-the-art methods in both visual fidelity and layout adherence while eliminating warping-induced geometric artifacts.

Figures (7)

• Figure 1: POCI-Diff introduces an interactive diffusion-based system for image generation from 3D layouts, enabling precise control over object type, position, scale, and orientation, as well as camera movement, while supporting layout-driven image editing with seamless integration of new or modified objects.
• Figure 2: Overview of our proposed pipeline. POCI-Diff enables one-shot 3D layout–guided scene generation and warping-free object translation, preserving object identity while maintaining global scene harmonization.
• Figure 3: Qualitative comparison on the 3D layout control task.
• Figure 4: Qualitative comparison for adding and removing objects while keeping the scene consistent.
• Figure 5: Examples of free camera control and object rotation that POCI-Diff can perform.