GeoDiffusion: A Training-Free Framework for Accurate 3D Geometric Conditioning in Image Generation

TL;DR

GeoDiffusion tackles the challenge of precise 3D geometric control in image generation by introducing a training-free framework that leverages a domain-specific 3D reference object to define parametric 3D constraints. The method projects these 3D differences into 2D space through viewpoint rendering, style transfer, and a GeoDrag-based editing pipeline, followed by diffusion-based image refinement. Its core contributions include a Point Fixation and Copy-and-Paste latent refinement in GeoDrag, enabling accurate, viewpoint-consistent edits across object categories, plus a geometry-guidance benchmark for evaluation. The results demonstrate improved geometric accuracy and competitive image fidelity and speed compared to existing dragging-based methods, with broad applicability to early-phase design workflows without extensive per-object training.

Abstract

Precise geometric control in image generation is essential for engineering \& product design and creative industries to control 3D object features accurately in image space. Traditional 3D editing approaches are time-consuming and demand specialized skills, while current image-based generative methods lack accuracy in geometric conditioning. To address these challenges, we propose GeoDiffusion, a training-free framework for accurate and efficient geometric conditioning of 3D features in image generation. GeoDiffusion employs a class-specific 3D object as a geometric prior to define keypoints and parametric correlations in 3D space. We ensure viewpoint consistency through a rendered image of a reference 3D object, followed by style transfer to meet user-defined appearance specifications. At the core of our framework is GeoDrag, improving accuracy and speed of drag-based image editing on geometry guidance tasks and general instructions on DragBench. Our results demonstrate that GeoDiffusion enables precise geometric modifications across various iterative design workflows.

Figures (5)

• Figure 1: (a) The GeoDiffusion architecture for accurate geometry guidance of objects in image space. A reference 3D object, its geometric features, a text prompt, and the geometric target are inputs for the framework. (b) For the dragging-based image modification, latent optimization with point fixation is done for $B$ iterations $g_k$ in each timestep $t$. At the end of each timestep, the patches $\mathcal{Q}$ around all points $p_i$ that are close to their target destination are copied and pasted to $\mathcal{G}$, and the initial patch is blurred. For the remaining denoising timesteps after the latent optimization, the patches $\mathcal{Q}$ are copied and pasted to $\mathcal{G}$ without blurring $\mathcal{Q}$. Best viewed when zoomed in.
• Figure 2: Geometry-guided generation of a car: The user inputs the length, height, and width as the geometric target features. With the reference source points (red) and the point translation function, the target points (blue) are derived in 3D space. Using the viewpoint and the text prompt for style transfer, the image is rendered and subsequently modified in image space. Reference and target points are visualized in the 3D scene and the rendered image.
• Figure 3: Qualitative results of the GeoDiffusion framework. a) Comparison to other methods. b) Framework results.
• Figure 4: Image modifications with GeoDrag on DragBench.
• Figure 5: GeoDiffusion with non-linear deformations.