A3D: Does Diffusion Dream about 3D Alignment?

TL;DR

This work proposes to embed these objects into a common latent space and optimize the continuous transitions between these objects, and enforce two kinds of properties of these transitions: smoothness of the transition and plausibility of the intermediate objects along the transition.

Abstract

We tackle the problem of text-driven 3D generation from a geometry alignment perspective. Given a set of text prompts, we aim to generate a collection of objects with semantically corresponding parts aligned across them. Recent methods based on Score Distillation have succeeded in distilling the knowledge from 2D diffusion models to high-quality representations of the 3D objects. These methods handle multiple text queries separately, and therefore the resulting objects have a high variability in object pose and structure. However, in some applications, such as 3D asset design, it may be desirable to obtain a set of objects aligned with each other. In order to achieve the alignment of the corresponding parts of the generated objects, we propose to embed these objects into a common latent space and optimize the continuous transitions between these objects. We enforce two kinds of properties of these transitions: smoothness of the transition and plausibility of the intermediate objects along the transition. We demonstrate that both of these properties are essential for good alignment. We provide several practical scenarios that benefit from alignment between the objects, including 3D editing and object hybridization, and experimentally demonstrate the effectiveness of our method. https://voyleg.github.io/a3d/

Figures (13)

• Figure 1: Our method A3D enables conditioning text-to-3D generation process on a set of text prompts to jointly generate a set of 3D objects with a shared structure (top). This enables a user to make hybrids combined of different parts from multiple aligned objects (middle), or perform text-driven structure-preserving transformation of an input 3D model (bottom).
• Figure 2: Collections of objects generated with existing text-to-3D methods lack structural consistency (left, shi2024mvdream). Shapes obtained with existing text-driven 3D editing methods lack text-to-asset alignment and visual quality (middle, mvedit2024). In contrast, our method enables the generation of structurally coherent, text-aligned assets with high visual quality (right).
• Figure 3: Overview of our method. The NeRF model, conditioned on the latent code $\bm{\mathrm{u}}$ sampled from the edges of the latent simplex, produces a render. The render is passed to the diffusion model, conditioned on the linear combination of the embeddings of the text prompts. Finally, the SDS loss is backpropagated to the NeRF model.
• Figure 4: Our method allows us to blend different objects seamlessly. A proper alignment of multiple 3D models provides the ability to replace parts of one object with similar components of the other objects. We manually select spatial anchor points (left) and assign them to a particular model. The latent code $\bm{\mathrm{u}}$ is linearly interpolated between anchors at every spatial location, resulting in a smooth distribution over 3D space (second column). The resulting objects are shown on the right.
• Figure 5: Pairs of objects generated with existing methods and our method. The top two rows show the results for one pair of prompts written below, the bottom two rows show the results for another pair of prompts. For each object, we show a color rendering and a rendering of the geometry below it.