LATTE3D: Large-scale Amortized Text-To-Enhanced3D Synthesis

TL;DR

LATTE3D tackles the slow, per-prompt optimization bottleneck in text-to-3D by introducing a large-scale amortized framework that jointly trains geometry and texture across 100k prompts with 34k shapes. It uses a two-stage pipeline with 3D-aware diffusion priors, pretraining for shape reconstruction, and a 3D regularization term to stabilize learning, enabling 400ms inference and fast test-time refinement. The model amortizes both stages, scales to large prompt sets, and supports 3D stylization, achieving competitive fidelity with far lower compute costs than prior methods. This approach enables real-time, high-fidelity 3D content creation and flexible user control, including post-hoc refinement and style variations, while highlighting future work on dynamic geometry and further reductions in SDS reliance.

Abstract

Recent text-to-3D generation approaches produce impressive 3D results but require time-consuming optimization that can take up to an hour per prompt. Amortized methods like ATT3D optimize multiple prompts simultaneously to improve efficiency, enabling fast text-to-3D synthesis. However, they cannot capture high-frequency geometry and texture details and struggle to scale to large prompt sets, so they generalize poorly. We introduce LATTE3D, addressing these limitations to achieve fast, high-quality generation on a significantly larger prompt set. Key to our method is 1) building a scalable architecture and 2) leveraging 3D data during optimization through 3D-aware diffusion priors, shape regularization, and model initialization to achieve robustness to diverse and complex training prompts. LATTE3D amortizes both neural field and textured surface generation to produce highly detailed textured meshes in a single forward pass. LATTE3D generates 3D objects in 400ms, and can be further enhanced with fast test-time optimization.

Figures (34)

• Figure 1: Samples generated in $\sim$400ms on a single A6000 GPU from text prompts. Objects without prompt labels are generated by our text-to-3D model trained on $\sim\!100$k prompts, while labeled objects are generated by our 3D stylization model trained on $12$k prompts. See the https://research.nvidia.com/labs/toronto-ai/LATTE3D/ for more.
• Figure 2: A quantitative comparison of SOTA text-to-3D methods on unseen prompts. We plot different methods' user study preference rates compared to Latte3D. For MVDream, we report results with varying optimization times.
• Figure 3: We overview our reconstruction pretraining here, which we use to achieve our shape initialization to improve prompt robustness.
• Figure 4: Latte3D consists of two networks: a texture network $T$ and geometry network $G$. When amortizing the first stage, the encoders of both networks share the same set of weights. The training objective includes an SDS gradient from a 3D-aware image prior and a regularization loss that compares the rendered predicted shape's mask with the rendered masks of 3D assets in a library. When amortizing surface-based refinement in stage-2, we freeze the geometry network $G$ and update the texture network $T$.
• Figure 5: Stylization application: Our model learns to generate diverse stylizations of the same shapes. Left column shows the original shape to be stylized.