VolumeDiffusion: Flexible Text-to-3D Generation with Efficient Volumetric Encoder

TL;DR

VolumeDiffusion advances text-to-3D generation by introducing a scalable volumetric feature-volume representation learned by a fast, feed-forward volume encoder. A diffusion process operates directly in high-dimensional voxel space, with a novel linear noise schedule and a low-frequency noise mechanism to robustly train on volumes, coupled with a refinement step to improve textures. Training on a filtered Objaverse subset enables diverse, text-controllable outputs and substantially faster data generation compared to optimization-based methods. The approach achieves strong qualitative and quantitative results, offering fine-grained control over object parts and enabling large-scale data-driven 3D generation.

Abstract

This paper introduces a pioneering 3D volumetric encoder designed for text-to-3D generation. To scale up the training data for the diffusion model, a lightweight network is developed to efficiently acquire feature volumes from multi-view images. The 3D volumes are then trained on a diffusion model for text-to-3D generation using a 3D U-Net. This research further addresses the challenges of inaccurate object captions and high-dimensional feature volumes. The proposed model, trained on the public Objaverse dataset, demonstrates promising outcomes in producing diverse and recognizable samples from text prompts. Notably, it empowers finer control over object part characteristics through textual cues, fostering model creativity by seamlessly combining multiple concepts within a single object. This research significantly contributes to the progress of 3D generation by introducing an efficient, flexible, and scalable representation methodology.

Figures (13)

• Figure 1: Framework of VolumeDiffusion. It comprises the volume encoding stage and the diffusion modeling stage. The encoder unprojects multi-view images into a feature volume and do refinements. The diffusion model learns to predict ground-truths given noised volumes and text conditions.
• Figure 2: Renderings of noised volumes. Volumes with common i.i.d. noise are still recognizable at large timesteps, while low-frequency noise effectively removes information.
• Figure 3: Reconstructions of the volume encoder.
• Figure 4: Comparison with state-of-the-art text-to-3D methods.
• Figure 5: Text-to-3D generations by VolumeDiffusion.