Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scaffold Diffusion: Sparse Multi-Category Voxel Structure Generation with Discrete Diffusion

Justin Jung

TL;DR

Scaffold Diffusion tackles the challenge of generating sparse multi-category 3D voxel structures under severe memory and class-imbalance constraints by treating voxels as tokens and applying a discrete diffusion language model with 3D positional encoding. It conditions generation on the set of occupied voxels, samples a sequence of token positions, and reconstructs a coherent 3D voxel map, achieving realistic structures on the 3D-Craft Minecraft house dataset with exceptionally high sparsity ($\approx$98% background). The approach, based on a Diffusion Transformer backbone and MDLM objective, outperforms autoregressive and prior discrete diffusion baselines in qualitative coherence and diversity, and benefits significantly from 3D sinusoidal positional embeddings. The work includes an interactive viewer for exploring samples and generation dynamics, signaling practical impact for game design, robotics, and virtual environments where sparse 3D structures are common.

Abstract

Generating realistic sparse multi-category 3D voxel structures is difficult due to the cubic memory scaling of voxel structures and moreover the significant class imbalance caused by sparsity. We introduce Scaffold Diffusion, a generative model designed for sparse multi-category 3D voxel structures. By treating voxels as tokens, Scaffold Diffusion uses a discrete diffusion language model to generate 3D voxel structures. We show that discrete diffusion language models can be extended beyond inherently sequential domains such as text to generate spatially coherent 3D structures. We evaluate on Minecraft house structures from the 3D-Craft dataset and demonstrate that, unlike prior baselines and an auto-regressive formulation, Scaffold Diffusion produces realistic and coherent structures even when trained on data with over 98% sparsity. We provide an interactive viewer where readers can visualize generated samples and the generation process: https://scaffold.deepexploration.org/

Scaffold Diffusion: Sparse Multi-Category Voxel Structure Generation with Discrete Diffusion

TL;DR

Scaffold Diffusion tackles the challenge of generating sparse multi-category 3D voxel structures under severe memory and class-imbalance constraints by treating voxels as tokens and applying a discrete diffusion language model with 3D positional encoding. It conditions generation on the set of occupied voxels, samples a sequence of token positions, and reconstructs a coherent 3D voxel map, achieving realistic structures on the 3D-Craft Minecraft house dataset with exceptionally high sparsity (98% background). The approach, based on a Diffusion Transformer backbone and MDLM objective, outperforms autoregressive and prior discrete diffusion baselines in qualitative coherence and diversity, and benefits significantly from 3D sinusoidal positional embeddings. The work includes an interactive viewer for exploring samples and generation dynamics, signaling practical impact for game design, robotics, and virtual environments where sparse 3D structures are common.

Abstract

Generating realistic sparse multi-category 3D voxel structures is difficult due to the cubic memory scaling of voxel structures and moreover the significant class imbalance caused by sparsity. We introduce Scaffold Diffusion, a generative model designed for sparse multi-category 3D voxel structures. By treating voxels as tokens, Scaffold Diffusion uses a discrete diffusion language model to generate 3D voxel structures. We show that discrete diffusion language models can be extended beyond inherently sequential domains such as text to generate spatially coherent 3D structures. We evaluate on Minecraft house structures from the 3D-Craft dataset and demonstrate that, unlike prior baselines and an auto-regressive formulation, Scaffold Diffusion produces realistic and coherent structures even when trained on data with over 98% sparsity. We provide an interactive viewer where readers can visualize generated samples and the generation process: https://scaffold.deepexploration.org/

Paper Structure

This paper contains 15 sections, 3 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. 3D and Voxel Generative Models
  4. Minecraft Generative Models
  5. Discrete Diffusion Models
  6. Preliminary
  7. Training Objective
  8. Method
  9. Experiments
  10. Implementation Details
  11. Baselines
  12. Qualitative Results
  13. Ablations
  14. Limitations and Future work
  15. Conclusion

Figures (3)

  • Figure 1: Progress of voxel structure generation with Scaffold Diffusion.
  • Figure 2: Sample generation quality comparison between Scaffold Diffusion and baselines; Scaffold Diffusion (top row), autoregressive baseline (middle row), and lee2023diffusion (bottom row). While Scaffold Diffusion can generate realistic and functional 3D structures, the autoregressive baseline generates structures dominated by a few block types or structures with implausible block placements. lee2023diffusion suffers from an over-representation of background voxels.
  • Figure 3: Diversity of generated samples. Scaffold Diffusion produces varied and realistic 3D structures for the same occupancy map.