DynaVol: Unsupervised Learning for Dynamic Scenes through Object-Centric Voxelization

TL;DR

DynaVol tackles unsupervised learning of 3D dynamic scene decomposition by introducing object-centric 4D voxel grids that track per-object occupancy over time. It combines canonical-space dynamics, volume slot attention, and a compositional NeRF renderer to jointly model local voxel evolution and global object properties, trained through a three-stage process (3D voxel warmup, 3D-to-4D expansion, and 4D optimization). The approach achieves state-of-the-art results on synthetic and real datasets for novel-view synthesis and scene decomposition, while enabling direct editing of geometry and motion without retraining. This work advances 3D unsupervised scene understanding and enables practical applications in scene editing and relational reasoning in dynamic environments.

Abstract

Unsupervised learning of object-centric representations in dynamic visual scenes is challenging. Unlike most previous approaches that learn to decompose 2D images, we present DynaVol, a 3D scene generative model that unifies geometric structures and object-centric learning in a differentiable volume rendering framework. The key idea is to perform object-centric voxelization to capture the 3D nature of the scene, which infers the probability distribution over objects at individual spatial locations. These voxel features evolve over time through a canonical-space deformation function, forming the basis for global representation learning via slot attention. The voxel features and global features are complementary and are both leveraged by a compositional NeRF decoder for volume rendering. DynaVol remarkably outperforms existing approaches for unsupervised dynamic scene decomposition. Once trained, the explicitly meaningful voxel features enable additional capabilities that 2D scene decomposition methods cannot achieve: it is possible to freely edit the geometric shapes or manipulate the motion trajectories of the objects.

Figures (13)

• Figure 1: DynaVol explores an unsupervised object-centric voxelization approach for dynamic scene decomposition. Unlike its 2D counterparts, such as SAVi kipf2021conditional, DynaVol ensures 3D consistency and provides additional capabilities, e.g., novel view synthesis and scene editing.
• Figure 2: The architecture of DynaVol consists of three groups of network components: the bi-directional dynamics modules ($f_\psi$, $f_\xi^\prime$), the slot attention module with a GRU and a 3D volume encoder ($Z_\omega$, $E_\theta$), and neural rendering modules ($N_{\phi^\prime}$, $N_\phi$) based on 3D and 4D voxels respectively. The training scheme involves a 3D voxel warmup stage (top), a 3D-to-4D voxel expansion stage (middle, whose pseudocode is given in Appendix.\ref{['sec:pseudocode']}), and a 4D voxel optimization stage (bottom).
• Figure 3: Novel view synthesis results in synthetic and real-world scenes. D-NeRF fails in 3ObjRand. We also visualize corresponding decomposition maps for real-world scenes. It's important to note that decomposing moving objects in a real-world scene often requires additional geometric priors. Relying solely on space-time continuity within a single scene may not be sufficient to separate objects like a hand and a banana, especially when they are in contact throughout the video.
• Figure 4: Visualization of scene decompostion results for 6ObjFall.
• Figure 5: Showcases of dynamic scene editing for both real-world (top) and synthetic scenes (bottom). We visualize the edited decomposition map and the corresponding rendering results.