Fast Dynamic Radiance Fields with Time-Aware Neural Voxels

TL;DR

<3-5 sentence high-level summary> TiNeuVox introduces time-aware neural voxels to dramatically accelerate dynamic NeRF training by combining a tiny deformation network for coarse motion with a multi-distance interpolation scheme over neural voxels. Temporal information is encoded both via a deformation module and through explicit enhancements in the radiance network, enabling accurate modeling of motions at multiple scales with a single-resolution voxel grid. The approach achieves state-of-the-art training speed and storage efficiency (as low as 8 minutes and 8 MB) while maintaining or improving rendering quality compared to prior dynamic NeRF methods. Extensive experiments on synthetic and real scenes validate the method's efficiency, ablations, and robustness to motion scales and scene complexity.

Abstract

Neural radiance fields (NeRF) have shown great success in modeling 3D scenes and synthesizing novel-view images. However, most previous NeRF methods take much time to optimize one single scene. Explicit data structures, e.g. voxel features, show great potential to accelerate the training process. However, voxel features face two big challenges to be applied to dynamic scenes, i.e. modeling temporal information and capturing different scales of point motions. We propose a radiance field framework by representing scenes with time-aware voxel features, named as TiNeuVox. A tiny coordinate deformation network is introduced to model coarse motion trajectories and temporal information is further enhanced in the radiance network. A multi-distance interpolation method is proposed and applied on voxel features to model both small and large motions. Our framework significantly accelerates the optimization of dynamic radiance fields while maintaining high rendering quality. Empirical evaluation is performed on both synthetic and real scenes. Our TiNeuVox completes training with only 8 minutes and 8-MB storage cost while showing similar or even better rendering performance than previous dynamic NeRF methods.

Figures (11)

• Figure 1: Overall framework of TiNeuVox. First, a deformation network $\Phi_d$ takes both point coordinates $\gamma(x, y, z)$ and encoded time embeddings ${\bm{t}}_i = \Phi_t(\gamma(t_i)$ as input to obtain the shifted coordinates $(x', y', z')$. Then voxel features in grids with different sampling strides are queried and interpolated according to deformed coordinates. To enhance temporal information, coordinates $\gamma(x, y, z)$ and time embeddings ${\bm{t}}_i$ are further concatenated with interpolated voxel features, $\gamma({\bm{v}}_s)$, $\gamma({\bm{v}}_m)$, and $\gamma({\bm{v}}_l)$, which are finally fed into the radiance network to produce the density $\sigma$ and color ${\bm{c}}$.
• Figure 2: Illustration of multi-distance interpolation.
• Figure 3: Qualitative comparisons between D-NeRF pumarola2021d and our TiNeuVox on synthetic scenes.
• Figure 4: Qualitative comparisons between TiNeuVox and other methods on real dynamic scenes.
• Figure 5: Visualization of gradient magnitudes on neural voxels with different interpolation distances. Red colors denote voxels from longer distances have larger gradient magnitudes, while yellow ones indicate nearer voxels.