D-TensoRF: Tensorial Radiance Fields for Dynamic Scenes

TL;DR

D-TensoRF extends tensorial radiance fields to dynamic scenes by representing the radiance field as a 5D tensor over (X, Y, Z, time) and factorizing it with CP and a new Matrix-Matrix decomposition. Temporal smoothing along the time axis captures motion coherence, enabling novel-view synthesis at specific times with a compact, fast model. The approach achieves competitive rendering quality with significantly smaller memory footprints and faster training than deformation-based dynamic NeRF methods, particularly excelling in perceptual quality (LPIPS). This makes dynamic 3D scene reconstruction more practical for applications requiring efficiency and scalability.

Abstract

Neural radiance field (NeRF) attracts attention as a promising approach to reconstructing the 3D scene. As NeRF emerges, subsequent studies have been conducted to model dynamic scenes, which include motions or topological changes. However, most of them use an additional deformation network, slowing down the training and rendering speed. Tensorial radiance field (TensoRF) recently shows its potential for fast, high-quality reconstruction of static scenes with compact model size. In this paper, we present D-TensoRF, a tensorial radiance field for dynamic scenes, enabling novel view synthesis at a specific time. We consider the radiance field of a dynamic scene as a 5D tensor. The 5D tensor represents a 4D grid in which each axis corresponds to X, Y, Z, and time and has 1D multi-channel features per element. Similar to TensoRF, we decompose the grid either into rank-one vector components (CP decomposition) or low-rank matrix components (newly proposed MM decomposition). We also use smoothing regularization to reflect the relationship between features at different times (temporal dependency). We conduct extensive evaluations to analyze our models. We show that D-TensoRF with CP decomposition and MM decomposition both have short training times and significantly low memory footprints with quantitatively and qualitatively competitive rendering results in comparison to the state-of-the-art methods in 3D dynamic scene modeling.

Figures (11)

• Figure 1: Tensor decomposition of 4D tensor. While CP decomposition factorizes the 4D tensor into a summation of outer products of vector factors, MM decomposition factorizes the tensor into a summation of outer products of matrix factors(Eqn.\ref{['eq:10']}).
• Figure 2: Smoothing regularization for models with CP decomposition(Left, Eqn.\ref{['eq:13']}) and MM decomposition(Right, Eqn.\ref{['eq:14']}). Smoothing regularization is applied while moving the predefined size of the window along the vector factor or the specific mode of matrix factor corresponding to the time axis.
• Figure 3: Overall framework of D-TensoRF with MM decomposition.
• Figure 4: Addtional rendering results of D-TensoRF with CP decomposition(Left) and MM decomposition(Right). D-TensoRF-CP tends to produce blurry results compared to D-TensoRF-MM.
• Figure 5: Qualitative results of D-TensoRF-CP, D-TensoRF-MM and baseline methods on the extended Synthetic NeRF scenes provided by D-NeRF(pumarola2020d).