Dynamic Appearance Particle Neural Radiance Field

TL;DR

This paper tackles the challenge of separately modeling appearance and motion in dynamic NeRFs by introducing Dynamic Appearance Particle NeRF (DAP-NeRF), a hybrid framework that couples a static Eulerian feature grid with a dynamic, particle-based appearance field. Each appearance particle carries a fixed visual feature ${oldsymbol v}^i$ and follows a time-varying trajectory ${oldsymbol p}^i(t)$ governed by a shared motion network, enabling physically meaningful motion representations learned from monocular video. The dynamic field is efficiently computed by a grid-based scheme that aggregates particle features onto a dynamic grid and performs fast tri-linear interpolation, while the static field remains time-invariant; the two fields are merged through a superposition that depends on local dynamic activity. Training uses standard photometric losses plus regularizers for the static/dynamic fields and particle motion, with a grid coarse-to-fine strategy and particle removal/re-sampling to maintain a compact dynamic representation. Empirical results across multiple datasets show strong novel-view synthesis and superior motion modeling, along with improved efficiency and practical benefits such as scene editing and collision detection, illustrating the method’s potential for physically interpretable dynamic scene understanding from monocular data.

Abstract

Neural Radiance Fields (NeRFs) have shown great potential in modeling 3D scenes. Dynamic NeRFs extend this model by capturing time-varying elements, typically using deformation fields. The existing dynamic NeRFs employ a similar Eulerian representation for both light radiance and deformation fields. This leads to a close coupling of appearance and motion and lacks a physical interpretation. In this work, we propose Dynamic Appearance Particle Neural Radiance Field (DAP-NeRF), which introduces particle-based representation to model the motions of visual elements in a dynamic 3D scene. DAP-NeRF consists of the superposition of a static field and a dynamic field. The dynamic field is quantized as a collection of appearance particles, which carries the visual information of a small dynamic element in the scene and is equipped with a motion model. All components, including the static field, the visual features and the motion models of particles, are learned from monocular videos without any prior geometric knowledge of the scene. We develop an efficient computational framework for the particle-based model. We also construct a new dataset to evaluate motion modeling. Experimental results show that DAP-NeRF is an effective technique to capture not only the appearance but also the physically meaningful motions in a 3D dynamic scene. Code is available at: https://github.com/Cenbylin/DAP-NeRF.

Figures (21)

• Figure 1: Method Overview. (a) Particles represent observed movements. Each particle $\boldsymbol p$ corresponds to a small volume of material that represents a semantically meaningful part of a moving object. The position $\boldsymbol p(t)$ forms an explicit dynamic model of the object part. $\boldsymbol v$ denotes the visual feature of the particle (See Sec. \ref{['subs:particle-dyn-model']}). (b) Particles are integrated into a grid-based feature field, enabling efficient computation and superposition with the static feature field (See Sec. \ref{['sec:eff_compute']}). (c) Colors of rays are computed by querying sampled points on the superpositional radiance field and performing volume rendering. Photometric loss is then calculated to optimize the model (see Sec. \ref{['sec:optim']}).
• Figure 2: Commonly adopted structure of dynamic NeRFs.
• Figure 3: Overview of the proposed hybrid representation. (a) Superposition of dynamic and static field modeled by particle-based and Eulerian (voxel-grid-based) representations. (b) Two main attributes represented by particles. The particle trajectory ${\boldsymbol{p}}^i(t)$ is time-varying, while the appearance feature ${\boldsymbol{v}}^i$ is time invariant.
• Figure 4: Computational structure for time-varying position of a particle. $\gamma$ is the positional encoding function Mildenhall20, $\phi_m$ and $\phi_t$ are $3$ and $2$-layer MLP, 'CAT' is concatenate operation and 'ADD' is element-wise addition.
• Figure 5: Training process of particles at $t=0$. Panes show the particles at $t=0$ during different stages of training. (See Sec. \ref{['sec:ap']})