DynPoint: Dynamic Neural Point For View Synthesis

TL;DR

DynPoint, an algorithm designed to facilitate the rapid synthesis of novel views for unconstrained monocular videos, exhibits strong robustness in handling long-duration videos without learning a canonical representation of video content.

Abstract

The introduction of neural radiance fields has greatly improved the effectiveness of view synthesis for monocular videos. However, existing algorithms face difficulties when dealing with uncontrolled or lengthy scenarios, and require extensive training time specific to each new scenario. To tackle these limitations, we propose DynPoint, an algorithm designed to facilitate the rapid synthesis of novel views for unconstrained monocular videos. Rather than encoding the entirety of the scenario information into a latent representation, DynPoint concentrates on predicting the explicit 3D correspondence between neighboring frames to realize information aggregation. Specifically, this correspondence prediction is achieved through the estimation of consistent depth and scene flow information across frames. Subsequently, the acquired correspondence is utilized to aggregate information from multiple reference frames to a target frame, by constructing hierarchical neural point clouds. The resulting framework enables swift and accurate view synthesis for desired views of target frames. The experimental results obtained demonstrate the considerable acceleration of training time achieved - typically an order of magnitude - by our proposed method while yielding comparable outcomes compared to prior approaches. Furthermore, our method exhibits strong robustness in handling long-duration videos without learning a canonical representation of video content.

Figures (10)

• Figure 1: Structure of DynPoint. The Stage 1 shows the pipeline of consistent depth estimation in Sec. \ref{['sec:depth']} and scene flow estimation in Sec. \ref{['sec:flow']}. Initially, the frames are employed in the Flow Net, Depth Net, and Scale Parameters to produce optic flows and depth. Then, surface points are calculated based on the estimated depth and utilized in the Scene Flow MLP. The Stage 2 shows the process of information aggregation presented in Sec. \ref{['sec:aggregate']}. Neural Point Clouds is firstly generated based on pre-computed scene flow. The Rendering MLP utilizes all neural points located within a specified radius from the queried point as inputs to predict the final color and density.
• Figure 2: Demonstration of Geometric Edge Mask and Scene Flow Estimation. The left section depicts the conceptual basis for designing the Geometric Edge Mask. The right part demonstrates the construction of the scene flow objective function shown in Sec. \ref{['sec:flow']}.
• Figure 3: Demonstration of View Synthesis Results on Nvidia Dataset. This demonstration compares the view synthesis outcomes of DynPoint with those of NSFF, HyperNeRF, and RoDynRF.
• Figure 4: Demonstration of View Synthesis Results on Nerfie Dataset. This demonstration compares the view synthesis outcomes of DynPoint with those of NSFF.
• Figure 5: Demonstration of Depth and Scene Flow Estimation. This figure presents the output of the target images obtained by warping the reference image using depth estimation (second row) or using both depth and scene flow estimation (third row). It is important to clarify that the figure is not intended for comparing view synthesis results. The synthesized figures generated based on scene flow inherently incorporate object motion as input, resulting in observable motion blur within the synthesized figures. Additionally, an error map represented by the intensity of red is provided to visualize the performance, where deeper shades of red indicate larger errors (in terms of pixel movement compared to corrected optic flow).