Joint Optimization of Neural Radiance Fields and Continuous Camera Motion from a Monocular Video

TL;DR

This work tackles the dependency of Neural Radiance Fields on known camera poses by introducing a prior-free framework that jointly optimizes continuous camera motion and a time-dependent NeRF from monocular video. Camera motion is represented as angular velocity and velocity, learned via a motion network and integrated to produce relative transforms, while a time-conditioned NeRF captures local geometry and motion to stabilize training. The approach employs SDF-based NeRF, multiple cross-time and photometric losses, and a staged training strategy to produce accurate poses and depth with competitive novel-view synthesis. Experiments on Co3D and Scannet demonstrate superior pose and depth estimation and rendering quality, with ablations confirming the critical roles of motion modeling, SDF-flow consistency, and time-dependent representation.

Abstract

Neural Radiance Fields (NeRF) has demonstrated its superior capability to represent 3D geometry but require accurately precomputed camera poses during training. To mitigate this requirement, existing methods jointly optimize camera poses and NeRF often relying on good pose initialisation or depth priors. However, these approaches struggle in challenging scenarios, such as large rotations, as they map each camera to a world coordinate system. We propose a novel method that eliminates prior dependencies by modeling continuous camera motions as time-dependent angular velocity and velocity. Relative motions between cameras are learned first via velocity integration, while camera poses can be obtained by aggregating such relative motions up to a world coordinate system defined at a single time step within the video. Specifically, accurate continuous camera movements are learned through a time-dependent NeRF, which captures local scene geometry and motion by training from neighboring frames for each time step. The learned motions enable fine-tuning the NeRF to represent the full scene geometry. Experiments on Co3D and Scannet show our approach achieves superior camera pose and depth estimation and comparable novel-view synthesis performance compared to state-of-the-art methods. Our code is available at https://github.com/HoangChuongNguyen/cope-nerf.

Figures (3)

• Figure 1: Overview of our method. Left: We jointly optimize the camera motion network $\phi_{v}$ to obtain the continuous camera motion represented as angular velocity $\boldsymbol{\omega}$ and velocity $\mathbf{v}$, and a time-dependent NeRF $(\phi_{g}\;,\phi_{c})$ to obtain the scene geometry (represented as Signed Distance Field) and appearance at different time steps. The camera velocities can be integrated to obtain the camera transformation $\mathbf{P}$ between any two frames. Such relative camera motion is used to get 3D correspondences across different time steps. Right: The training our pipeline involves the standard rendering loss $\mathcal{L}_{\text{rgb}}$, and several consistency losses including, the consistency between SDF and camera motion $\mathcal{L}_{\text{flow}}$, the photometric consistency $\mathcal{L}_{\text{photo}}$, and the geometry consistency loss $\mathcal{L}_{\text{sdf}}$.
• Figure 2: Qualitative results on the Co3D (top) and Scannet (bottom) dataset. Our synthesized images are more photo-realistic compared to the other methods. In terms of geometry, our method produces the most accurate depth maps among all methods.
• Figure 3: Camera trajectory visualization. Our poses are better aligned with the ground-truth compared to the others.