FreeDriveRF: Monocular RGB Dynamic NeRF without Poses for Autonomous Driving via Point-Level Dynamic-Static Decoupling

TL;DR

FreeDriveRF tackles dynamic scene reconstruction for autonomous driving from monocular RGB sequences without known camera poses. It decouples dynamic and static sampling at the point level via a semantically supervised separation field and enforces temporal consistency with warped rays guided by 3D scene flow and 2D optical flow. Dynamic objects are incorporated into the joint optimization of camera poses and radiance fields, yielding improved pose accuracy and high-quality dynamic reconstructions. Evaluations on KITTI and Waymo demonstrate state-of-the-art performance in dynamic scene modeling and pose optimization, enabling robust outdoor driving scene understanding from monocular data.

Abstract

Dynamic scene reconstruction for autonomous driving enables vehicles to perceive and interpret complex scene changes more precisely. Dynamic Neural Radiance Fields (NeRFs) have recently shown promising capability in scene modeling. However, many existing methods rely heavily on accurate poses inputs and multi-sensor data, leading to increased system complexity. To address this, we propose FreeDriveRF, which reconstructs dynamic driving scenes using only sequential RGB images without requiring poses inputs. We innovatively decouple dynamic and static parts at the early sampling level using semantic supervision, mitigating image blurring and artifacts. To overcome the challenges posed by object motion and occlusion in monocular camera, we introduce a warped ray-guided dynamic object rendering consistency loss, utilizing optical flow to better constrain the dynamic modeling process. Additionally, we incorporate estimated dynamic flow to constrain the pose optimization process, improving the stability and accuracy of unbounded scene reconstruction. Extensive experiments conducted on the KITTI and Waymo datasets demonstrate the superior performance of our method in dynamic scene modeling for autonomous driving.

Figures (8)

• Figure 1: Our method reconstructs autonomous driving scenes from monocular RGB sequences without ground truth poses. During optimization, camera poses and rendered masks are updated, guiding dynamic modeling. At the bottom are the rendered RGB and depth maps for both dynamic and static components.
• Figure 2: FreeDriveRF Overview. Our method processes monocular RGB sequences by first sampling rays and inputting points into a dynamic-static separation field, generating a probability $\mathcal{P}$ for each point under mask supervision to distinguish between static and dynamic points for separate modeling. Meanwhile, a scene flow field $\text{MLP}_{\{\text{fw}, \text{bw}\}}$ from dynamic part guides optimization and supervision process. The densities and appearance features are combined with the view direction to compute the final color. Volumetric rendering produces three maps supervised by ground truth and priors, with camera poses jointly optimizing with radiance fields.
• Figure 3: Visual comparison of rendered RGB and masks with or without the proposed sampling level dynamic-static decoupling.
• Figure 4: Static background reconstruction. Our sampling point level dynamic-static decoupling reconstructs occluded static regions more effectively and produces fewer artifacts compared to others.
• Figure 5: Comparison of pose trajectory on sequence 03 of KITTI. We present our qualitative results compared with liu2023robust, meuleman2023progressively, and without optical flow constraint for pose optimization.