Manydepth2: Motion-Aware Self-Supervised Monocular Depth Estimation in Dynamic Scenes

TL;DR

Manydepth2 tackles dynamic-scene self-supervised monocular depth estimation by introducing a motion-aware cost volume built from optical flow and a pseudo-static reference frame, and refining depth with an attention-enhanced HRNet. The method jointly estimates depth and rigid motion while employing a photometric self-supervision loss with L1, SSIM, and depth-consistency terms. It achieves notable RMSE improvements on KITTI-2015 and Cityscapes, demonstrates strong odometry accuracy gains, and remains computationally efficient enough to train on a single RTX 3090. This work advances robust depth perception in dynamic environments and provides a practical, scalable approach for real-world autonomous perception systems.

Abstract

Despite advancements in self-supervised monocular depth estimation, challenges persist in dynamic scenarios due to the dependence on assumptions about a static world. In this paper, we present Manydepth2, to achieve precise depth estimation for both dynamic objects and static backgrounds, all while maintaining computational efficiency. To tackle the challenges posed by dynamic content, we incorporate optical flow and coarse monocular depth to create a pseudo-static reference frame. This frame is then utilized to build a motion-aware cost volume in collaboration with the vanilla target frame. Furthermore, to improve the accuracy and robustness of the network architecture, we propose an attention-based depth network that effectively integrates information from feature maps at different resolutions by incorporating both channel and non-local attention mechanisms. Compared to methods with similar computational costs, Manydepth2 achieves a significant reduction of approximately five percent in root-mean-square error for self-supervised monocular depth estimation on the KITTI-2015 dataset. The code could be found at https://github.com/kaichen-z/Manydepth2.

Figures (5)

• Figure 1: Qualitative comparison on Cityscapes. The first row presents the RGB images. The second row shows the error maps of the depth estimates produced by ManyDepth. The third row displays the error maps of the depth estimates generated by Manydepth2.
• Figure 2: Illustration for the structure of Manydepth2. During Stage 1, target and reference frame $\bm{I}_t$ and $\bm{I}_r$ are processed Flow Net, Pose Net, and Depth Net to generate optical flow $\bm{f}_{r\leftarrow t}$, transformation matrix $[\bm{R}|\bm{t}]_{rt}$, and coarse depth $\bm{D}^c_t$. During Stage 2, the outputs are used to generate the motion-aware cost volume $\bm{\mathcal{V}}_m$. Finally, the motion-aware cost volume $\bm{\mathcal{V}}_m$ and the target frame $\bm{I}_t$ are used by Attention-Based Depth Net to produce the refined depth $\bm{D}_t$.
• Figure 3: Relationship between optical flow and depth in the dynamic scenario. This figure demonstrates that in a dynamic scenario, there is a discrepancy between the static optical flow and the real optical flow.
• Figure 4: Qualitative results on KITTI Dataset. The initial row displays the RGB images of target frames where depth information has been estimated. The subsequent rows, specifically the second, third, and fourth, showcase depth maps generated by Monodepth2 trained with stereo techniques, ManyDepth, and Manydepth2, respectively.
• Figure 5: Trajectory visualization on Seq. 09. and Seq. 10. The ground truth trajectories are represented by red lines. Trajectories generated by the FeatDepth shu2020feature results are indicated by green lines, while our method's trajectories are depicted in blue lines.