SPIdepth: Strengthened Pose Information for Self-supervised Monocular Depth Estimation

TL;DR

SPIdepth tackles self-supervised monocular depth estimation by prioritizing a strengthened pose network during training to better capture scene geometry, building on the Self Query Layer from SQLdepth. The method integrates DepthNet with a ConvNext-based encoder, a robust PoseNet, and a Self Query Layer to produce depth estimates from single frames, achieving state-of-the-art results on KITTI and strong cross-dataset performance on Cityscapes and Make3D. Key contributions include a detailed ablation showing the superiority of strengthened pose information over mere backbone scaling, and demonstrated generalization to dynamic scenes and zero-shot Make3D without motion masks. The work implies that pose information is a critical lever for improving depth in real-world, single-image deployment scenarios, with public code available for replication and extension.

Abstract

Self-supervised monocular depth estimation has garnered considerable attention for its applications in autonomous driving and robotics. While recent methods have made strides in leveraging techniques like the Self Query Layer (SQL) to infer depth from motion, they often overlook the potential of strengthening pose information. In this paper, we introduce SPIdepth, a novel approach that prioritizes enhancing the pose network for improved depth estimation. Building upon the foundation laid by SQL, SPIdepth emphasizes the importance of pose information in capturing fine-grained scene structures. By enhancing the pose network's capabilities, SPIdepth achieves remarkable advancements in scene understanding and depth estimation. Experimental results on benchmark datasets such as KITTI, Cityscapes, and Make3D showcase SPIdepth's state-of-the-art performance, surpassing previous methods by significant margins. Specifically, SPIdepth tops the self-supervised KITTI benchmark. Additionally, SPIdepth achieves the lowest AbsRel (0.029), SqRel (0.069), and RMSE (1.394) on KITTI, establishing new state-of-the-art results. On Cityscapes, SPIdepth shows improvements over SQLdepth of 21.7% in AbsRel, 36.8% in SqRel, and 16.5% in RMSE, even without using motion masks. On Make3D, SPIdepth in zero-shot outperforms all other models. Remarkably, SPIdepth achieves these results using only a single image for inference, surpassing even methods that utilize video sequences for inference, thus demonstrating its efficacy and efficiency in real-world applications. Our approach represents a significant leap forward in self-supervised monocular depth estimation, underscoring the importance of strengthening pose information for advancing scene understanding in real-world applications. The code and pre-trained models are publicly available at https://github.com/Lavreniuk/SPIdepth.

Figures (2)

• Figure 1: The SPIdepth architecture. An encoder-decoder extracts features from frame $I_t$, which are then input into the Self Query Layer to obtain the depth map $D_t$. Strengthened PoseNet predicts the relative pose between frame $I_t$ and reference frame $I_t'$ using a powerful pose network, needed only during training. Pixels from frame $I_t'$ are used to reconstruct frame $I_t$ with depth map $D_t$ and relative pose $T_{t' \rightarrow t}$. The loss function is based on the differences between the warped image $I_{t' \rightarrow t}$ and the source image $I_t$.
• Figure 2: Qualitative results on the KITTI dataset. From left to right: Input RGB image, Ground Truth, SQLdepth prediction, and SPIdepth prediction.