Mining Supervision for Dynamic Regions in Self-Supervised Monocular Depth Estimation

TL;DR

The paper tackles the challenge of self-supervised monocular depth estimation in dynamic scenes by decoupling static-background depth from dynamic-object depth and generating scale-consistent pseudo depth labels for moving objects. It introduces a Depth Scale Alignment (DSA) module and a dynamic-object depth estimation pipeline that yields pseudo labels $D_r^{label}$, enabling end-to-end training of a depth network with reliable supervision for dynamic regions. The approach demonstrates substantial improvements over prior self-/unsupervised methods, achieving state-of-the-art performance on Cityscapes for dynamic regions and notable gains on KITTI as well, with iterative self-training offering further refinements. This work enhances robustness of monocular depth estimation in real-world, dynamic environments, with potential impact on autonomous driving and related 3D perception tasks.

Abstract

This paper focuses on self-supervised monocular depth estimation in dynamic scenes trained on monocular videos. Existing methods jointly estimate pixel-wise depth and motion, relying mainly on an image reconstruction loss. Dynamic regions1 remain a critical challenge for these methods due to the inherent ambiguity in depth and motion estimation, resulting in inaccurate depth estimation. This paper proposes a self-supervised training framework exploiting pseudo depth labels for dynamic regions from training data. The key contribution of our framework is to decouple depth estimation for static and dynamic regions of images in the training data. We start with an unsupervised depth estimation approach, which provides reliable depth estimates for static regions and motion cues for dynamic regions and allows us to extract moving object information at the instance level. In the next stage, we use an object network to estimate the depth of those moving objects assuming rigid motions. Then, we propose a new scale alignment module to address the scale ambiguity between estimated depths for static and dynamic regions. We can then use the depth labels generated to train an end-to-end depth estimation network and improve its performance. Extensive experiments on the Cityscapes and KITTI datasets show that our self-training strategy consistently outperforms existing self/unsupervised depth estimation methods.

Figures (5)

• Figure 1: Depth predictions for an image. Areas with more intense red color in the error maps represent higher error. Our method produces more accurate depths for moving objects in the image.
• Figure 2: Our proposed framework. Left: How the pseudo depth label $\mathbf{D}_r^{label}$ is extracted and used to train the final depth network. Right: How the DSA module is trained (training path) and utilized to produce $\mathbf{D}_r^{label}$ (inference path). The DSA is trained to predict a scale $\beta^o$ that minimizes the difference between two depth predictions of the same static object. It is then used to align the depth scale of every dynamic object to the depth scale of the static region. $\mathbf{D}_r^{label}$ is constructed from depth of static regions in $\mathbf{D}_r^{scene}$ and depth of dynamic objects after alignment $\beta^o \mathbf{D}_r^o$.
• Figure 3: Qualitative segmentation results. Static cars (two in the middle, stopped by the traffic light) originally ignored by the pseudo label are successfully segmented out by both our self-supervised model as well as SEEM.
• Figure 4: Qualitative results on Cityscapes. The error map is computed as the per-pixel relative absolute depth error. Red areas indicate high errors. While the depth maps are not clearly distinguishable, the error maps indicate that both lee2021learning and li2021unsupervised fail to predict accurate depth for moving objects in the images. In contrast, our models are able to predict the depth for these objects with smaller errors. Our model, in conjunction with a pre-trained segmentation model SEEMzou2023segment, can achieve even better predictions.
• Figure 5: Depth prediction errors at different stage/self-training iteration. The largest performance gains are achieved while fine-tuning our method with the pseudo label $\mathbf{D}_r^{label}$. Their performance slightly improves through our iterative training strategy.