WeatherDepth: Curriculum Contrastive Learning for Self-Supervised Depth Estimation under Adverse Weather Conditions

TL;DR

The proposed WeatherDepth, a self-supervised robust depth estimation model with curriculum contrastive learning, to tackle performance degradation in complex weather conditions is proposed and state-of-the-art (SoTA) performance on both synthetic and real weather datasets is demonstrated.

Abstract

Depth estimation models have shown promising performance on clear scenes but fail to generalize to adverse weather conditions due to illumination variations, weather particles, etc. In this paper, we propose WeatherDepth, a self-supervised robust depth estimation model with curriculum contrastive learning, to tackle performance degradation in complex weather conditions. Concretely, we first present a progressive curriculum learning scheme with three simple-to-complex curricula to gradually adapt the model from clear to relative adverse, and then to adverse weather scenes. It encourages the model to gradually grasp beneficial depth cues against the weather effect, yielding smoother and better domain adaption. Meanwhile, to prevent the model from forgetting previous curricula, we integrate contrastive learning into different curricula. By drawing reference knowledge from the previous course, our strategy establishes a depth consistency constraint between different courses toward robust depth estimation in diverse weather. Besides, to reduce manual intervention and better adapt to different models, we designed an adaptive curriculum scheduler to automatically search for the best timing for course switching. In the experiment, the proposed solution is proven to be easily incorporated into various architectures and demonstrates state-of-the-art (SoTA) performance on both synthetic and real weather datasets. Source code and data are available at \url{https://github.com/wangjiyuan9/WeatherDepth}.

Figures (3)

• Figure 1: Typical examples on real weather images. Compared with Robust-Depth* (the SoTA robust depth estimation model under adverse weather), our WeatherDepth* produces more accurate results against (a) snowflakes, (b) raindrops on the lens, and (c) water surface reflections. Note both solutions adopt the same baseline model (MonoViT).
• Figure 2: Comparison of simulated adverse weather. The other augmentations in the third column are from previous weather depth estimation studies iccvr22rbt, which also adopt data augmentation. Obviously, our WeatherKITTI augmentation is significantly more natural than their results.
• Figure 3: WeatherDepth pipeline Through three progressive stages, our model-agnostic approach can estimate depth reliably under weather environments. Except for the last stage, we input the loss of estimation model into the curriculum scheduler, to change the level properly. And we input image pairs $I_{aug}$ and $I_{cst}$ to obtain depth maps $D_{aug}$ and $D_{cst}$. $D_{cst}$ is detached as the contrastive target to compute contrastive loss, which is weighted and backpropagated together with the original loss.