Diffusion Models for Monocular Depth Estimation: Overcoming Challenging Conditions

TL;DR

This work tackles the robustness gap in monocular depth estimation under adverse and out-of-distribution conditions by leveraging diffusion models with depth-aware control to synthesize challenging scenes from easy images while preserving underlying 3D structure. It introduces diffusion-distilled data, pairing easy samples with diffusion-generated hard counterparts, and a self-distillation training protocol that uses a scale-and-shift-invariant loss $L_{ssi}$ to fine-tune pre-trained depth networks. The approach demonstrates consistent improvements over strong baselines across multiple autonomous-driving datasets and challenging surface types, including adverse weather and non-Lambertian materials, without requiring real challenging data. The proposed framework offers a scalable, domain-agnostic pathway to robust monocular depth estimation with broad practical implications for navigation and robotics.

Abstract

We present a novel approach designed to address the complexities posed by challenging, out-of-distribution data in the single-image depth estimation task. Starting with images that facilitate depth prediction due to the absence of unfavorable factors, we systematically generate new, user-defined scenes with a comprehensive set of challenges and associated depth information. This is achieved by leveraging cutting-edge text-to-image diffusion models with depth-aware control, known for synthesizing high-quality image content from textual prompts while preserving the coherence of 3D structure between generated and source imagery. Subsequent fine-tuning of any monocular depth network is carried out through a self-distillation protocol that takes into account images generated using our strategy and its own depth predictions on simple, unchallenging scenes. Experiments on benchmarks tailored for our purposes demonstrate the effectiveness and versatility of our proposal.

Figures (5)

• Figure 1: Framework Results. From top to bottom: source image, depth predictions from the original Depth Anything yang2024depth, and results from our fine-tuned version.
• Figure 2: Method Overview.Left: Image generation and self-distillation. Diffusion Distilled Data (upper): Easy image ($e_i$) and text prompt ($p_c$) input to conditional diffusion models generate adverse scenes ($h_i^c$). Depth Label Distillation (lower): Pre-trained network estimates depth ($d_i$) from easy image ($e_i$). Pairs ($e_i, h_i^c$) used for fine-tuning with scale-and-shift-invariant loss. Right: Fine-tuned network handles diverse inputs in testing, from simple to complex scenarios.
• Figure 3: Generated Images -- Weather Conditions. (a-b): RGB and depth maps from KITTI 2015 Menze2015CVPR. (c-f): images generated by a diffusion model mou2023t2i, conditioned by the depth map from (b) and text prompts indicated in each subfigure.
• Figure 4: Generated Images -- ToM Objects. From top to bottom: easy scenes from Stable Diffusion stable-diffusion-xl, depth from Depth Anything yang2024depth, transformed scenes using mou2023t2i.
• Figure 5: Qualitative Results. From top to bottom: RGB images, depth maps predicted by the original models and the fine-tuned versions using our method.