Jasmine: Harnessing Diffusion Prior for Self-supervised Depth Estimation

TL;DR

Jasmine tackles monocular depth estimation under a self-supervised regime by injecting Stable Diffusion priors into the learning process. It introduces a Mix-batch Image Reconstruction surrogate task to preserve SD priors and mitigate reprojection artifacts, and a Scale-Shift GRU to align scale and shift between SSI and SI depth estimates. The approach achieves state-of-the-art results on KITTI and demonstrates robust zero-shot generalization across diverse datasets, while providing thorough ablations and de-normalization analyses. This work paves the way for SD-guided, non-annotated 3D perception with strong cross-domain performance and detailed depth restoration capabilities.

Abstract

In this paper, we propose Jasmine, the first Stable Diffusion (SD)-based self-supervised framework for monocular depth estimation, which effectively harnesses SD's visual priors to enhance the sharpness and generalization of unsupervised prediction. Previous SD-based methods are all supervised since adapting diffusion models for dense prediction requires high-precision supervision. In contrast, self-supervised reprojection suffers from inherent challenges (e.g., occlusions, texture-less regions, illumination variance), and the predictions exhibit blurs and artifacts that severely compromise SD's latent priors. To resolve this, we construct a novel surrogate task of hybrid image reconstruction. Without any additional supervision, it preserves the detail priors of SD models by reconstructing the images themselves while preventing depth estimation from degradation. Furthermore, to address the inherent misalignment between SD's scale and shift invariant estimation and self-supervised scale-invariant depth estimation, we build the Scale-Shift GRU. It not only bridges this distribution gap but also isolates the fine-grained texture of SD output against the interference of reprojection loss. Extensive experiments demonstrate that Jasmine achieves SoTA performance on the KITTI benchmark and exhibits superior zero-shot generalization across multiple datasets.

Figures (8)

• Figure 1: Without any high-precision depth supervision, Jasmine achieves remarkably detailed and accurate depth estimation results through zero-shot generalization across diverse scenarios.
• Figure 2: Finetuning Protocol of Jasmine. The $\textbf{I}_t$ and $\textbf{I}_m$ are each concatenated with $\mathbf{n}$e2eft and fed into the VAE encoder $\varepsilon$. Next, the U-Net performs single-step denoising guided by the task switcher $s$, and subsequently decodes the SSI-depth prediction $D_{SSI}$ and the reconstructed image with the $\mathcal{D}$ (Sec. \ref{['mir']}). Afterward, the $D_{SSI}$ is processed by the SSG for distribution refinement, yielding the final depth estimation $D_{SI}$. The $L_{tc}, L_{e}, L_{ph}$ and $L_{s}$ are supervision loss and they are detailed in Sec. \ref{['steady']}. The edge extraction module is detailed in Sec \ref{['sup:loss']}
• Figure 3: The attempts to preserve the SD prior. The meanings of (a)-(f) are detailed in Sec. \ref{['mir']}. Notably, while (e) demonstrates superior visual quality, it erroneously interprets surface textures (e.g., house windows) as depth edges. (g) shows the performance variations under different $\lambda$ settings for photometric supervision (Eq. \ref{['loss2']}) and latent supervision (Eq. \ref{['loss1']}). The complete metrics and their definitions are provided in Sec. \ref{['sup:mirabla']}.
• Figure 4: (a): Model Structure of SSG. It corresponds to the gray rectangle shown in Fig. \ref{['pipeline']}, standing for an iteration within two consecutive ones. The pipeline of SSG is comprehensively described in Sec. \ref{['dssg']} (DepthHead is omit in (a) for clear). (b): Depth distribution alignment visualization. We statistically analyze each stage of Jasmine's SSG module on the KITTI test set. The standard SI and SSI depths are obtained by applying Eq. \ref{['eq:ssi']} and dividing the maximum value to the depth GT, respectively.
• Figure 5: Qualitative results on KITTI, DrivingStereo, and CityScape datasets. We compare Jasmine with the most generalizable and best-performing SSMDE methods in both in-domain and zero-shot scenarios.