GeoDiff: Geometry-Guided Diffusion for Metric Depth Estimation

TL;DR

GeoDiff tackles metric depth estimation by marrying pretrained diffusion priors with stereo geometry in a training-free, test-time optimization framework. By reframing depth as an inverse problem and parameterizing metric depth through a learnable scale and shift, GeoDiff uses geometry-guided diffusion with a differentiable stereo reprojection term to recover accurate metric depths from monocular inputs. The approach demonstrates strong zero-shot performance across indoor, outdoor, and challenging specular/translucent scenes, often surpassing or matching state-of-the-art methods without retraining and while handling arbitrary two-view configurations. Its use of a diffusion prior as an implicit regularizer and its reliance on photometric guidance rather than explicit smoothness constraints contribute to finer depth detail and robust handling of non-Lambertian surfaces, albeit with slower inference and dependence on quality priors. Overall, GeoDiff provides a practical, plug-and-play pathway to metric depth from monocular data when stereo cues are available, broadening the applicability of diffusion-based depth priors in real-world scenarios.

Abstract

We introduce a novel framework for metric depth estimation that enhances pretrained diffusion-based monocular depth estimation (DB-MDE) models with stereo vision guidance. While existing DB-MDE methods excel at predicting relative depth, estimating absolute metric depth remains challenging due to scale ambiguities in single-image scenarios. To address this, we reframe depth estimation as an inverse problem, leveraging pretrained latent diffusion models (LDMs) conditioned on RGB images, combined with stereo-based geometric constraints, to learn scale and shift for accurate depth recovery. Our training-free solution seamlessly integrates into existing DB-MDE frameworks and generalizes across indoor, outdoor, and complex environments. Extensive experiments demonstrate that our approach matches or surpasses state-of-the-art methods, particularly in challenging scenarios involving translucent and specular surfaces, all without requiring retraining.

Figures (13)

• Figure 1: Results on the challenging Booster zamaramirez2024booster dataset. Comparison of depth estimation performance on transparent and reflective objects. From left to right: input image, followed by depth maps from GeoDiff (ours), DepthPro bochkovskii2024depth, Marigold ke2024repurposing, and IGEV-Stereo xu2023iterative. Our method generates sharp, accurate metric depth maps in a zero-shot setting, leveraging stereo pairs for improved depth recovery.
• Figure 2: Top. Prior methods ke2024repurposingfu2024geowizard focus on fine-tuning diffusion models to estimate relative depth. Bottom. In contrast, our approach, without retraining, combines a pretrained monocular model with geometric guidance from stereo cues to directly predict metric depth in meters, achieving superior accuracy in even challenging scenes such as transparent and reflective surfaces.
• Figure 3: Overview of the proposed framework. GeoDiff is built upon DDIM song2020denoising sampling (black arrow) process with geometric guidance (yellow arrow), taking a stereo pair (or two views with known poses) as input and producing metric depth in meters for the left image. The process begins by encoding the left image through a VAE encoder and concatenating it with random noise to form the depth latent. During sampling, the one-step latent prediction $\hat{z}_0^{(x)}$ is computed using Tweedie's formula and decoded to pixel space. This prediction is then transformed to metric scale depth $\tilde{x}_0$ via learnable scale $\hat{s}$ and shift $\hat{t}$ parameters. A differentiable warping module $\bm{P}_{\bm{y}_2 \rightarrow \bm{y}_1}\left(\tilde{\bm{x}}_0, \bm{y}_2\right)$ leverages camera parameters to re-render the left image (detailed in Section \ref{['subsec:stereo-reproj']} and Supplemental \ref{['subsec:supp_diff_warping']}). The sampling process is guided by minimizing a geometric loss $\mathcal{L}_{geo}$ defined in Equation \ref{['eq:geo-loss']}.
• Figure 4: Qualitative results. Our method showcases superior depth prediction compared to competing methods. For instance, in the challenging Mirror scene (second column), our model accurately predicts the reflective surface, outperforming both GeoWizard fu2024geowizard and RAFT-Stereo, which struggle in such cases. Additionally, our approach preserves finer details (last column), showcasing the effectiveness of our proposed geometric guidance.
• Figure 5: From top to bottom. Top: Two known-pose arbitrary images. Middle: Depth map predictions of our method and Dust3r wang2024dust3r. Bottom: Our point cloud and Dust3r point cloud.