Language as Prior, Vision as Calibration: Metric Scale Recovery for Monocular Depth Estimation

TL;DR

This work tackles metric depth estimation from monocular images by addressing the global scale ambiguity with a frozen relative-depth backbone. It introduces an envelope-and-selection framework where language, via captions, supplies an uncertainty-aware bound over calibration parameters, while a vision branch selects an image-specific calibration inside that bound using a multiscale feature pyramid. The calibration map is global and affine in inverse depth, with $ ilde{\boldsymbol{\theta}}=\boldsymbol{\mu}(\mathbf{T})+\mathbf{r}(\mathbf{T})\odot\boldsymbol{\delta}(\mathbf{I})$, and $(\alpha,\beta)$ computed as $\alpha=\mathrm{softplus}(\tilde{\alpha})$, $\beta=\beta_{\min}+ (\beta_{\max}-\beta_{\min})\sigma(\tilde{\beta})$, yielding $\widehat{D}(x)=1/\max(\alpha Y(x)+\beta,\varepsilon)$. Training uses an online per-image least-squares oracle to supervise the envelope and calibration, along with depth supervision and regularization. Experiments on NYUv2 and KITTI demonstrate improved metric-depth accuracy, with zero-shot transfer to SUN-RGBD and DDAD showing enhanced robustness to domain shift compared with language-only baselines, highlighting the practical potential of language-guided, vision-conditioned metric grounding without extra sensors. The approach offers a scalable path to robust metric depth in diverse settings by combining language priors with image-specific calibration on a frozen backbone.

Abstract

Relative-depth foundation models transfer well, yet monocular metric depth remains ill-posed due to unidentifiable global scale and heightened domain-shift sensitivity. Under a frozen-backbone calibration setting, we recover metric depth via an image-specific affine transform in inverse depth and train only lightweight calibration heads while keeping the relative-depth backbone and the CLIP text encoder fixed. Since captions provide coarse but noisy scale cues that vary with phrasing and missing objects, we use language to predict an uncertainty-aware envelope that bounds feasible calibration parameters in an unconstrained space, rather than committing to a text-only point estimate. We then use pooled multi-scale frozen visual features to select an image-specific calibration within this envelope. During training, a closed-form least-squares oracle in inverse depth provides per-image supervision for learning the envelope and the selected calibration. Experiments on NYUv2 and KITTI improve in-domain accuracy, while zero-shot transfer to SUN-RGBD and DDAD demonstrates improved robustness over strong language-only baselines.

Figures (2)

• Figure 1: Framework overview. Given an image $\mathbf{I}$, a frozen backbone $\Phi$ outputs inverse relative depth $\mathbf{Y}$ and multi-scale features. The caption $\mathbf{T}$ is auto-generated by LLaVA v1.6. A frozen CLIP text encoder maps $\mathbf{T}$ to an uncertainty-aware envelope $(\boldsymbol{\mu}(\mathbf{T}),\mathbf{r}(\mathbf{T}))$, and a vision-conditioned selector predicts an image-specific calibration within this envelope. Training uses an online inverse-depth least-squares Oracle-Solver to provide per-image targets for envelope consistency, calibration distillation, and depth reconstruction losses.
• Figure 2: Qualitative results on NYUv2. Depth predictions and absolute error maps of our method and RSA are shown for comparison. Both approaches build upon the same DPT backbone and differ only in the calibration strategy. While the overall depth structures remain similar, our method consistently reduces estimation errors across indoor scenes, particularly in regions where scale mismatch is visually prominent.