RSA: Resolving Scale Ambiguities in Monocular Depth Estimators through Language Descriptions

TL;DR

The method, RSA, takes as input a text caption describing objects present in an image and outputs the parameters of a linear transformation which can be applied globally to a relative depth map to yield metric-scaled depth predictions that are comparable to an upper bound of fitting relative depth to ground truth via a linear transformation.

Abstract

We propose a method for metric-scale monocular depth estimation. Inferring depth from a single image is an ill-posed problem due to the loss of scale from perspective projection during the image formation process. Any scale chosen is a bias, typically stemming from training on a dataset; hence, existing works have instead opted to use relative (normalized, inverse) depth. Our goal is to recover metric-scaled depth maps through a linear transformation. The crux of our method lies in the observation that certain objects (e.g., cars, trees, street signs) are typically found or associated with certain types of scenes (e.g., outdoor). We explore whether language descriptions can be used to transform relative depth predictions to those in metric scale. Our method, RSA, takes as input a text caption describing objects present in an image and outputs the parameters of a linear transformation which can be applied globally to a relative depth map to yield metric-scaled depth predictions. We demonstrate our method on recent general-purpose monocular depth models on indoors (NYUv2, VOID) and outdoors (KITTI). When trained on multiple datasets, RSA can serve as a general alignment module in zero-shot settings. Our method improves over common practices in aligning relative to metric depth and results in predictions that are comparable to an upper bound of fitting relative depth to ground truth via a linear transformation.

Figures (5)

• Figure 1: Can we infer the scale of 3D scenes from their descriptions? Consider the description above, one may observe that the scale of the 3D scene is closely related to the objects (and their typical sizes) populating it.
• Figure 2: Overview. We infer scale and shift from the language description of an image to transform the inverse relative depth from the depth model into metric depth (absolute depth in meters) prediction.
• Figure 3: Left: Predicted inverse scale w.r.t. median depth ground truth. Larger scenes tend to have larger median ground truth depth. For RSA trained on combined KITTI and NYUv2 with Depth Anything model, we fit an inverse proportional function for the predicted inverse scale in the test set (each point is an image), to verify that the scale is proportional to the median depth, that larger scenes are predicted with larger scales.
• Figure 4: Visualization of depth estimations on NYUv2. Building upon DPT, while a better scale factor does not change the structure of the depth prediction, leading to visually similar depth maps, it significantly reduces the overall error (darker in the error map). Note: Zeros in ground truth indicate the absence of valid depth values.
• Figure 5: Visualization of depth estimations on KITTI. Building upon DPT, while a better scale factor does not change the structure of the depth prediction, it significantly reduces the overall error (darker in the error map). Note: Zeros in ground truth depth indicate the absence of valid depth values.