Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Simple yet Effective Test-Time Adaptation for Zero-Shot Monocular Metric Depth Estimation

Rémi Marsal, Alexandre Chapoutot, Philippe Xu, David Filliat

TL;DR

This work tackles the scale ambiguity in zero-shot monocular metric depth estimation by introducing a plug-and-play test-time rescaling method. It regresses affine parameters from a few sparse 3D reference points to convert an affine-invariant disparity into metric depth, using bilinear sampling and robust RANSAC-based regression without any fine-tuning. Across indoor and outdoor benchmarks, the approach yields substantial improvements over other zero-shot methods and demonstrates robustness to sparse-depth density, calibration errors, and noisy measurements, while maintaining favorable inference costs. The method remains compatible with foundation-model depth predictors like Depth Anything, offering a cost-effective pathway to accurate metric depth in diverse environments, with code available publicly.

Abstract

The recent development of \emph{foundation models} for monocular depth estimation such as Depth Anything paved the way to zero-shot monocular depth estimation. Since it returns an affine-invariant disparity map, the favored technique to recover the metric depth consists in fine-tuning the model. However, this stage is not straightforward, it can be costly and time-consuming because of the training and the creation of the dataset. The latter must contain images captured by the camera that will be used at test time and the corresponding ground truth. Moreover, the fine-tuning may also degrade the generalizing capacity of the original model. Instead, we propose in this paper a new method to rescale Depth Anything predictions using 3D points provided by sensors or techniques such as low-resolution LiDAR or structure-from-motion with poses given by an IMU. This approach avoids fine-tuning and preserves the generalizing power of the original depth estimation model while being robust to the noise of the sparse depth, of the camera-LiDAR calibration or of the depth model. Our experiments highlight enhancements relative to zero-shot monocular metric depth estimation methods, competitive results compared to fine-tuned approaches and a better robustness than depth completion approaches. Code available at github.com/ENSTA-U2IS-AI/depth-rescaling.

A Simple yet Effective Test-Time Adaptation for Zero-Shot Monocular Metric Depth Estimation

TL;DR

This work tackles the scale ambiguity in zero-shot monocular metric depth estimation by introducing a plug-and-play test-time rescaling method. It regresses affine parameters from a few sparse 3D reference points to convert an affine-invariant disparity into metric depth, using bilinear sampling and robust RANSAC-based regression without any fine-tuning. Across indoor and outdoor benchmarks, the approach yields substantial improvements over other zero-shot methods and demonstrates robustness to sparse-depth density, calibration errors, and noisy measurements, while maintaining favorable inference costs. The method remains compatible with foundation-model depth predictors like Depth Anything, offering a cost-effective pathway to accurate metric depth in diverse environments, with code available publicly.

Abstract

The recent development of \emph{foundation models} for monocular depth estimation such as Depth Anything paved the way to zero-shot monocular depth estimation. Since it returns an affine-invariant disparity map, the favored technique to recover the metric depth consists in fine-tuning the model. However, this stage is not straightforward, it can be costly and time-consuming because of the training and the creation of the dataset. The latter must contain images captured by the camera that will be used at test time and the corresponding ground truth. Moreover, the fine-tuning may also degrade the generalizing capacity of the original model. Instead, we propose in this paper a new method to rescale Depth Anything predictions using 3D points provided by sensors or techniques such as low-resolution LiDAR or structure-from-motion with poses given by an IMU. This approach avoids fine-tuning and preserves the generalizing power of the original depth estimation model while being robust to the noise of the sparse depth, of the camera-LiDAR calibration or of the depth model. Our experiments highlight enhancements relative to zero-shot monocular metric depth estimation methods, competitive results compared to fine-tuned approaches and a better robustness than depth completion approaches. Code available at github.com/ENSTA-U2IS-AI/depth-rescaling.

Paper Structure

This paper contains 21 sections, 4 equations, 3 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Monocular depth estimation
  4. Scale estimation for monocular depth
  5. Zero-shot monocular metric depth estimation
  6. Method
  7. Experiments
  8. Dataset and metrics
  9. Implementation details
  10. Comparison with monocular depth esimation methods
  11. Rescaling with LiDAR
  12. Rescaling with structure-from-motion
  13. Comparison with depth completion methods
  14. Zero-shot depth completion
  15. Robustness to reduction of the sparse depth density
  16. ...and 6 more sections

Figures (3)

  • Figure 1: Illustration of our method. First, an affine-invariant disparity map $d$ is predicted from an image with a neural network such as Depth Anything yang2024depth while in parallel a sensor is used to estimate a set of 3D points $P$. $P$ and the corresponding values in $d$ are then used to estimate the scaling parameters $\alpha$ and $\beta$ using a RANSAC fischler1981random. The parameters are applied to $d$ to recover the metric depth $D$.
  • Figure 2: Quantitative study of the robustness of depth completion methods: (top) impact of the number of LiDAR beams on performance on KITTI kitti, (middle) impact of camera-LiDAR calibration error by applying a horizontal or vertical shift to depth samples on NYUv2 nyu, (bottom) impact of random noise on depth samples on the performance on NYUv2 nyu.
  • Figure 3: Qualitative comparison between Depth Anything V1 yang2024depth and V2 yang2024depthv2. From top to bottom: (1) image from the KITTI dataset kitti, (2) the disparity map predicted by Depth Anything V1 and (3) the one of Depth Anything V2.