Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Do More With What You Have: Transferring Depth-Scale from Labeled to Unlabeled Domains

Alexandra Dana, Nadav Carmel, Amit Shomer, Ofer Manela, Tomer Peleg

TL;DR

The paper tackles the challenge of obtaining absolute depth predictions in a new domain without target GT depths and with potential sensor intrinsics differences. It proposes a lightweight, cross-domain depth-scale transfer (DST) framework that leverages self-supervised monocular depth estimation trained on mixed-source and target data, together with a single learning factor $G_{dscale}$ derived from a linear mapping between source up-to-scale depths and ground-truth depths via Theil-Sen regression. By aligning the field-of-view across domains and exploiting the observed linear depth ranking, the authors demonstrate that absolute depth can be transferred from source GT-rich datasets to target domains, achieving competitive AbsRel scores on KITTI, DDAD, and nuScenes without target GTs. The approach is flexible, practical for continuous learning, and reduces reliance on tailored synthetic data, having broad implications for scalable, sensor-agnostic depth understanding in real-world applications.

Abstract

Transferring the absolute depth prediction capabilities of an estimator to a new domain is a task with significant real-world applications. This task is specifically challenging when images from the new domain are collected without ground-truth depth measurements, and possibly with sensors of different intrinsics. To overcome such limitations, a recent zero-shot solution was trained on an extensive training dataset and encoded the various camera intrinsics. Other solutions generated synthetic data with depth labels that matched the intrinsics of the new target data to enable depth-scale transfer between the domains. In this work we present an alternative solution that can utilize any existing synthetic or real dataset, that has a small number of images annotated with ground truth depth labels. Specifically, we show that self-supervised depth estimators result in up-to-scale predictions that are linearly correlated to their absolute depth values across the domain, a property that we model in this work using a single scalar. In addition, aligning the field-of-view of two datasets prior to training, results in a common linear relationship for both domains. We use this observed property to transfer the depth-scale from source datasets that have absolute depth labels to new target datasets that lack these measurements, enabling absolute depth predictions in the target domain. The suggested method was successfully demonstrated on the KITTI, DDAD and nuScenes datasets, while using other existing real or synthetic source datasets, that have a different field-of-view, other image style or structural content, achieving comparable or better accuracy than other existing methods that do not use target ground-truth depths.

Do More With What You Have: Transferring Depth-Scale from Labeled to Unlabeled Domains

TL;DR

The paper tackles the challenge of obtaining absolute depth predictions in a new domain without target GT depths and with potential sensor intrinsics differences. It proposes a lightweight, cross-domain depth-scale transfer (DST) framework that leverages self-supervised monocular depth estimation trained on mixed-source and target data, together with a single learning factor derived from a linear mapping between source up-to-scale depths and ground-truth depths via Theil-Sen regression. By aligning the field-of-view across domains and exploiting the observed linear depth ranking, the authors demonstrate that absolute depth can be transferred from source GT-rich datasets to target domains, achieving competitive AbsRel scores on KITTI, DDAD, and nuScenes without target GTs. The approach is flexible, practical for continuous learning, and reduces reliance on tailored synthetic data, having broad implications for scalable, sensor-agnostic depth understanding in real-world applications.

Abstract

Transferring the absolute depth prediction capabilities of an estimator to a new domain is a task with significant real-world applications. This task is specifically challenging when images from the new domain are collected without ground-truth depth measurements, and possibly with sensors of different intrinsics. To overcome such limitations, a recent zero-shot solution was trained on an extensive training dataset and encoded the various camera intrinsics. Other solutions generated synthetic data with depth labels that matched the intrinsics of the new target data to enable depth-scale transfer between the domains. In this work we present an alternative solution that can utilize any existing synthetic or real dataset, that has a small number of images annotated with ground truth depth labels. Specifically, we show that self-supervised depth estimators result in up-to-scale predictions that are linearly correlated to their absolute depth values across the domain, a property that we model in this work using a single scalar. In addition, aligning the field-of-view of two datasets prior to training, results in a common linear relationship for both domains. We use this observed property to transfer the depth-scale from source datasets that have absolute depth labels to new target datasets that lack these measurements, enabling absolute depth predictions in the target domain. The suggested method was successfully demonstrated on the KITTI, DDAD and nuScenes datasets, while using other existing real or synthetic source datasets, that have a different field-of-view, other image style or structural content, achieving comparable or better accuracy than other existing methods that do not use target ground-truth depths.
Paper Structure (15 sections, 5 equations, 4 figures, 4 tables)

This paper contains 15 sections, 5 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Method
  4. Depth prediction architecture
  5. Adjusting the FOV of the source data to the target domain
  6. Estimating the depth scaling factor using a linear estimator
  7. Overview of our solution
  8. Datasets
  9. Depth evaluation metrics
  10. Results
  11. Studying per-domain the relationship between GT and predicted up-to-scale depths
  12. Training the MDE on images from two domains
  13. Straightforward attempts for depth-scale-transfer between domains
  14. Accuracy comparison to other methods
  15. Discussion

Figures (4)

  • Figure 1: Demonstrating various methods for achieving absolute depth predictions on the DDAD target dataset (front camera), when no GT depth labels from DDAD are available for training. The $AbsRel$ metric (lower is better) was measured on the DDAD validation dataset for each presented method. Mixed-supervised models ($\square$) were trained using self-supervision on both DDAD and another source dataset (KITTI or Parallel Domain (PD)) and with full-supervision on source GT depths using the mentioned loss.
  • Figure 2: Methods for inferring absolute depth from single target RGB images, when no target GT depth measurements are available.
  • Figure 3: Overview of our solution. (A) The FOV of source data is adjusted to the target FOV. (B) The depth and pose networks are trained using self-supervision on both source and target training images (mixed batches). (C) Data from the source domain is used to generate the GT vs. predicted up-to-scale depths mapping. We apply the linear Theil-Sen regressor to calculate the $G_{dscale}$ depth-scaling factor for the trained MDE. (D) Estimated target up-to-scale depths are multiplied by the $G_{dscale}$ factor, resulting in absolute depth predictions.
  • Figure 4: Scatter plots of the GT vs. the predicted up-to-scale depth values. Our MDE was separately trained on various datasets using self-supervision and inferred on test images from the same domain. (A) All test data. The red line depicts the linear fit applied on all data (see Eq. (\ref{['eq:GT_to_pred_fitting']}). (B) Predictions with $AbsRel_{norm}>15\%$ were filtered out and the fitting was recalculated. (C) The GT vs. the predicted depth relationship was similarly linearly fitted per image and depicted by different green lines.