Enhancing Monocular Depth Estimation with Multi-Source Auxiliary Tasks

TL;DR

Monocular Depth Estimation (MDE) remains data-hungry due to limited high-quality labels. The authors propose an alternating training scheme that uses auxiliary datasets from related vision tasks while keeping a pre-trained Vision Foundation backbone frozen, sharing a DPT decoder, and weighting the main MDE task with a task-focusing parameter $\alpha$. They demonstrate robust average improvements of about 11% on in-domain MDE datasets and substantial data-efficiency gains (80–99% fewer labeled samples), with Multi-Label Dense Classification (MLDC) often outperforming semantic segmentation as an auxiliary task. The approach transfers across multiple datasets and backbones (including Depth Anything), though domain mismatch can limit gains on out-of-domain data like KITTI. These results underscore the value of carefully selecting auxiliary tasks and datasets to boost MDE without backbone fine-tuning, offering a data-efficient path for leveraging related tasks in depth estimation.

Abstract

Monocular depth estimation (MDE) is a challenging task in computer vision, often hindered by the cost and scarcity of high-quality labeled datasets. We tackle this challenge using auxiliary datasets from related vision tasks for an alternating training scheme with a shared decoder built on top of a pre-trained vision foundation model, while giving a higher weight to MDE. Through extensive experiments we demonstrate the benefits of incorporating various in-domain auxiliary datasets and tasks to improve MDE quality on average by ~11%. Our experimental analysis shows that auxiliary tasks have different impacts, confirming the importance of task selection, highlighting that quality gains are not achieved by merely adding data. Remarkably, our study reveals that using semantic segmentation datasets as Multi-Label Dense Classification (MLDC) often results in additional quality gains. Lastly, our method significantly improves the data efficiency for the considered MDE datasets, enhancing their quality while reducing their size by at least 80%. This paves the way for using auxiliary data from related tasks to improve MDE quality despite limited availability of high-quality labeled data. Code is available at https://jugit.fz-juelich.de/ias-8/mdeaux.

Figures (11)

• Figure 1: (Top) NYUv2 results with MIX6 auxiliary MLDC. From left to right: input, ground truth, error difference (w.r.t. ground truth) between the DINOv2 baseline and ours. Green indicates ours is better, while red vice versa. (Bottom) AbsRel ($\downarrow$) for varying values of the task focusing parameter $\alpha$ with multiple tasks (markers). The solid and dashed lines represent the mean and standard error of DINOv2, respectively.
• Figure 2: Overview of the proposed training pipeline. We use a frozen pre-trained DINOv2 ViT-G backbone ($f$) as a feature extractor and jointly train a DPT decoder ($g_\theta$) with auxiliary datasets from related vision tasks (semantic segmentation, dense classification, or image reconstruction). We train 2 task-specific heads: MDE ($h_\phi$) and auxiliary ($h_\psi$). We alternate MDE steps and auxiliary steps. During backpropagation, each head has its own learning rate ($\eta_{h_\phi}$ and $\eta_{h_\psi}$), while the decoder shares a common learning rate $\eta_{g_\theta}$, scaled by $\alpha$ for MDE and $1 - \alpha$ for the auxiliary task.
• Figure 3: Absolute Relative Error (AbsRel) of MDE on NYUv2, SUN RGBD and DIODE Outdoor using the DINOv2 baseline and our method with multiple auxiliary datasets and tasks. Dots and bars depict the mean and standard error of AbsRel ($\downarrow$), respectively.
• Figure 4: AbsRel of models trained with various fractions of the dataset. The dataset sizes are reported in \ref{['tab:datasets_depth']}. DINOv2 baseline (red circles) represents the model trained without auxiliary tasks, whereas Proposed (blue squares) depicts our method jointly trained with MIX6 MLDC auxiliary task with $\alpha=0.9$.
• Figure 5: AbsRel ($\downarrow$) when varying the learning rate. (Left) Learning rate tuning for the DINOv2 baseline by a factor $\gamma$. (Right) Our method re-designed to do unscaled depth steps and auxiliary steps scaled by a factor $\beta$, using MIX6 MLDC task. For both plots, dashed and dotted-dashed lines represent the performance of the baseline ($\gamma=1$) and of our method with $\alpha=0.9$, respectively.