Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

M${^2}$Depth: Self-supervised Two-Frame Multi-camera Metric Depth Estimation

Yingshuang Zou, Yikang Ding, Xi Qiu, Haoqian Wang, Haotian Zhang

TL;DR

This work addresses scale-aware, surround-depth estimation for autonomous driving by introducing M$^2$Depth, a self-supervised framework that processes temporally adjacent two-frame inputs from multiple cameras. It constructs spatial-temporal 3D cost volumes via plane-sweep, and fuses them with a spatial-temporal fusion (STF) module, while incorporating SAM priors through a multi-grained feature fusion (MFF) to improve depth edges and details. The model jointly learns pose, monocular priors, and multi-camera depth under weak supervision, augmented by SfM-based scale guidance and adaptive depth sampling. Experiments on DDAD and nuScenes show state-of-the-art performance with favorable memory and computation characteristics, indicating strong potential for reliable, real-time surrounding depth in autonomous systems. A key contribution is the first integration of SAM features into depth estimation, enabling finer semantic-guided depth accuracy across camera views.

Abstract

This paper presents a novel self-supervised two-frame multi-camera metric depth estimation network, termed M${^2}$Depth, which is designed to predict reliable scale-aware surrounding depth in autonomous driving. Unlike the previous works that use multi-view images from a single time-step or multiple time-step images from a single camera, M${^2}$Depth takes temporally adjacent two-frame images from multiple cameras as inputs and produces high-quality surrounding depth. We first construct cost volumes in spatial and temporal domains individually and propose a spatial-temporal fusion module that integrates the spatial-temporal information to yield a strong volume presentation. We additionally combine the neural prior from SAM features with internal features to reduce the ambiguity between foreground and background and strengthen the depth edges. Extensive experimental results on nuScenes and DDAD benchmarks show M${^2}$Depth achieves state-of-the-art performance. More results can be found in https://heiheishuang.xyz/M2Depth .

M${^2}$Depth: Self-supervised Two-Frame Multi-camera Metric Depth Estimation

TL;DR

This work addresses scale-aware, surround-depth estimation for autonomous driving by introducing MDepth, a self-supervised framework that processes temporally adjacent two-frame inputs from multiple cameras. It constructs spatial-temporal 3D cost volumes via plane-sweep, and fuses them with a spatial-temporal fusion (STF) module, while incorporating SAM priors through a multi-grained feature fusion (MFF) to improve depth edges and details. The model jointly learns pose, monocular priors, and multi-camera depth under weak supervision, augmented by SfM-based scale guidance and adaptive depth sampling. Experiments on DDAD and nuScenes show state-of-the-art performance with favorable memory and computation characteristics, indicating strong potential for reliable, real-time surrounding depth in autonomous systems. A key contribution is the first integration of SAM features into depth estimation, enabling finer semantic-guided depth accuracy across camera views.

Abstract

This paper presents a novel self-supervised two-frame multi-camera metric depth estimation network, termed MDepth, which is designed to predict reliable scale-aware surrounding depth in autonomous driving. Unlike the previous works that use multi-view images from a single time-step or multiple time-step images from a single camera, MDepth takes temporally adjacent two-frame images from multiple cameras as inputs and produces high-quality surrounding depth. We first construct cost volumes in spatial and temporal domains individually and propose a spatial-temporal fusion module that integrates the spatial-temporal information to yield a strong volume presentation. We additionally combine the neural prior from SAM features with internal features to reduce the ambiguity between foreground and background and strengthen the depth edges. Extensive experimental results on nuScenes and DDAD benchmarks show MDepth achieves state-of-the-art performance. More results can be found in https://heiheishuang.xyz/M2Depth .
Paper Structure (52 sections, 22 equations, 12 figures, 11 tables)

This paper contains 52 sections, 22 equations, 12 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Multi-frame Depth Estimation
  4. Multi-camera Depth Estimation
  5. SAM and Applications
  6. Methods
  7. Problem Formulation
  8. Network Overview
  9. Mono Prior Estimation
  10. Multi-grained Feature Fusion
  11. Spatial-temporal Cost Volume
  12. Initial Volume Construction
  13. Spatial-temporal Volume Fusion
  14. Depth Prediction
  15. Loss Function
  16. ...and 37 more sections

Figures (12)

  • Figure 1: Point clouds comparison on DDAD guizilini2020ddad_packnet dataset. By transforming the predicted depth into point clouds, we show that our method achieves more consistent and accurate estimation compared with SurroundDepth wei2023surrounddepth. The visualized point clouds are fused using the surrounding depth of one frame, and the blue boxes highlight the challenging area spanning multiple cameras.
  • Figure 2: Overview of M$^2$Depth. Given images $\{\mathbf{I}_{t}^c\}_{c=1}^C$ and $\{\mathbf{I}_{t-1}^c\}_{c=1}^C$ from multiple cameras and two frames, M$^2$Depth first estimates the pose of the front camera $\mathbf{P}_{t \to t-1}$, which will be used to infer the poses of all other cameras $\{\mathbf{P}^0_{t \to t-1}\}_{c=1}^C$. In mono prior estimation, the multi-grained feature fusion (MFF) module aggregates the internal features $\{\mathbf{C}_t^c\}_{c=1}^C$ from image encoder and the SAM features $\{\mathbf{S}_t^c\}_{c=1}^C$ from SAM encoder to improve feature expression in multi-grained. The depth prior and constraints across multiple cameras are employed to construct 3D cost volumes $\{\mathbf{V}_t^c\}_{c=1}^C$ within the temporal-spatial domain, which will be then used by the spatial-temporal fusion (STF) module to strengthen the accuracy and consistency of cost volumes. Finally, the depth decoder takes as inputs the $\{\mathbf{V}_t^c\}_{c=1}^C$ and $\{\mathbf{S}_t^c\}_{c=1}^C$ to produce the surrounding depth.
  • Figure 3: Details of the multi-grained feature fusion (MFF) module. MFF takes the internal features $\{\mathbf{C}_t^c\}_{c=1}^C$ and the SAM features $\{\mathbf{S}_t^c\}_{c=1}^C$ as inputs, and utilizes a $\mathcal{F}_{\mathrm{attn}}$ block to yield the weight map, which fetches the complementary info between $\{\mathbf{C}_t^c\}_{c=1}^C$ and $\{\mathbf{S}_t^c\}_{c=1}^C$, and will be used in $\mathcal{F}_{\mathrm{fusion}}$ block to produce the fused feature $\{\mathbf{M}_t^c\}_{c=1}^C$.
  • Figure 4: Overview of the volume construction and STF module. Given the reference image feature $\mathbf{F}_{t}^{c}$ and its temporal-spatial adjacent features, we first warp the adjacent features to reference view to form the initial volumes $\mathbf{V}_{t,sp}^{c}$ and $\mathbf{V}_{t,tp}^{c}$ in spatial domain and temporal domain respectively. After that, STF fuses the initial volumes by computing the correlation between $\mathbf{F}_{t}^{c}$ and $\mathbf{V}_{t,sp}^{c}$, $\mathbf{V}_{t,tp}^{c}$ and produces the weight maps $\mathbf{W}_{t,sp}^{c}$, $\mathbf{W}_{t,tp}^{c}$, which will be used as fusion weights to guide the volume fusion.
  • Figure 5: Qualitative comparison of predicted surrounding depth on DDAD dataset guizilini2020ddad_packnet. Given the input surrounding images (the top row), we show the visualized depth maps and depth errors of SurroundDepth wei2023surrounddepth and M$^2$Depth. The depth maps are visualized in the range of [$0$, $50m$]. Our method is able to produce more accurate depth with less error and sharper depth edge across multiple cameras.
  • ...and 7 more figures