Bilateral Propagation Network for Depth Completion

TL;DR

This work addresses depth completion from sparse depth measurements by introducing BP-Net, which propagates depth at the earliest stage using a Bilateral Propagation Module. The Coefficients for propagation are generated by an MLP conditioned on radiometric difference and spatial distance, enabling edge-preserving, content-aware depth propagation. The model is built as a multi-scale, three-stage pipeline (Pre, MF, Post) with a specialized depth parameterization and multi-scale loss, achieving state-of-the-art results on NYUv2 and KITTI depth completion. Ablation studies validate the importance of early-stage propagation, the learning-based coefficient generation, and the multi-scale design, with code and models released for reproducibility.

Abstract

Depth completion aims to derive a dense depth map from sparse depth measurements with a synchronized color image. Current state-of-the-art (SOTA) methods are predominantly propagation-based, which work as an iterative refinement on the initial estimated dense depth. However, the initial depth estimations mostly result from direct applications of convolutional layers on the sparse depth map. In this paper, we present a Bilateral Propagation Network (BP-Net), that propagates depth at the earliest stage to avoid directly convolving on sparse data. Specifically, our approach propagates the target depth from nearby depth measurements via a non-linear model, whose coefficients are generated through a multi-layer perceptron conditioned on both \emph{radiometric difference} and \emph{spatial distance}. By integrating bilateral propagation with multi-modal fusion and depth refinement in a multi-scale framework, our BP-Net demonstrates outstanding performance on both indoor and outdoor scenes. It achieves SOTA on the NYUv2 dataset and ranks 1st on the KITTI depth completion benchmark at the time of submission. Experimental results not only show the effectiveness of bilateral propagation but also emphasize the significance of early-stage propagation in contrast to the refinement stage. Our code and trained models will be available on the project page.

Figures (6)

• Figure 1: Depth completion is to produce a dense depth map from sparse depth measurements with a synchronized color image. (a) Earlier methods take a 1-stage approach to solve this problem by multi-model fusion network (MF.). (b) Some recent works include a post-processing stage (Post.) to refine the results. (c) Our BP-Net further introduces a pre-processing stage (Pre.) to make the multi-modal fusion (MF.) network more effective for better results.
• Figure 2: Overview of the proposed approach. Dense depth is estimated in a multi-scale scheme, from low-resolution to high-resolution. In each scale, dense depth map is produced from sparse depth map and image feature by 3-stage processing depicted in \ref{['fig:3stages']}.
• Figure 3: Bilateral Propagation Module. The target depth is a convex combination of depth candidates (Cand.) weighted by coefficients (Coeff.), which are yielded by a MLP $\mathbb{F}$ conditioned on both spatial distance (S.D.) and radiometric difference (R.D.) from the nearby depth measurements.
• Figure 4: Qualitative comparison with 'GuideNet' guidenet, 'NLSPN' nlspn and 'CFormer' cformer on indoor and outdoor scenes. Sparse depth points are enlarged for better visualization. Our method is shown in the last row, with highlighted rectangles for easy comparison.
• Figure 5: The RMSE performance under different numbers of neighboring pixels.