Gaussian Belief Propagation Network for Depth Completion

TL;DR

Depth completion under sparse, irregular measurements is addressed by GBPN, a hybrid framework that learns a scene-specific MRF via GMCN and performs dense depth inference with Gaussian Belief Propagation. The MRF adapts its data terms, smoothness, and even non-local edges based on image content, while GBP yields a probabilistic depth distribution $X \sim \mathcal{N}(\mu, \Lambda^{-1})$ with an explicit confidence measure $\Lambda$. A probability-based loss supervises both depth estimates and their uncertainty, enabling robust end-to-end training. Across indoor and outdoor benchmarks, GBPN achieves state-of-the-art accuracy and demonstrates strong robustness to varying sparsity patterns and cross-domain generalization, with practical benefits for downstream tasks requiring reliable depth and uncertainty estimates.

Abstract

Depth completion aims to predict a dense depth map from a color image with sparse depth measurements. Although deep learning methods have achieved state-of-the-art (SOTA), effectively handling the sparse and irregular nature of input depth data in deep networks remains a significant challenge, often limiting performance, especially under high sparsity. To overcome this limitation, we introduce the Gaussian Belief Propagation Network (GBPN), a novel hybrid framework synergistically integrating deep learning with probabilistic graphical models for end-to-end depth completion. Specifically, a scene-specific Markov Random Field (MRF) is dynamically constructed by the Graphical Model Construction Network (GMCN), and then inferred via Gaussian Belief Propagation (GBP) to yield the dense depth distribution. Crucially, the GMCN learns to construct not only the data-dependent potentials of MRF but also its structure by predicting adaptive non-local edges, enabling the capture of complex, long-range spatial dependencies. Furthermore, we enhance GBP with a serial \& parallel message passing scheme, designed for effective information propagation, particularly from sparse measurements. Extensive experiments demonstrate that GBPN achieves SOTA performance on the NYUv2 and KITTI benchmarks. Evaluations across varying sparsity levels, sparsity patterns, and datasets highlight GBPN's superior performance, notable robustness, and generalizable capability.

Figures (10)

• Figure 1: Overview of the proposed approach. Markov Random Field (MRF) is constructed depending on parameters dynamically generated from the Graphical Model Construction Network (GMCN), and then optimized via Gaussian Belief Propagation (GBP) for the distribution of dense depth map.
• Figure 2: RMSE($mm$) on NYUv2 testset under various sparsity.
• Figure 3: Qualitative comparisons under different sparsity. All comparing methods are trained with 500 depth points and directly tested with depth input from various sparsity levels.
• Figure 4: Visualization of estimated depth in Gaussian with $\mu$ and $\Lambda$. The results on KITTI are on the left, and results on NYUv2 are on the right.
• Figure 5: Visualization of intermediate results under the serial propagation of Gaussian Belief Propagation. Messages are propagated iteratively corresponding to local directional sweeps: left-to-right, top-to-bottom, right-to-left, and bottom-to-top.