Learning Pixel-wise Continuous Depth Representation via Clustering for Depth Completion

TL;DR

This paper tackles depth completion by addressing the limitations of classification-based depth representations, which discretize depth into pixel-shared categories and cause boundary smearing. It introduces CluDe, a clustering-based framework that learns pixel-wise continuous depth by first deriving a dataset-wide set of depth centers via a clustering transformer and then refining with per-pixel depth offsets through a hierarchical translation module. The method implicitly learns initial depth centers and progressively translates depth guidance to achieve continuous depth representations, yielding sharp boundaries and competitive performance on KITTI and VOID, including state-of-the-art results among classification-based approaches. The work demonstrates that representing depth with adaptive centers and offsets can better capture real depth distributions and has practical implications for robust depth completion in real-world scenes.

Abstract

Depth completion is a long-standing challenge in computer vision, where classification-based methods have made tremendous progress in recent years. However, most existing classification-based methods rely on pre-defined pixel-shared and discrete depth values as depth categories. This representation fails to capture the continuous depth values that conform to the real depth distribution, leading to depth smearing in boundary regions. To address this issue, we revisit depth completion from the clustering perspective and propose a novel clustering-based framework called CluDe which focuses on learning the pixel-wise and continuous depth representation. The key idea of CluDe is to iteratively update the pixel-shared and discrete depth representation to its corresponding pixel-wise and continuous counterpart, driven by the real depth distribution. Specifically, CluDe first utilizes depth value clustering to learn a set of depth centers as the depth representation. While these depth centers are pixel-shared and discrete, they are more in line with the real depth distribution compared to pre-defined depth categories. Then, CluDe estimates offsets for these depth centers, enabling their dynamic adjustment along the depth axis of the depth distribution to generate the pixel-wise and continuous depth representation. Extensive experiments demonstrate that CluDe successfully reduces depth smearing around object boundaries by utilizing pixel-wise and continuous depth representation. Furthermore, CluDe achieves state-of-the-art performance on the VOID datasets and outperforms classification-based methods on the KITTI dataset.

Figures (12)

• Figure 1: Results of classification-based and clustering-based methods. It is observed that the clustering-based method produces more accurate logistic scores, thereby preserving the shape of the object while effectively suppressing stripe artifacts between the person and the wall.
• Figure 2: In the classification-based framework, the main purpose is to predict logistic scores for pre-defined depth categories. In contrast, the proposed clustering-based framework focuses on obtaining clusters via depth centers. Additionally, we estimate offsets to improve the continuity of depth for each pixel.
• Figure 3: The architecture of CluDe. Given a pair of sparse depth map and RGB image, CluDe first extracts multi-scale features. Then, the lowest scale feature is fed into a clustering transformer, which groups tokens into depth clusters and predicts depth result. Next, a hierarchical translation module estimates depth offsets to adjust depth guidance for refinement. For clarity, we omit some features used in depth predication and iteration.
• Figure 4: The pipeline of clustering. Self-attention is used to propagate information between depth centers and embedding tokens, while cross-attention is utilized to group similar tokens together. This process updates initial depth centers and simultaneously generates depth clusters.
• Figure 5: From top to bottom, we show the RGB image, sparse depth map, and corrected sparse depth map. The region containing outliers is highlighted with a red box, and a zoomed-in view is provided on the right.