DFormerv2: Geometry Self-Attention for RGBD Semantic Segmentation

TL;DR

DFormerv2 addresses RGB-D semantic segmentation by reframing depth as geometry priors rather than as learnable depth embeddings. It introduces Geometry Self-Attention (GSA), which modulates attention weights with depth-derived and spatial priors, and uses axis-decomposed attention to maintain efficiency. The four-stage pyramid encoder builds multi-scale features while depth-derived priors are generated per stage; a lightweight decoder yields strong segmentation results with lower computational cost. Empirical results on NYU DepthV2, SUNRGBD, and Deliver show state-of-the-art performance and favorable accuracy–cost trade-offs, with ablations confirming the benefit of combining depth and spatial priors and the effectiveness of memory-based fusion. Overall, the work demonstrates that explicit 3D geometry cues can be leveraged in transformers to enhance RGB-D scene understanding in challenging conditions.

Abstract

Recent advances in scene understanding benefit a lot from depth maps because of the 3D geometry information, especially in complex conditions (e.g., low light and overexposed). Existing approaches encode depth maps along with RGB images and perform feature fusion between them to enable more robust predictions. Taking into account that depth can be regarded as a geometry supplement for RGB images, a straightforward question arises: Do we really need to explicitly encode depth information with neural networks as done for RGB images? Based on this insight, in this paper, we investigate a new way to learn RGBD feature representations and present DFormerv2, a strong RGBD encoder that explicitly uses depth maps as geometry priors rather than encoding depth information with neural networks. Our goal is to extract the geometry clues from the depth and spatial distances among all the image patch tokens, which will then be used as geometry priors to allocate attention weights in self-attention. Extensive experiments demonstrate that DFormerv2 exhibits exceptional performance in various RGBD semantic segmentation benchmarks. Code is available at: https://github.com/VCIP-RGBD/DFormer.

Figures (9)

• Figure 1: Comparisons among the main RGBD segmentation pipelines and our approach. (a) Use dual encoders to encode RGB and depth respectively and design fusion modules to fusion them zhang2022cmxjia2024geminifusion; (b) Adopt an unified RGBD encoder to extract and fuse RGBD features bachmann2022multimaeyin2023dformer; (c) Our DFormerv2 use depth to form a geometry prior of the scene and then enhance the visual features.
• Figure 2: Comparison between geometry self-attention (GSA) and other attention mechanisms, i.e., vanilla attention dosovitskiy2021vit, window attention liu2021swindong2022cswin, and local attention yang2021focalwang2021crossformer. The 'star' sign means the current query's position. In GSA, colors closer to red represent smaller decay rates, while colors farer away red represent larger ones. In other attention mechanisms, the bright color means the receptive field.
• Figure 3: Illustration of the geometry prior. (a) Generation process of the geometry prior. (b) Some visualization of the geometry prior, where the 'blue star' means the current query.
• Figure 4: Illustration of our DFormerv2. (a) Overall architecture of our DFormerv2, which contains an encoder with pyramid structure and a decoder head that receives input from the last three stage features. (b) Detailed structure of the basic building block. (c) Detailed illustration of the proposed geometry self-attention mechanism.
• Figure 5: Performance-computation comparisons between our DFormerv2 and other SOTA methods on NYU DepthV2 silberman2012nyu_dataset.