Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HDBFormer: Efficient RGB-D Semantic Segmentation with A Heterogeneous Dual-Branch Framework

Shuobin Wei, Zhuang Zhou, Zhengan Lu, Zizhao Yuan, Binghua Su

TL;DR

HDBFormer tackles RGB-D indoor semantic segmentation by acknowledging modality differences and employing a heterogeneous dual-branch architecture. The method combines RGB two-stream encoders with a lightweight LDFormer for depth, linked by the Modality Information Interaction Module (MIIM) that fuses global and local information across modalities. Empirical results on NYUDepthv2 and SUN-RGBD show state-of-the-art performance, validating both the depth-efficient encoder and the targeted fusion strategy. The contributions—LDFormer, MIIM, and graded feature processing with iterative fusion—offer a robust framework for efficient multimodal fusion in complex indoor scenes and can extend to other cross-modal applications.

Abstract

In RGB-D semantic segmentation for indoor scenes, a key challenge is effectively integrating the rich color information from RGB images with the spatial distance information from depth images. However, most existing methods overlook the inherent differences in how RGB and depth images express information. Properly distinguishing the processing of RGB and depth images is essential to fully exploiting their unique and significant characteristics. To address this, we propose a novel heterogeneous dual-branch framework called HDBFormer, specifically designed to handle these modality differences. For RGB images, which contain rich detail, we employ both a basic and detail encoder to extract local and global features. For the simpler depth images, we propose LDFormer, a lightweight hierarchical encoder that efficiently extracts depth features with fewer parameters. Additionally, we introduce the Modality Information Interaction Module (MIIM), which combines transformers with large kernel convolutions to interact global and local information across modalities efficiently. Extensive experiments show that HDBFormer achieves state-of-the-art performance on the NYUDepthv2 and SUN-RGBD datasets. The code is available at: https://github.com/Weishuobin/HDBFormer.

HDBFormer: Efficient RGB-D Semantic Segmentation with A Heterogeneous Dual-Branch Framework

TL;DR

HDBFormer tackles RGB-D indoor semantic segmentation by acknowledging modality differences and employing a heterogeneous dual-branch architecture. The method combines RGB two-stream encoders with a lightweight LDFormer for depth, linked by the Modality Information Interaction Module (MIIM) that fuses global and local information across modalities. Empirical results on NYUDepthv2 and SUN-RGBD show state-of-the-art performance, validating both the depth-efficient encoder and the targeted fusion strategy. The contributions—LDFormer, MIIM, and graded feature processing with iterative fusion—offer a robust framework for efficient multimodal fusion in complex indoor scenes and can extend to other cross-modal applications.

Abstract

In RGB-D semantic segmentation for indoor scenes, a key challenge is effectively integrating the rich color information from RGB images with the spatial distance information from depth images. However, most existing methods overlook the inherent differences in how RGB and depth images express information. Properly distinguishing the processing of RGB and depth images is essential to fully exploiting their unique and significant characteristics. To address this, we propose a novel heterogeneous dual-branch framework called HDBFormer, specifically designed to handle these modality differences. For RGB images, which contain rich detail, we employ both a basic and detail encoder to extract local and global features. For the simpler depth images, we propose LDFormer, a lightweight hierarchical encoder that efficiently extracts depth features with fewer parameters. Additionally, we introduce the Modality Information Interaction Module (MIIM), which combines transformers with large kernel convolutions to interact global and local information across modalities efficiently. Extensive experiments show that HDBFormer achieves state-of-the-art performance on the NYUDepthv2 and SUN-RGBD datasets. The code is available at: https://github.com/Weishuobin/HDBFormer.

Paper Structure

This paper contains 11 sections, 8 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. methodology
  3. Encoder Module
  4. Modality Information Interaction Module
  5. Decoder Module
  6. experiment
  7. Dataset and Evaluation Metrics
  8. Implementation Details
  9. Comparison with State-of-the-Art Method
  10. Ablation Studies
  11. Conclusion

Figures (3)

  • Figure 1: The difference between traditional fusion frameworks and HDBFormer fusion framework.
  • Figure 2: The network structure diagram of the HDBFormer.
  • Figure 3: The segmentation results of HDBFormer on the NYUDepthv2 dataset are compared with CMX zhang2023cmx, DFormer yin2023dformer and DPLNet dong2023efficient.