Depth-guided Texture Diffusion for Image Semantic Segmentation

TL;DR

This paper tackles the modality gap between depth maps and RGB images in image semantic segmentation by introducing depth-guided texture diffusion. It extracts texture in the RGB domain via frequency-domain high-pass filtering, diffuses these cues into depth maps, and enforces structural alignment with RGB data using a structural consistency loss. The resulting texture-enriched depth is fused with RGB through a joint embedding and an adaptor to integrate depth cues across network layers, yielding improved performance on camouflaged object detection, salient object detection, and indoor semantic segmentation, including source-free depth settings. The approach demonstrates state-of-the-art results and robust ablation-supported insights, highlighting the importance of texture in depth for reliable multi-modal fusion and 3D-aware segmentation in real-world scenarios.

Abstract

Depth information provides valuable insights into the 3D structure especially the outline of objects, which can be utilized to improve the semantic segmentation tasks. However, a naive fusion of depth information can disrupt feature and compromise accuracy due to the modality gap between the depth and the vision. In this work, we introduce a Depth-guided Texture Diffusion approach that effectively tackles the outlined challenge. Our method extracts low-level features from edges and textures to create a texture image. This image is then selectively diffused across the depth map, enhancing structural information vital for precisely extracting object outlines. By integrating this enriched depth map with the original RGB image into a joint feature embedding, our method effectively bridges the disparity between the depth map and the image, enabling more accurate semantic segmentation. We conduct comprehensive experiments across diverse, commonly-used datasets spanning a wide range of semantic segmentation tasks, including Camouflaged Object Detection (COD), Salient Object Detection (SOD), and indoor semantic segmentation. With source-free estimated depth or depth captured by depth cameras, our method consistently outperforms existing baselines and achieves new state-of-theart results, demonstrating the effectiveness of our Depth-guided Texture Diffusion for image semantic segmentation.

Figures (8)

• Figure 1: Visual samples illustrating the absence of detailed textures in depth maps compared to the corresponding texture images and ground truth (GT) images.
• Figure 2: The architecture of our main framework. It comprises three primary modules: Texture Extraction (TE), which extracts texture features; Texture Diffusion (TXD), which diffuses texture features within depth maps; and Joint Embedding(JEB), which performs joint embedding of the texture-enriched depth and RGB images for improved feature integration.
• Figure 3: The architecture of our Texture Extraction (TE) module.
• Figure 4: The architecture of our Texture Diffusion (TXD) module. It utilizes iterative message propagation to diffuse texture into an enhanced intermediate representation.
• Figure 5: Qualitative comparison of our model against 9 other state-of-the-art methods on the COD datasets.