PDDM: Pseudo Depth Diffusion Model for RGB-PD Semantic Segmentation Based in Complex Indoor Scenes

TL;DR

This work tackles the cost and alignment challenges of RGB-D semantic segmentation by replacing real depth with pseudo depth (PD) derived from monocular depth estimation. It introduces PDAM to fuse multiple PD maps into a single PD modality and PDDM, a diffusion-based framework that fuses PD with RGB through a simple latent-space fusion, treating PD as structured noise. Empirical results on NYUv2 and SUNRGB-D show that PDAM provides robust PD cues and that PDDM achieves state-of-the-art performance, with substantial mIoU gains over baselines. The proposed PDAM is plug-and-play and can generalize to other RGB-D methods, illustrating the practical viability of pseudo depth for indoor scene understanding.

Abstract

The integration of RGB and depth modalities significantly enhances the accuracy of segmenting complex indoor scenes, with depth data from RGB-D cameras playing a crucial role in this improvement. However, collecting an RGB-D dataset is more expensive than an RGB dataset due to the need for specialized depth sensors. Aligning depth and RGB images also poses challenges due to sensor positioning and issues like missing data and noise. In contrast, Pseudo Depth (PD) from high-precision depth estimation algorithms can eliminate the dependence on RGB-D sensors and alignment processes, as well as provide effective depth information and show significant potential in semantic segmentation. Therefore, to explore the practicality of utilizing pseudo depth instead of real depth for semantic segmentation, we design an RGB-PD segmentation pipeline to integrate RGB and pseudo depth and propose a Pseudo Depth Aggregation Module (PDAM) for fully exploiting the informative clues provided by the diverse pseudo depth maps. The PDAM aggregates multiple pseudo depth maps into a single modality, making it easily adaptable to other RGB-D segmentation methods. In addition, the pre-trained diffusion model serves as a strong feature extractor for RGB segmentation tasks, but multi-modal diffusion-based segmentation methods remain unexplored. Therefore, we present a Pseudo Depth Diffusion Model (PDDM) that adopts a large-scale text-image diffusion model as a feature extractor and a simple yet effective fusion strategy to integrate pseudo depth. To verify the applicability of pseudo depth and our PDDM, we perform extensive experiments on the NYUv2 and SUNRGB-D datasets. The experimental results demonstrate that pseudo depth can effectively enhance segmentation performance, and our PDDM achieves state-of-the-art performance, outperforming other methods by +6.98 mIoU on NYUv2 and +2.11 mIoU on SUNRGB-D.

Figures (4)

• Figure 1: (a) Previous RGB-D segmentation methods utilize RGB and depth images from RGB-D cameras. (b) Our RGB-PD segmentation adopts pseudo depth images from RGB images by meta depth estimation methods and integrates them into a single modality by pseudo depth aggregation module (PDAM). It can be observed that the real depth image exhibits some noise, resulting in rougher edges, whereas the edges in pseudo depth images are comparatively smoother.
• Figure 2: The overall architecture of the proposed PDDM. Given an RGB image, we first predict $L$ pseudo depth maps through Meta Depth Estimation methods and integrate them into a single aggregated pseudo depth with PDAM. Then, the RGB image is compressed to the latent features by the frozen VAE encoder $\mathcal{E}$. With both inputs, a pre-trained text-to-image diffusion UNet model is employed to extract internal features. With the features from diffusion UNet and VAE as inputs, the SegHead generates the final segmentation results.
• Figure 3: The structure of PDAM. It aims to aggregate effective information from multiple pseudo depth maps.
• Figure 4: Framework of mid-term fusion (PDDM-M) and late fusion (PDDM-L).