DepthMatch: Semi-Supervised RGB-D Scene Parsing through Depth-Guided Regularization
Jianxin Huang, Jiahang Li, Sergey Vityazev, Alexander Dvorkovich, Rui Fan
TL;DR
DepthMatch tackles the scarcity of pixel-level labels in RGB-D scene parsing by introducing a semi-supervised framework that leverages unlabeled RGB-D data. It combines Complementary Patch Mix-up Augmentation (CPMA), Lightweight Spatial Priors Injectors (LSPI), and a Depth-Guided Boundary Loss (DGBL) within a consistency-regularized training pipeline that uses a DINOv2 encoder and a DPT decoder. The approach enables efficient fusion of heterogeneous features and boundary-aware learning on unlabeled data, achieving state-of-the-art results on NYUv2 and KITTI Semantics while maintaining high throughput (74 FPS) and reducing annotation requirements. Overall, the method demonstrates strong applicability to both indoor and outdoor scenes and points to extending semi-supervised RGB-D parsing to other tasks.
Abstract
RGB-D scene parsing methods effectively capture both semantic and geometric features of the environment, demonstrating great potential under challenging conditions such as extreme weather and low lighting. However, existing RGB-D scene parsing methods predominantly rely on supervised training strategies, which require a large amount of manually annotated pixel-level labels that are both time-consuming and costly. To overcome these limitations, we introduce DepthMatch, a semi-supervised learning framework that is specifically designed for RGB-D scene parsing. To make full use of unlabeled data, we propose complementary patch mix-up augmentation to explore the latent relationships between texture and spatial features in RGB-D image pairs. We also design a lightweight spatial prior injector to replace traditional complex fusion modules, improving the efficiency of heterogeneous feature fusion. Furthermore, we introduce depth-guided boundary loss to enhance the model's boundary prediction capabilities. Experimental results demonstrate that DepthMatch exhibits high applicability in both indoor and outdoor scenes, achieving state-of-the-art results on the NYUv2 dataset and ranking first on the KITTI Semantics benchmark.