Efficient RGB-D Scene Understanding via Multi-task Adaptive Learning and Cross-dimensional Feature Guidance

TL;DR

This paper presents an efficient RGB-D scene understanding model that performs a range of tasks, including semantic segmentation, instance segmentation, orientation estimation, panoptic segmentation, and scene classification, and incorporates an enhanced fusion encoder.

Abstract

Scene understanding plays a critical role in enabling intelligence and autonomy in robotic systems. Traditional approaches often face challenges, including occlusions, ambiguous boundaries, and the inability to adapt attention based on task-specific requirements and sample variations. To address these limitations, this paper presents an efficient RGB-D scene understanding model that performs a range of tasks, including semantic segmentation, instance segmentation, orientation estimation, panoptic segmentation, and scene classification. The proposed model incorporates an enhanced fusion encoder, which effectively leverages redundant information from both RGB and depth inputs. For semantic segmentation, we introduce normalized focus channel layers and a context feature interaction layer, designed to mitigate issues such as shallow feature misguidance and insufficient local-global feature representation. The instance segmentation task benefits from a non-bottleneck 1D structure, which achieves superior contour representation with fewer parameters. Additionally, we propose a multi-task adaptive loss function that dynamically adjusts the learning strategy for different tasks based on scene variations. Extensive experiments on the NYUv2, SUN RGB-D, and Cityscapes datasets demonstrate that our approach outperforms existing methods in both segmentation accuracy and processing speed.

Figures (13)

• Figure 1: Multi-task outputs of the network. Semantic segmentation provides foreground masks for instance segmentation, and pixel-level instances are generated by combining instance centers with instance offsets. Panoptic segmentation is achieved by integrating semantic and instance segmentation.
• Figure 2: Multi-task scene understanding network structure. This network features an improved feature fusion encoder that handles redundant information from RGB and depth to enhance feature extraction. The semantic decoder includes NFCL and CFIL to enrich scene representations from various dimensions. The instance decoder employs the non-bottleneck 1D architecture to generate instance centers, instance offsets, and raw orientations. The integration of instance segmentation with semantic segmentation enables panoptic segmentation. Scene classification is performed by a task head with a fully connected layer. In addition, the multi-task adaptive loss function optimizes the training strategy based on data variations.
• Figure 3: Fusion encoder. This encoder adopts merging layers for channel expansion and downsampling. Given the similarity between channel features, the fusion blocks extract redundant channel features to optimize computational efficiency.
• Figure 4: Normalized focus channel layer. This layer learns the variance adjustment parameters via batch normalization. It then computes channel weights that quantify the importance of each channel. The input features are rearranged, weighted according to these channel weights, and subsequently processed through an activation function.
• Figure 5: Context feature interaction layer. This layer addresses the limitation of the semantic decoder in integrating local and global information by employing a multi-scale pooling operation and efficient channel compression.