IAM: Enhancing RGB-D Instance Segmentation with New Benchmarks

TL;DR

This work tackles the shortage of RGB-D instance-segmentation benchmarks by introducing three real-world indoor datasets (NYUDv2-IS, SUN-RGBD-IS, Box-IS) and proposes Intra-modal Attention Mix (IAM) for efficient RGB-D fusion. IAM emphasizes intra-modal relationships while maintaining cross-modal integration, augmented by Channel-wise Dynamic Fusion to adaptively calibrate channel-level contributions. Across DETR and SOLQ baselines, IAM substantially improves $\mathrm{AP}^{seg}$ over early fusion and other baselines (e.g., gains of $\approx$6.7–7.2 percentage points on NYUDv2-IS) and demonstrates robustness to depth quality variations. The combination of enriched intra-modal attention and dynamic fusion yields more precise boundaries and better instance separation, facilitating practical applications in indoor navigation, robotics, and assistive systems.

Abstract

Image segmentation is a vital task for providing human assistance and enhancing autonomy in our daily lives. In particular, RGB-D segmentation-leveraging both visual and depth cues-has attracted increasing attention as it promises richer scene understanding than RGB-only methods. However, most existing efforts have primarily focused on semantic segmentation and thus leave a critical gap. There is a relative scarcity of instance-level RGB-D segmentation datasets, which restricts current methods to broad category distinctions rather than fully capturing the fine-grained details required for recognizing individual objects. To bridge this gap, we introduce three RGB-D instance segmentation benchmarks, distinguished at the instance level. These datasets are versatile, supporting a wide range of applications from indoor navigation to robotic manipulation. In addition, we present an extensive evaluation of various baseline models on these benchmarks. This comprehensive analysis identifies both their strengths and shortcomings, guiding future work toward more robust, generalizable solutions. Finally, we propose a simple yet effective method for RGB-D data integration. Extensive evaluations affirm the effectiveness of our approach, offering a robust framework for advancing toward more nuanced scene understanding.

Figures (12)

• Figure 1: Feature visualizations from the input image and depth map (a), as well as RGB and RGB-D models. (b) RGB features, (c) depth features, and (d) RGB-D features are shown. The first row presents fine-grained features, while the second row highlights progressively coarser features.
• Figure 2: Examples of datasets: RGB image, depth map, and ground truth labels.
• Figure 3: Examples from the constructed datasets. (a) Diversity of categories per image, comparing the number of classes per image. (b) Distribution of instances per category, illustrating the proportions of object categories.
• Figure 4: Cumulative distribution function showing the relative scale of objects.
• Figure 5: Scatter plots illustrating the relative proportions of bounding boxes for each instance within an image. The x-axis represents the relative width of the bounding boxes, while the y-axis represents the relative height. Each point on the plot corresponds to an instance, showcasing its unique bounding box proportions.