FusionVision: A comprehensive approach of 3D object reconstruction and segmentation from RGB-D cameras using YOLO and fast segment anything

TL;DR

FusionVision addresses robust 3D object segmentation from RGB-D data by unifying RGB-based detection with depth-informed segmentation and 3D reconstruction. The pipeline integrates YOLO-based RGB detection, FastSAM-based instance segmentation, and RGB-D alignment, followed by voxel-downsampling and denoising to produce accurate 3D object representations and bounding boxes. Quantitative results show high segmentation performance (mean IoU around $0.94$–$0.96$, Dice around $0.92$) and real-time capabilities (approximately $27$–$34$ fps on a RTX 2080 Ti), with significant reduction (~85%) of non-essential point-cloud data. This approach enables improved 3D scene understanding for robotics, augmented reality, and autonomous systems, and the authors provide public code for reproduction.

Abstract

In the realm of computer vision, the integration of advanced techniques into the processing of RGB-D camera inputs poses a significant challenge, given the inherent complexities arising from diverse environmental conditions and varying object appearances. Therefore, this paper introduces FusionVision, an exhaustive pipeline adapted for the robust 3D segmentation of objects in RGB-D imagery. Traditional computer vision systems face limitations in simultaneously capturing precise object boundaries and achieving high-precision object detection on depth map as they are mainly proposed for RGB cameras. To address this challenge, FusionVision adopts an integrated approach by merging state-of-the-art object detection techniques, with advanced instance segmentation methods. The integration of these components enables a holistic (unified analysis of information obtained from both color \textit{RGB} and depth \textit{D} channels) interpretation of RGB-D data, facilitating the extraction of comprehensive and accurate object information. The proposed FusionVision pipeline employs YOLO for identifying objects within the RGB image domain. Subsequently, FastSAM, an innovative semantic segmentation model, is applied to delineate object boundaries, yielding refined segmentation masks. The synergy between these components and their integration into 3D scene understanding ensures a cohesive fusion of object detection and segmentation, enhancing overall precision in 3D object segmentation. The code and pre-trained models are publicly available at https://github.com/safouaneelg/FusionVision/.

Figures (9)

• Figure 1: Example of RGB-D camera scene capturing and 3D reconstruction: (a) scene 3D reconstruction from RGB-D depth-channel. (b) RGB stream capture from RGB sensor. (c) Visual estimation of depth with the ColorMap JET (the closer object are represented in green and far ones are the dark blue regions)
• Figure 2: Complex YOLO framework for 3D object reconstruction and localization icpram20
• Figure 3: Proposed Pipeline for Real-Time 3D Object Segmentation Using Fused YOLO and FastSAM Applied on RGB-D Sensor.
• Figure 4: Visual representation of RGB camera alignment with the depth sensor
• Figure 5: Example of acquired images for YOLO training: the top two images are original, the bottom ones are augmented images