Real-time 3D Semantic Scene Perception for Egocentric Robots with Binocular Vision

TL;DR

The paper presents a lightweight, real-time pipeline for real-world 3D semantic scene perception on egocentric robots with binocular vision. It integrates RGB-D-based egocentric segmentation, object-focused 2D keypoint detection with a 2D positional embedding, per-object masked feature matching, and KDE-informed weighted 3D-Point-Cloud registration to align views and enable grasping. Key contributions include a robust KDE-based weighting scheme for correspondences, an end-to-end segmentation-matching-registration framework, and a validated deployment on a Baxter robot equipped with an Intel RealSense D435i. The approach achieves real-time performance on conventional hardware and demonstrates accurate multiview alignment and successful manipulation in indoor scenes, with clear limitations identified for transparent/shiny objects and far-field perception. Overall, the work advances practical 3D semantic perception for indoor manipulation by tightly coupling segmentation, feature matching, and weighted registration in a multi-view, RGB-D context.

Abstract

Perceiving a three-dimensional (3D) scene with multiple objects while moving indoors is essential for vision-based mobile cobots, especially for enhancing their manipulation tasks. In this work, we present an end-to-end pipeline with instance segmentation, feature matching, and point-set registration for egocentric robots with binocular vision, and demonstrate the robot's grasping capability through the proposed pipeline. First, we design an RGB image-based segmentation approach for single-view 3D semantic scene segmentation, leveraging common object classes in 2D datasets to encapsulate 3D points into point clouds of object instances through corresponding depth maps. Next, 3D correspondences of two consecutive segmented point clouds are extracted based on matched keypoints between objects of interest in RGB images from the prior step. In addition, to be aware of spatial changes in 3D feature distribution, we also weigh each 3D point pair based on the estimated distribution using kernel density estimation (KDE), which subsequently gives robustness with less central correspondences while solving for rigid transformations between point clouds. Finally, we test our proposed pipeline on the 7-DOF dual-arm Baxter robot with a mounted Intel RealSense D435i RGB-D camera. The result shows that our robot can segment objects of interest, register multiple views while moving, and grasp the target object. The source code is available at https://github.com/mkhangg/semantic_scene_perception.

Figures (11)

• Figure 1: Overview of 3D semantic scene perception for vision-based indoor robots with binocular vision and multiple egocentric views.
• Figure 2: The 3D semantic scene perception pipeline when the robot takes two (or multiple) views of a scene includes (1) egocentric segmentation to create point clouds of objects of interest (Sec. \ref{['sec:segmentation']}), (2) extracting and matching corresponding features on masked RGB images to infer 3D correspondences via depth maps (Sec. \ref{['sec:feature_matching']}), (3) finding optimal transformations based on weighted 3D correspondences and reconstructing the 3D scene (Sec. \ref{['sec:scene_reconstruction']}), and (4) returning the aligned point cloud from multiple views with segmented objects.
• Figure 3: The egocentric object segmentation process includes (a) segmenting masks on the RGB image using YOLOv8n segmentation model, (b) obtaining and aggregating binary masks of the objects of interest, (c) aligning the corresponding depth image, (d) rectifying non-masked depth pixels on the aligned depth image with obtained masks, and (e) creating point clouds of such objects.
• Figure 4: The architecture of the feature-extraction network with 2D-positional embedding, where $\bigoplus$ denotes embedding notation.
• Figure 5: The 3D correspondences matching process includes (a) extracting and matching keypoints between masked RGB images, (b) finding corresponding depth pixels on rectified depth images, and (c) mapping 3D correspondences between point clouds of object instances.