Simultaneous 3D Object Segmentation and 6-DOF Pose Estimation

TL;DR

The results show that the proposed single-shot method for simultaneous 3D object segmentation and 6-DOF pose estimation in pure 3D point clouds scenes can be well generalized into multiple scenarios and provide performance comparable to or better than the state-of-the-arts.

Abstract

We propose a single-shot method for simultaneous 3D object segmentation and 6-DOF pose estimation in pure 3D point clouds scenes based on a consensus that \emph{one point only belongs to one object}, i.e., each point has the potential power to predict the 6-DOF pose of its corresponding object. Unlike the recently proposed methods of the similar task, which rely on 2D detectors to predict the projection of 3D corners of the 3D bounding boxes and the 6-DOF pose must be estimated by a PnP like spatial transformation method, ours is concise enough not to require additional spatial transformation between different dimensions. Due to the lack of training data for many objects, the recently proposed 2D detection methods try to generate training data by using rendering engine and achieve good results. However, rendering in 3D space along with 6-DOF is relatively difficult. Therefore, we propose an augmented reality technology to generate the training data in semi-virtual reality 3D space. The key component of our method is a multi-task CNN architecture that can simultaneously predicts the 3D object segmentation and 6-DOF pose estimation in pure 3D point clouds. For experimental evaluation, we generate expanded training data for two state-of-the-arts 3D object datasets \cite{PLCHF}\cite{TLINEMOD} by using Augmented Reality technology (AR). We evaluate our proposed method on the two datasets. The results show that our method can be well generalized into multiple scenarios and provide performance comparable to or better than the state-of-the-arts.

Figures (6)

• Figure 2: Our method directly operates on the raw point clouds scene and can predict point-wise semantic segmentation and 6-DOF pose estimation. (A) The input point clouds scene, (B) the point-wise semantic segmentaion, (C) the result of the 6-DOF pose estimation with 3D bounding boxes.
• Figure 3: The Augmented Reality technology can be used to quickly create 3D object recognition datasets in fixed working scenarios. (A) the generated AR scenario for LC-HF dataset PLCHF, (B) the generated AR scenario for LineMod dataset TLINEMOD.
• Figure 4: The framework of our simultaneous 3D object detection and 6-DOF pose estimation method. For a given 3D point clouds scenes with multiple attributes, we A) sampling/grouping the input scene and generate low density and high dimensional feature data, B&C) enable to learn point-wise local features via Pointnet PointNet2 architecture, D) predict the point-wise class probability, 3D control points of object 3D bounding boxes and the confidence score, E) refine the prediction via non-maxima suppression and Iterative Closest Point (ICP) ICP.
• Figure 5: The architecture of the Set Abstraction (SA) layer, where contains a sampling layer for sampling keypoints, a grouping layer for finding neighboring points and a Pointnet PointNet layer for extracting new features.
• Figure 6: The generation of AR datasets (A) the AR scene, (B) the rendered 3D point clouds, (C) the generated semantic and instance labels.