Glissando-Net: Deep sinGLe vIew category level poSe eStimation ANd 3D recOnstruction

TL;DR

Glissando-Net addresses category-level 6D object pose estimation and 3D reconstruction from a single RGB image by jointly training an RGB image autoencoder and a point-cloud VAE, and by introducing a feature-transform module that fuses 2D and 3D representations. During training, it leverages paired RGB and canonical point clouds to learn a latent shape distribution, while inference proceeds with only RGB, discarding the 3D encoder. The approach achieves state-of-the-art or competitive results on NOCS and Pix3D, demonstrates robustness to occlusion, and shows cross-domain generalization to Objectron, all while avoiding per-category latent spaces and depth input. The key contributions are the dual autoencoder design, the 2D-3D feature transform for enhanced interaction, and the joint decoding of shape and pose to exploit 3D priors learned during training.

Abstract

We present a deep learning model, dubbed Glissando-Net, to simultaneously estimate the pose and reconstruct the 3D shape of objects at the category level from a single RGB image. Previous works predominantly focused on either estimating poses(often at the instance level), or reconstructing shapes, but not both. Glissando-Net is composed of two auto-encoders that are jointly trained, one for RGB images and the other for point clouds. We embrace two key design choices in Glissando-Net to achieve a more accurate prediction of the 3D shape and pose of the object given a single RGB image as input. First, we augment the feature maps of the point cloud encoder and decoder with transformed feature maps from the image decoder, enabling effective 2D-3D interaction in both training and prediction. Second, we predict both the 3D shape and pose of the object in the decoder stage. This way, we better utilize the information in the 3D point clouds presented only in the training stage to train the network for more accurate prediction. We jointly train the two encoder-decoders for RGB and point cloud data to learn how to pass latent features to the point cloud decoder during inference. In testing, the encoder of the 3D point cloud is discarded. The design of Glissando-Net is inspired by codeSLAM. Unlike codeSLAM, which targets 3D reconstruction of scenes, we focus on pose estimation and shape reconstruction of objects, and directly predict the object pose and a pose invariant 3D reconstruction without the need of the code optimization step. Extensive experiments, involving both ablation studies and comparison with competing methods, demonstrate the efficacy of our proposed method, and compare favorably with the state-of-the-art.

Figures (13)

• Figure 1: Category-level 3D reconstruction: on the top are input images from NOCS dataset wang2019normalized and on the bottom are predicted 3D shapes; on the left are examples of training instances and on the right are testing instances. Our model predicts complete 3D shapes of instances unknown during training based on their RGB appearance and category shape prior learned from data.
• Figure 2: Overview of Glissando-Net: the RGB image is processed by an encoder-decoder; the feature maps from the RGB decoder are then transformed and concatenated with features in the point cloud decoder, which takes in a latent code $\mathbf{z}$; the outputs from the point-cloud decoder are processed by a fully-connect layer to regress the 3D shape in canonical pose and two independent multilayer perceptrons (MLP) to estimate its pose; in training, the ground-truth canonical shape is an additional input to a point cloud encoder to learn the distribution of $\mathbf{z}$ and feature maps from the RGB decoder are transformed to augment point cloud features.
• Figure 3: Details of RGB and point cloud encoder-decoder from Glissando-Net. Red arrows indicate skip connections between RGB encoder and decoder layers. Features 1–4 from the RGB decoder are fed into the point cloud encoder-decoder. The point cloud encoder uses set abstraction (SA) layers with sampled points from PointNet++, which are projected back to the RGB feature map using RoI align. Features 1–4 are concatenated with the point cloud decoder outputs via 2D convolution and fully connected layers.
• Figure 4: Qualitative results on NOCS dataset. Top: input image. Middle: ground-truth 3D shape with ground-truth rotation. Bottom: predicted 3D shape with predicted rotation.
• Figure 5: Reconstructed 3D shape with occluded RGB input. Column 1 and 3, top to bottom: original RGB image and the ones occluded from bottom, center, left, right, and top. Column 2 and 4: Corresponding reconstructed shapes, shown in the same pose for visualization.