PaRot: Patch-Wise Rotation-Invariant Network via Feature Disentanglement and Pose Restoration

TL;DR

A novel Patch-wise Rotation-invariant network (PaRot), which achieves rotation invariance via feature disentanglement and produces consistent predictions for samples with arbitrary rotations and achieves competitive results in rotated 3D object classification and part segmentation tasks.

Abstract

Recent interest in point cloud analysis has led rapid progress in designing deep learning methods for 3D models. However, state-of-the-art models are not robust to rotations, which remains an unknown prior to real applications and harms the model performance. In this work, we introduce a novel Patch-wise Rotation-invariant network (PaRot), which achieves rotation invariance via feature disentanglement and produces consistent predictions for samples with arbitrary rotations. Specifically, we design a siamese training module which disentangles rotation invariance and equivariance from patches defined over different scales, e.g., the local geometry and global shape, via a pair of rotations. However, our disentangled invariant feature loses the intrinsic pose information of each patch. To solve this problem, we propose a rotation-invariant geometric relation to restore the relative pose with equivariant information for patches defined over different scales. Utilising the pose information, we propose a hierarchical module which implements intra-scale and inter-scale feature aggregation for 3D shape learning. Moreover, we introduce a pose-aware feature propagation process with the rotation-invariant relative pose information embedded. Experiments show that our disentanglement module extracts high-quality rotation-robust features and the proposed lightweight model achieves competitive results in rotated 3D object classification and part segmentation tasks. Our project page is released at: https://patchrot.github.io/.

Figures (9)

• Figure 1: Illustration of pose information loss when generating patch-wise rotation-invariant features. We visualise our learned patch-wise rotation-invariant features of an air plane instance under two different orientations. As it can be seen, for B and C that have exactly same geometric shapes and different poses (i.e. orientation and global position), the same features are generated. However, the features for patches on tail (A), left wing (B, C) and right wing (D) will be very similar. The information for distinguishing their difference between are lost.
• Figure 2: The overall architecture of PaRot for 3D classification and segmentation, where KNN and T denote $k$ nearest neighboring and translation, respectively. We generate local scale patches and global scale patches with FPS, KNN, and translation operations. Then we disentangle patches of different scales into shape content information and orientation information separately. The learned orientations are utilised to determine geometric relations for restoring patch-wise relative pose and guiding intra- and inter-scale invariant features learning and pose-aware feature propagation.
• Figure 3: Frameworks of disentanglement module. The input patch is randomly rotated twice and then are independently fed through into the module as branch $a$ and $b$. The output of branch $a$ is hierarchically encoded while that of branch $b$ is used to assist learning.
• Figure 4: Illustrations of rotation-equivariant orientation learning. The output orientation of branch $a$ is rotated back by $\mathbf{R}_{a}^{-1}$ which is used as the predicted orientation of the original patch $\mathbf{M}$.
• Figure 5: Angles used in geometric relation representation. For two patches $\mathbf{M}_m$ and $\mathbf{M}_n$, we use different colors to illustrate different vectors in their orientation matrices $\mathbf{O}_{m}$ and $\mathbf{O}_{n}$.