Key-Grid: Unsupervised 3D Keypoints Detection using Grid Heatmap Features

TL;DR

Key-Grid introduces an unsupervised 3D keypoint detector capable of handling both rigid and deformable objects by integrating a grid heatmap derived from keypoint skeletons into a reconstruction autoencoder. The encoder predicts keypoints and edge weights; the grid heatmap provides a dense geometric prior that guides the decoder during reconstruction. Training uses a Chamfer-based similarity loss together with a farthest-point keypoint loss to promote accurate, well-distributed keypoints, achieving state-of-the-art semantic consistency and localization on ShapeNetCoreV2 and ClothesNet, with demonstrated robustness to noise and downsampling and potential SE(3)-equivariant extensions via USEEK. Overall, Key-Grid advances unsupervised keypoint learning by leveraging a grid-based latent representation that remains effective under large shape variations and deformations, enabling more reliable downstream 3D understanding and manipulation.

Abstract

Detecting 3D keypoints with semantic consistency is widely used in many scenarios such as pose estimation, shape registration and robotics. Currently, most unsupervised 3D keypoint detection methods focus on the rigid-body objects. However, when faced with deformable objects, the keypoints they identify do not preserve semantic consistency well. In this paper, we introduce an innovative unsupervised keypoint detector Key-Grid for both the rigid-body and deformable objects, which is an autoencoder framework. The encoder predicts keypoints and the decoder utilizes the generated keypoints to reconstruct the objects. Unlike previous work, we leverage the identified keypoint in formation to form a 3D grid feature heatmap called grid heatmap, which is used in the decoder section. Grid heatmap is a novel concept that represents the latent variables for grid points sampled uniformly in the 3D cubic space, where these variables are the shortest distance between the grid points and the skeleton connected by keypoint pairs. Meanwhile, we incorporate the information from each layer of the encoder into the decoder section. We conduct an extensive evaluation of Key-Grid on a list of benchmark datasets. Key-Grid achieves the state-of-the-art performance on the semantic consistency and position accuracy of keypoints. Moreover, we demonstrate the robustness of Key-Grid to noise and downsampling. In addition, we achieve SE-(3) invariance of keypoints though generalizing Key-Grid to a SE(3)-invariant backbone.

Figures (13)

• Figure 1: Examples of the keypoints detected by Key-Grid. The detected keypoints preserve semantic consistency under: (Top) large intra-category shape variations of rigid-body objects from the ShapeNetCoreV2 chang2015shapenet dataset; (Bottom) dramatic deformations of soft-body objects from the ClothesNet zhou2023clothesnet dataset.
• Figure 2: Pipeline of Key-Grid. In the encoder section, given a point cloud, we detect the keypoints by utilizing the PointNet++. Then, we utilize the detected keypoints to form the grid heatmap. In the decoder section, we use each layer of the PointNet++ and the grid heatmap to reconstruct the input point cloud. "MLP" stands for multi-layer perceptron, which contains Batch-norm and ReLU.
• Figure 3: Example of distance definition on the grid heatmap.
• Figure 4: Different methods on the Hat and Long Pant categories during the dropping and dragging processes.(a) and (b): Keypoint detection of Hat under the dropping and dragging deformation. (c) and (d): Keypoint detection of Long Pant under the dropping and dragging deformation. We use lines to connect keypoints of the same color, representing the positional changes of the same keypoints in the deformation process of the objects.
• Figure 5: Keypoints detected on the Fold Clothes, the Deep Fash3D V2 dataset and the SUN3D dataset.(a) and (b): Eight keypoints identified by different methods during the folding process of clothes. The lines connect the keypoints with the same color, which means the positions of these keypoints change in the deformation process. (c): Grid Heatmaps and Skeleton Structures on the fold clothes. In the skeleton structures, we use purple dots to connect the keypoints identified by SM to construct the skeleton. In the grid heatmap, we use colors to represent the values of $D(\mathbf{p})$, with yellow indicating smaller values. The yellow dots capture the geometric structure of the folded clothes. (d): Keypoints detected by Key-Grid on the Deep Fash3D V2 and the SUN3D dataset.