ESP-Zero: Unsupervised enhancement of zero-shot classification for Extremely Sparse Point cloud

TL;DR

This work tackles zero-shot classification of extremely sparse 3D point clouds, where pre-trained encoders struggle to align with text embeddings. It introduces an unsupervised model adaptation that freezes a pretrained point-cloud transformer and augments it with fused cross-attention (FCA) layers containing learnable tokens, along with a complementary learning-based self-distillation scheme to pull representations away from irrelevant text embeddings. The approach achieves notable gains on unseen classes in ModelNet40 and PartNet, and ablation studies attribute the improvements to both FCA refinements and the negative-label distillation, while showing that point cloud completion alone is insufficient for extreme sparsity. These results suggest a practical path to deploy zero-shot 3D recognition in highly sparse data without expensive re-training, preserving alignment with the text-embedding space and enabling robust generalization.

Abstract

In recent years, zero-shot learning has attracted the focus of many researchers, due to its flexibility and generality. Many approaches have been proposed to achieve the zero-shot classification of the point clouds for 3D object understanding, following the schema of CLIP. However, in the real world, the point clouds could be extremely sparse, dramatically limiting the effectiveness of the 3D point cloud encoders, and resulting in the misalignment of point cloud features and text embeddings. To the point cloud encoders to fit the extremely sparse point clouds without re-running the pre-training procedure which could be time-consuming and expensive, in this work, we propose an unsupervised model adaptation approach to enhance the point cloud encoder for the extremely sparse point clouds. We propose a novel fused-cross attention layer that expands the pre-trained self-attention layer with additional learnable tokens and attention blocks, which effectively modifies the point cloud features while maintaining the alignment between point cloud features and text embeddings. We also propose a complementary learning-based self-distillation schema that encourages the modified features to be pulled apart from the irrelevant text embeddings without overfitting the feature space to the observed text embeddings. Extensive experiments demonstrate that the proposed approach effectively increases the zero-shot capability on extremely sparse point clouds, and overwhelms other state-of-the-art model adaptation approaches.

Figures (5)

• Figure 1: Zero-shot classification accuracy on the ModelNet-40 dataset. Our approach dramatically increases the zero-shot capability on the extremely sparse point clouds.
• Figure 2: The proposed approach enhances the zero-shot ability on the extremely sparse point clouds.
• Figure 3: The overall framework of the proposed approach. The Dense point cloud is down-sampled to a sparse point cloud, grouped by KNN, and encoded to point cloud tokens. The initial model is modified with a trainable FCA block when processing the tokens of the sparse point cloud. We then distill the model with infoNCE loss and CL loss with the assistance of text embeddings.
• Figure 4: The structure of FCA. The learnable tokens are firstly processed by an SA block, then combined with the encoded point cloud tokens, pass through an encoder block, and only output the point cloud tokens.
• Figure 5: The difference between pseudo label and complementary label. Via pseudo labeling (a), the sparse representation is encouraged to be aligned with the text embedding of the pseudo label. On the contrary, complementary learning (b) encourages the sparse representation to be apart from the unmatched text embeddings.