Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Meta Episodic learning with Dynamic Task Sampling for CLIP-based Point Cloud Classification

Shuvozit Ghose, Yang Wang

TL;DR

This work tackles the challenge of few-shot CLIP-based point cloud classification, where random N-way K-shot sampling can underfit for diverse 3D shapes. It introduces a meta-episodic learning framework that learns a robust adapter initialization through bi-level optimization, enabling rapid adaptation to new tasks from limited examples. A key contribution is dynamic task sampling guided by a per-class performance memory, which prioritizes underrepresented or harder classes within episodes. Empirically, the approach yields 3–6% average gains on ModelNet40 and ScanObjectNN across two CLIP-based baselines, demonstrating improved generalization to real-world, unseen class distributions while maintaining efficiency with single-step adaptation.

Abstract

Point cloud classification refers to the process of assigning semantic labels or categories to individual points within a point cloud data structure. Recent works have explored the extension of pre-trained CLIP to 3D recognition. In this direction, CLIP-based point cloud models like PointCLIP, CLIP2Point have become state-of-the-art methods in the few-shot setup. Although these methods show promising performance for some classes like airplanes, desks, guitars, etc, the performance for some classes like the cup, flower pot, sink, nightstand, etc is still far from satisfactory. This is due to the fact that the adapter of CLIP-based models is trained using randomly sampled N-way K-shot data in the standard supervised learning setup. In this paper, we propose a novel meta-episodic learning framework for CLIP-based point cloud classification, addressing the challenges of limited training examples and sampling unknown classes. Additionally, we introduce dynamic task sampling within the episode based on performance memory. This sampling strategy effectively addresses the challenge of sampling unknown classes, ensuring that the model learns from a diverse range of classes and promotes the exploration of underrepresented categories. By dynamically updating the performance memory, we adaptively prioritize the sampling of classes based on their performance, enhancing the model's ability to handle challenging and real-world scenarios. Experiments show an average performance gain of 3-6\% on ModelNet40 and ScanobjectNN datasets in a few-shot setup.

Meta Episodic learning with Dynamic Task Sampling for CLIP-based Point Cloud Classification

TL;DR

This work tackles the challenge of few-shot CLIP-based point cloud classification, where random N-way K-shot sampling can underfit for diverse 3D shapes. It introduces a meta-episodic learning framework that learns a robust adapter initialization through bi-level optimization, enabling rapid adaptation to new tasks from limited examples. A key contribution is dynamic task sampling guided by a per-class performance memory, which prioritizes underrepresented or harder classes within episodes. Empirically, the approach yields 3–6% average gains on ModelNet40 and ScanObjectNN across two CLIP-based baselines, demonstrating improved generalization to real-world, unseen class distributions while maintaining efficiency with single-step adaptation.

Abstract

Point cloud classification refers to the process of assigning semantic labels or categories to individual points within a point cloud data structure. Recent works have explored the extension of pre-trained CLIP to 3D recognition. In this direction, CLIP-based point cloud models like PointCLIP, CLIP2Point have become state-of-the-art methods in the few-shot setup. Although these methods show promising performance for some classes like airplanes, desks, guitars, etc, the performance for some classes like the cup, flower pot, sink, nightstand, etc is still far from satisfactory. This is due to the fact that the adapter of CLIP-based models is trained using randomly sampled N-way K-shot data in the standard supervised learning setup. In this paper, we propose a novel meta-episodic learning framework for CLIP-based point cloud classification, addressing the challenges of limited training examples and sampling unknown classes. Additionally, we introduce dynamic task sampling within the episode based on performance memory. This sampling strategy effectively addresses the challenge of sampling unknown classes, ensuring that the model learns from a diverse range of classes and promotes the exploration of underrepresented categories. By dynamically updating the performance memory, we adaptively prioritize the sampling of classes based on their performance, enhancing the model's ability to handle challenging and real-world scenarios. Experiments show an average performance gain of 3-6\% on ModelNet40 and ScanobjectNN datasets in a few-shot setup.
Paper Structure (11 sections, 3 equations, 4 figures, 4 tables, 2 algorithms)

This paper contains 11 sections, 3 equations, 4 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methodology
  4. Baseline CLIP-based point cloud models
  5. Meta Episodic Learning for Point Cloud Classification
  6. Dynamic Task Sampling
  7. Experiments
  8. Competitors
  9. Result Analysis and Discussion
  10. Ablation Study
  11. Conclusion

Figures (4)

  • Figure 1: (a) cup (side view) (b) flower pot (side view) (c) sink (top view). We visualize the 3d object of the corresponding point cloud for better visual understanding.
  • Figure 2: The adapter learns following a bi-level optimization process. While the adapter learns to recognize and discriminate the features of the inner loop update, the outer loop extracts meta-features of the point clouds that generalize across tasks. Additionally, we introduce dynamic task sampling within the episode based on performance memory to ensure underrepresented class sampling.
  • Figure 3: (a) Zero-shot result of PointCLIP zhang2022pointclip. We can observe that CLIP's Visual encoder has already captured certain classes such as airplanes, desks, guitars, etc. (b) Few-shot result of PointCLIP zhang2022pointclip. Some classes like the cup, flower pot, sink, nightstand, etc., exhibit lower performance in the few-shot setup.
  • Figure 4: Accuracy of competitors with varying adaptation steps on ModelNet40 using prompt “point cloud of a big [CLASS]”.