Point-PRC: A Prompt Learning Based Regulation Framework for Generalizable Point Cloud Analysis

TL;DR

The paper tackles 3D domain generalization for large multi-modal point cloud models by introducing Point-PRC, a regulation framework that couples lightweight prompt learning with pre-trained 3D knowledge. It comprises three constraints—Mutual Agreement Constraint (MAC), Text Diversity Constraint (TDC), and Model Ensemble Constraint (MEC)—and optimizes a joint objective where $\mathcal{L}_{RC} = \alpha L_p + \beta L_t + \gamma L_D$. The authors also curate three new 3DDG benchmarks (base-to-new, cross-dataset, few-shot) and demonstrate consistent improvements in both generalization and task performance across ULIP/ULIP-2 and PointCLIP-based models, validating the approach as model-agnostic and scalable. Overall, Point-PRC advances open-vocabulary 3D recognition by enabling prompts to interact with large 3D knowledge without overfitting, with practical impact on robust deployment of 3D vision systems.

Abstract

This paper investigates the 3D domain generalization (3DDG) ability of large 3D models based on prevalent prompt learning. Recent works demonstrate the performances of 3D point cloud recognition can be boosted remarkably by parameter-efficient prompt tuning. However, we observe that the improvement on downstream tasks comes at the expense of a severe drop in 3D domain generalization. To resolve this challenge, we present a comprehensive regulation framework that allows the learnable prompts to actively interact with the well-learned general knowledge in large 3D models to maintain good generalization. Specifically, the proposed framework imposes multiple explicit constraints on the prompt learning trajectory by maximizing the mutual agreement between task-specific predictions and task-agnostic knowledge. We design the regulation framework as a plug-and-play module to embed into existing representative large 3D models. Surprisingly, our method not only realizes consistently increasing generalization ability but also enhances task-specific 3D recognition performances across various 3DDG benchmarks by a clear margin. Considering the lack of study and evaluation on 3DDG, we also create three new benchmarks, namely base-to-new, cross-dataset and few-shot generalization benchmarks, to enrich the field and inspire future research. Code and benchmarks are available at \url{https://github.com/auniquesun/Point-PRC}.

Figures (5)

• Figure 1: Motivation of our research: to promote the performances on downstream 3D tasks while maintaining good generalization of large 3D models. The experiments are conducted on ShapeNetCoreV2. ULIP-2 can reach 71.22% zero-shot recognition accuracy on this dataset. Recent works built on ULIP-2 introduce lightweight prompt tuning (PT) to further boost target tasks (75.80% accuracy). However, we observe the improvements come at the expenses of a severe drop in 3D domain generalization (e.g., 57.07% accuracy on new classes, much behind 71.22%), and develop a systematic regulation constraint (RC) framework to address this challenge.
• Figure 2: The overall architecture of our point cloud analysis prompt regulation constraint framework, namely Point-PRC, consisting of three core components as in the figure.
• Figure 3: Illustration of diverse questions to LLMs, including GPT-3.5, GPT-4 and PointLLM. The responses given by LLMs are regarded as the text descriptions to the point cloud and fed into the text encoder.
• Figure 4: Comparison of few-shot generalization. The solid and dashed lines represent the models with and without our framework. Zero-shot performances of ULIP and ULIP-2 are marked with star symbols. The figure in the upper left presents the average results over 5 datasets.
• Figure 5: Ablation study for the prompt depth and length. We compare the harmonic mean on five datasets of the base-to-new benchmark and the average results are displayed in dashed lines.