Point-In-Context: Understanding Point Cloud via In-Context Learning

TL;DR

This work introduces Point-In-Context (PIC), the first in-context learning framework for 3D point clouds, featuring a vanilla generalist (PIC-G) for multitask inference and an extended segmenter (PIC-S) for cross-dataset part segmentation. A Joint Sampling module addresses information leakage and unordered 3D structure, enabling effective Masked Point Modeling in 3D. The authors construct ShapeNet-based in-context datasets and a Human & Object Segmentation benchmark, including a one-shot AKB-48 test to assess cross-domain generalization. PIC-S, with In-Context Labeling and In-Context Enhancing, achieves state-of-the-art results on segmentation tasks and shows strong generalization to unseen datasets. Overall, PIC establishes a scalable, prompt-driven approach to 3D understanding with potential for broad applicability across tasks and datasets.

Abstract

With the emergence of large-scale models trained on diverse datasets, in-context learning has emerged as a promising paradigm for multitasking, notably in natural language processing and image processing. However, its application in 3D point cloud tasks remains largely unexplored. In this work, we introduce Point-In-Context (PIC), a novel framework for 3D point cloud understanding via in-context learning. We address the technical challenge of effectively extending masked point modeling to 3D point clouds by introducing a Joint Sampling module and proposing a vanilla version of PIC called Point-In-Context-Generalist (PIC-G). PIC-G is designed as a generalist model for various 3D point cloud tasks, with inputs and outputs modeled as coordinates. In this paradigm, the challenging segmentation task is achieved by assigning label points with XYZ coordinates for each category; the final prediction is then chosen based on the label point closest to the predictions. To break the limitation by the fixed label-coordinate assignment, which has poor generalization upon novel classes, we propose two novel training strategies, In-Context Labeling and In-Context Enhancing, forming an extended version of PIC named Point-In-Context-Segmenter (PIC-S), targeting improving dynamic context labeling and model training. By utilizing dynamic in-context labels and extra in-context pairs, PIC-S achieves enhanced performance and generalization capability in and across part segmentation datasets. PIC is a general framework so that other tasks or datasets can be seamlessly introduced into our PIC through a unified data format. We conduct extensive experiments to validate the versatility and adaptability of our proposed methods in handling a wide range of tasks and segmenting multi-datasets. Our PIC-S is capable of generalizing unseen datasets and performing novel part segmentation by customizing prompts.

Figures (9)

• Figure 1: Illustration of in-context learning for different tasks. (a) In-context learning in NLPgpt3, with different text prompts for corresponding tasks: translation and sentiment analysis. (b) In-context learning in 2D visionvisualprompt, with 2D visual prompts for different tasks: segmentation and inpainting. (c) Our proposed Point-In-Context-Generalist (PIC-G) for 3D point cloud multitasking, with 3D visual prompts for different tasks: reconstruction, denoising, registration, etc. (d) Our proposed Point-In-Context-Segmenter (PIC-S) for multiple part segmentation datasets in 3D point clouds, including ShapeNetPart shapenetpart, Human3D human3d, BEHAVE behave, and AKB-48 akb48. Note that our PIC-S can generalize to unseen segmentation dataset AKB-48, which is not included in the training set.
• Figure 2: (a) The pre-training pipeline used in previous works. When performing Masked Point Modeling (MPM), these works pointbertpointmaepointm2ae use the center position of the target patches for position embedding, which results in information leakage. (b) Difference between 2D images and 3D point clouds. For 2D images, the semantic information at corresponding pixel positions in the input and target images is consistent. However, after performing grouping operations on 3D point clouds, the order of points will inevitably be disrupted, resulting in inconsistency between the point sequences of the input and output point clouds. (c) Joint Sampling module involves recording the indices of sampled center points and employing the K-nearest neighbor strategy to sample both the input and target point clouds concurrently.
• Figure 3: Overall scheme of our Point-In-Context-Generalist. Top: Training pipeline of the Masked Point Modeling (MPM) framework. During training, each sample comprises two pairs of input and target point clouds that tackle the same task. These pairs are fed into the transformer model to perform the masked point reconstruction task, which follows a random masking process. Bottom: In-context inference on multitask. Our Point-In-Context could infer results on various downstream point cloud tasks, including reconstruction, denoising, registration, and part segmentation.
• Figure 4: Comparison of generating targets between In-Context Labeling (PIC-S) and pre-defined label map (PIC-G).$P$ represents the number of all parts ($P\gg N_{B} > |C_i|$).
• Figure 5: Visualization of predictions from PIC-G-Sep on ShapeNet In-Context Datasets. For part segmentation, we visualize the generated target together with the mapping back, both adding category-specific colors for a better look.