PiSA: A Self-Augmented Data Engine and Training Strategy for 3D Understanding with Large Models

TL;DR

PiSA addresses the data bottleneck in 3D multimodal learning by introducing PiSA-Engine, a self-augmented data generation framework that combines 3D-space annotation with 2D cross-validation and iterative 3D data bootstrap. The authors train PointLLM-PiSA in a co-evolution loop, improving data quality and model performance through successive LOOP phases and leveraging PiSA-Bench for comprehensive evaluation across description, color, shape, count, spatial relations, and usage. The framework yields state-of-the-art results in zero-shot 3D object captioning and generative classification, and demonstrates strong transfer to downstream 3D understanding tasks, including 3D shape classification, with a plug-and-play capability. These contributions offer a scalable path to high-quality 3D instruction data and robust 3D MLLMs, with practical impact for research and applications in 3D scene understanding, robotics, and digital twins.

Abstract

3D Multimodal Large Language Models (MLLMs) have recently made substantial advancements. However, their potential remains untapped, primarily due to the limited quantity and suboptimal quality of 3D datasets. Current approaches attempt to transfer knowledge from 2D MLLMs to expand 3D instruction data, but still face modality and domain gaps. To this end, we introduce PiSA-Engine (Point-Self-Augmented-Engine), a new framework for generating instruction point-language datasets enriched with 3D spatial semantics. We observe that existing 3D MLLMs offer a comprehensive understanding of point clouds for annotation, while 2D MLLMs excel at cross-validation by providing complementary information. By integrating holistic 2D and 3D insights from off-the-shelf MLLMs, PiSA-Engine enables a continuous cycle of high-quality data generation. We select PointLLM as the baseline and adopt this co-evolution training framework to develop an enhanced 3D MLLM, termed PointLLM-PiSA. Additionally, we identify limitations in previous 3D benchmarks, which often feature coarse language captions and insufficient category diversity, resulting in inaccurate evaluations. To address this gap, we further introduce PiSA-Bench, a comprehensive 3D benchmark covering six key aspects with detailed and diverse labels. Experimental results demonstrate PointLLM-PiSA's state-of-the-art performance in zero-shot 3D object captioning and generative classification on our PiSA-Bench, achieving significant improvements of 46.45% (+8.33%) and 63.75% (+16.25%), respectively. We will release the code, datasets, and benchmark.

Figures (7)

• Figure 1: The Illustration of PiSA. PiSA-Engine (left) leverages the complementary strengths of both 3D and 2D MLLMs to achieve high-quality 3D dataset generation cause 3D contents cover holistic information and 2D MLLMs excel at cross-validation. We enhance PointLLM with data generated by PiSA-Engine, namely PointLLM-PiSA (middle). Previous benchmarks are coarse and lack details, hence we propose a comprehensive benchmark PiSA-Bench (right), which supports evaluating downstream tasks more completely and accurately.
• Figure 2: Performance Comparison of Different Models on PiSA-Bench. Considering the randomness of the generative task, we conduct five tests and take the average as the final result.
• Figure 3: Comparison between Previous and Proposed Benchmarks, PiSA-Bench. In the testing set of PointLLM, each point cloud is paired with limited content, which is not comprehensive. However, the annotations in PiSA-Bench cover six comprehensive aspects. For the classification task, we provide class, subclass, and synonyms to avoid misjudgment.
• Figure 4: Sample Comparisons. We showcase one example of Inaccuracy and Hallucination (highlighted in red) and 3D information including depth, spatial, relative position and geometric Information (highlighted in green) within existing 3D MLLMs guo2023pointxu2023pointllm and ours. These samples show our method produces more accurate and detailed results than the baseline and human-annotated ground truths.
• Figure 5: The Overview of the PiSA-Engine. After enhanced by data generated by the PiSA-Engine during the LOOP$_{t}$ phase, the PointLLM-PiSA can produce higher-quality outputs, which can inject more precise 3D knowledge into the training data for LOOP$_{t+1}$ phase. The above shows this recursive training strategy.