P2M2-Net: Part-Aware Prompt-Guided Multimodal Point Cloud Completion

TL;DR

This work tackles the challenge of completing severely occluded 3D point clouds with controllable, part-aware guidance. It introduces P2M2-Net, a Transformer-based framework that fuses multimodal features from 3D point data and text prompts, trained with a novel PartNet-Prompt dataset and a cross-modal contrastive pre-training stage. The method enables both deterministic and prompt-guided diverse completions, demonstrated on PartNet-based benchmarks with quantitative and qualitative superiority over several baselines. The approach advances controllable 3D shape editing and generation, with potential applications in fine-grained shape understanding and retrieval, and points to future work in expanding diversity through generative models.

Abstract

Inferring missing regions from severely occluded point clouds is highly challenging. Especially for 3D shapes with rich geometry and structure details, inherent ambiguities of the unknown parts are existing. Existing approaches either learn a one-to-one mapping in a supervised manner or train a generative model to synthesize the missing points for the completion of 3D point cloud shapes. These methods, however, lack the controllability for the completion process and the results are either deterministic or exhibiting uncontrolled diversity. Inspired by the prompt-driven data generation and editing, we propose a novel prompt-guided point cloud completion framework, coined P2M2-Net, to enable more controllable and more diverse shape completion. Given an input partial point cloud and a text prompt describing the part-aware information such as semantics and structure of the missing region, our Transformer-based completion network can efficiently fuse the multimodal features and generate diverse results following the prompt guidance. We train the P2M2-Net on a new large-scale PartNet-Prompt dataset and conduct extensive experiments on two challenging shape completion benchmarks. Quantitative and qualitative results show the efficacy of incorporating prompts for more controllable part-aware point cloud completion and generation. Code and data are available at https://github.com/JLU-ICL/P2M2-Net.

Figures (7)

• Figure 1: Given an input point cloud with a large missing region (left column), our P2M2-Net can use different text prompts to guide the shape completion and generate diverse outputs in a controllable manner.
• Figure 2: Overview of P2M2-Net. (a) Illustration of the cross-modal contrastive pre-training. Embeddings of the same part sare pulled closer; (b) Pipeline of the multimodal Transformer for prompt-guided completion.
• Figure 3: Qualitative comparisons on PartNet and PartNet-Scan benchmarks.
• Figure 4: Qualitative comparison with MPC wu2020multimodal for shapes in the PartNet benchmark.
• Figure 5: Qualitative results on PartNet and PartNet-Scan. We visualize the generated results from different prompts. P2M2-Net not only preserves the originally observed structure but also achieves diverse generated results that comply with the prompt.