MGPC: Multimodal Network for Generalizable Point Cloud Completion With Modality Dropout and Progressive Decoding

TL;DR

This work tackles the challenge of generalizing point cloud completion to novel objects and real-world scenarios by leveraging multiple modalities. It introduces MGPC, a multimodal framework that encodes point clouds, RGB images, and text into tokens, fuses them with a Transformer using modality dropout, and refines geometry through a progressive decoder. A large-scale MGPC-1M dataset with over 1,000 categories and 1 million paired samples is built using a VLM-assisted data generation pipeline to support robust learning and evaluation. Experiments show MGPC outperforms single-modal and cross-modal baselines on MGPC-1M and demonstrates strong zero-shot performance on in-the-wild data, highlighting improved generalization and practical applicability for 3D perception tasks.

Abstract

Point cloud completion aims to recover complete 3D geometry from partial observations caused by limited viewpoints and occlusions. Existing learning-based works, including 3D Convolutional Neural Network (CNN)-based, point-based, and Transformer-based methods, have achieved strong performance on synthetic benchmarks. However, due to the limitations of modality, scalability, and generative capacity, their generalization to novel objects and real-world scenarios remains challenging. In this paper, we propose MGPC, a generalizable multimodal point cloud completion framework that integrates point clouds, RGB images, and text within a unified architecture. MGPC introduces an innovative modality dropout strategy, a Transformer-based fusion module, and a novel progressive generator to improve robustness, scalability, and geometric modeling capability. We further develop an automatic data generation pipeline and construct MGPC-1M, a large-scale benchmark with over 1,000 categories and one million training pairs. Extensive experiments on MGPC-1M and in-the-wild data demonstrate that the proposed method consistently outperforms prior baselines and exhibits strong generalization under real-world conditions.

Figures (8)

• Figure 1: (a) Overview of the proposed framework and dataset. Our method encodes point clouds, RGB images, and text using modality-specific encoders, followed by a scalable Transformer with modality dropout, and predicts completed point clouds via a progressive generator with multi-scale outputs. The proposed multimodal benchmark substantially increases both category coverage and paired-sample scale compared to existing datasets. (b) Qualitative comparison on a novel real-world sample. Methods pretrained on prior datasets struggle to generalize, producing incomplete or noisy reconstructions.
• Figure 2: Overview of the proposed MGPC architecture. The framework comprises three stages: (i) Multimodal Token Extraction, which encodes point clouds, RGB images, and text into modality-specific tokens; (ii) Modality Fusion, which fuses these tokens using a scalable Transformer with a modality dropout module; and (iii) Progressive Decoding, which generates completed point clouds in a coarse-to-fine manner with multiscale supervision.
• Figure 3: Illustration of the proposed progressive generator. Deconvolution and feature replication operations are used to regress point offsets for point cloud upsampling. The process is repeated to generate completed point clouds at multiple scales.
• Figure 4: Overview of the proposed data generation pipeline. Mesh models from Objaverse, ShapeNet, and GSO are rendered in Blender to produce RGB-D images, which are processed and normalized to form data pairs with text. A VLM-based filter is applied to ensure data quality.
• Figure 5: Visual comparison with single-modal baselines using 8192 output points on MGPC-1M. (Best viewed magnified.)