GenPC: Zero-shot Point Cloud Completion via 3D Generative Priors

TL;DR

GenPC addresses real-world partial point cloud completion by leveraging explicit priors from pre-trained image-to-3D generators. It introduces Depth Prompting to convert partial scans into depth-conditioned inputs and Geometric Preserving Fusion to align generated shapes with the input geometry, enabling fast, zero-shot completion. The approach achieves state-of-the-art zero-shot performance on Redwood, ScanNet, and KITTI while reducing inference time compared with diffusion-based methods. This work demonstrates that explicit 3D priors can robustly complete real-world scans and generalize to unseen categories, advancing practical 3D completion.

Abstract

Existing point cloud completion methods, which typically depend on predefined synthetic training datasets, encounter significant challenges when applied to out-of-distribution, real-world scans. To overcome this limitation, we introduce a zero-shot completion framework, termed GenPC, designed to reconstruct high-quality real-world scans by leveraging explicit 3D generative priors. Our key insight is that recent feed-forward 3D generative models, trained on extensive internet-scale data, have demonstrated the ability to perform 3D generation from single-view images in a zero-shot setting. To harness this for completion, we first develop a Depth Prompting module that links partial point clouds with image-to-3D generative models by leveraging depth images as a stepping stone. To retain the original partial structure in the final results, we design the Geometric Preserving Fusion module that aligns the generated shape with input by adaptively adjusting its pose and scale. Extensive experiments on widely used benchmarks validate the superiority and generalizability of our approach, bringing us a step closer to robust real-world scan completion.

Figures (8)

• Figure 1: Difference between our GenPC with previous zero-shot point cloud completion method sds-complete. (a) SDS-Complete sds-complete uses the SDS loss to directly extract prior knowledge from a 2D diffusion model, featuring time-consuming optimization and suboptimal completion results. (b) The proposed GenPC leverages explicit priors provided by a 3D generative model, achieving improved completion quality with significantly reduced inference time.
• Figure 2: The architecture of GenPC. The Depth Prompting module first prompts the depth-guided 2D generative model with the partial input and generates an RGB image, which is fed into an image-to-3D generative model, producing a 3D shape. The Geometric Preserving Fusion module then integrates the generated shape with the partial point cloud.
• Figure 3: Illustration of Depth Prompting. (a) Overview. First, we uniformly position cameras around the partial point cloud $P_{in}$ to select a scanning viewpoint. From this viewpoint, we project to obtain depth and the corresponding mask, and then apply mask inpainting to achieve high-quality depth. (b) Viewpoint Selection: For each viewpoint $V_i$, we perform a spherical flip on $P_{in}$ for each camera to obtain a mirrored point cloud $\hat{P_{in}}$, then create a convex hull around $\hat{P_{in}} \cup V_i$, identifying the points on this hull as visible points. The camera with the greatest number of visible points is chosen as the scan viewpoint $V_{scan}$. The top of (b) is a true viewpoint, all points lie on the convex hull. The bottom of (b) is the opposite viewpoint, only two lie on the convex hull.
• Figure 4: Illustration of Geometric Preserving Fusion. In the Dynamic Scale Adaptation stage, an optimal scale factor is selected to align $P_{partial}$ and $P_{gen}$, producing an initial completed point cloud $P_{all}$. Then, to reduce the accumulated error in the previous steps, an optional Refining operation can be performed, where $P_{all}$ is initialized as 3D Gaussians and optimized by the SDS loss.
• Figure 5: Visual comparisons with recent methods PointrSnowflakenetAdaPoinTr on the Redwood dataset.