ComPC: Completing a 3D Point Cloud with 2D Diffusion Priors

TL;DR

This paper tackles the challenge of completing 3D point clouds captured from partial views, especially across unseen categories. It introduces ComPC, a training-free test-time framework that uses pre-trained 2D diffusion priors guided through 3D Gaussian Splatting to fill missing regions, starting from a reference viewpoint. The method comprises Partial Gaussian Initialization (PGI) to generate a guiding reference image, Zero-shot Fractal Completion (ZFC) to optimize 3D Gaussians under diffusion guidance with a Preservation Constraint, and Point Cloud Extraction (PCE) to produce uniform surface points, enabling efficient, generalizable completion. Experiments on synthetic data, Redwood real scans, and ShapeNet/Kitti variants show robust performance, improved generalization to unseen categories, and faster optimization relative to prior prompt-based methods, with a practical, training-free workflow disclosed on the project page.

Abstract

3D point clouds directly collected from objects through sensors are often incomplete due to self-occlusion. Conventional methods for completing these partial point clouds rely on manually organized training sets and are usually limited to object categories seen during training. In this work, we propose a test-time framework for completing partial point clouds across unseen categories without any requirement for training. Leveraging point rendering via Gaussian Splatting, we develop techniques of Partial Gaussian Initialization, Zero-shot Fractal Completion, and Point Cloud Extraction that utilize priors from pre-trained 2D diffusion models to infer missing regions and extract uniform completed point clouds. Experimental results on both synthetic and real-world scanned point clouds demonstrate that our approach outperforms existing methods in completing a variety of objects. Our project page is at \url{https://tianxinhuang.github.io/projects/ComPC/}.

Figures (18)

• Figure 1: Different point cloud completion methods. (a) Existing network-based completion methods; (b) Test-time SDS-complete kasten2024point with text prompts to guide Neural surface for completion; (c) Our method based on 3D Gaussian Splatting (GS) guided by the diffusion model from Zero 1-to-3 liu2023zero conditioned on the reference image rendered from partial points.
• Figure 2: Illustration of our framework. ①In Partial Gaussian Initialization (PGI), Reference Viewpoint Estimation estimates a camera pose $V_p$ where $P_{in}$ can be most completely observed. We initialize 3D Gaussians $G_{in}$ from $P_{in}$ and render the reference image $I_{in}$ under $V_p$. ②In Zero-shot Fractal Completion (ZFC), 3D Gaussians $G_m$ begins with an initialization using noisy $P_{N}$ and undergoes optimization guided by view-dependent guidance from the diffusion model $f_Z$ in Zero 1-to-3 liu2023zero based on a randomly chosen camera pose $V_i$. Additionally, it incorporates a Preservation Constraint computed with respect to $V_p$. $G_{in}$ is mixed with $G_m$ to form $G_{all}$, introducing the partial geometry from $P_{in}$. ③After ZFC, we use Point Cloud Extraction (PCE) to extract surface points $P_{surf}$ from centers of $G_{all}$, and convert $P_{surf}$ into uniform $P_{out}$ with Grid Pulling.
• Figure 3: Differences between our binarized opacity and original continuous opacity. $\prec$ denotes smaller but not approaching.
• Figure 4: Illustration of Grid Pulling module. $g(\cdot)$ is a MLP-based SDF learned from the completed point cloud $P_{surf}$. Merge denotes merge layer from huang2021rfnet. Given the 3D grids $\mathcal{G}$, $r$ is the diagonal length of a unit grid. Sampled points would be $P_s = \{p \mid g(p)<0.5r, p \in \mathcal{G}\}$.
• Figure 5: Qualitative comparison on synthetic data.