Unsupervised 3D Point Cloud Completion via Multi-view Adversarial Learning

TL;DR

MAL-UPC addresses unsupervised 3D point cloud completion from single-view partial observations by introducing a Pattern Retrieval Network to exploit region-level geometric repetitiveness and a Multi-view Adversarial Network to capture category-specific priors via depth-map rendering. The framework renders partial completions into silhouettes and multi-view depth maps, using a category-specific image bank and a 2D discriminator to refine geometry without requiring complete 3D supervision. It combines a partial-chamfer loss, rendering loss, density loss, and adversarial loss, with a density-aware radius estimation to improve depth-map rendering quality. Experiments on ShapeNet-based synthetic data and real-world scans show strong performance against self-supervised and some unpaired methods, with ablations confirming the contributions of each module. The approach provides a practical path toward unsupervised, geometry-informed point cloud completion suitable for real-world applications.

Abstract

In real-world scenarios, scanned point clouds are often incomplete due to occlusion issues. The tasks of self-supervised and weakly-supervised point cloud completion involve reconstructing missing regions of these incomplete objects without the supervision of complete ground truth. Current methods either rely on multiple views of partial observations for supervision or overlook the intrinsic geometric similarity that can be identified and utilized from the given partial point clouds. In this paper, we propose MAL-UPC, a framework that effectively leverages both region-level and category-specific geometric similarities to complete missing structures. Our MAL-UPC does not require any 3D complete supervision and only necessitates single-view partial observations in the training set. Specifically, we first introduce a Pattern Retrieval Network to retrieve similar position and curvature patterns between the partial input and the predicted shape, then leverage these similarities to densify and refine the reconstructed results. Additionally, we render the reconstructed complete shape into multi-view depth maps and design an adversarial learning module to learn the geometry of the target shape from category-specific single-view depth images of the partial point clouds in the training set. To achieve anisotropic rendering, we design a density-aware radius estimation algorithm to improve the quality of the rendered images. Our MAL-UPC outperforms current state-of-the-art self-supervised methods and even some unpaired approaches. We will make the source code publicly available at https://github.com/ltwu6/malspc

Figures (10)

• Figure 1: Flowchart of the proposed unsupervised point cloud completion method. Given a depth map $\mathbf{D}_0$, we can obtain an incomplete point cloud $\mathbf{P}_{in}$ by back-projecting it into 3D space. We first employ an auto-encoder to generate a coarse shape, and then introduce a pattern retrieval module to guide the refinement and densification of the shape based on the position and curvature patterns of $\mathbf{P}_{in}$. Then, we render the generated point cloud into a silhouette map from the input viewpoint and depth maps from various random viewpoints. The silhouette map is supervised by the mask image binarized from $\mathbf{D}_0$, while the depth maps are used for adversarial learning with the category-specific single-view depth maps of the partial point clouds from the training set.
• Figure 2: The flowchart of our Pattern Retrieval Network. "FC" and "MLP" denote the Fully-Connected layer and the Multi-Layer Perceptron layer. Given a partial point cloud $\mathbf{P}_{in}$, we first utilize an auto-encoder to generate a coarse shape $\mathbf{P}_c$. Next, we separately extract position encodings and curvature encodings for both $\mathbf{P}_{in}$ and $\mathbf{P}_c$, and then leverage a cross-attention-based module to identify and retrieve the most related encodings for each point in $\mathbf{P}_c$ from $\mathbf{P}_{in}$. Subsequently, $\mathbf{P}_c$ and the selected encodings are upsampled and fused to refine the shape.
• Figure 3: Depth maps rendered from point clouds with an equal point number. The rendering process employs a fixed and identical point radius. With a small foreground area, the visible surface can be well represented. However, when the foreground area is large, a fixed point radius cannot cover the entire visible surface, resulting in some foreground pixels being covered by invisible points.
• Figure 4: Qualitative results on the synthetic dataset. (a) Partial points. (b) Ground truth. (c) FoldingNet foldingnet. (d) PCN pcn. (e) AnchorFormer chen2023anchorformer. (f) KTNet cao2023kt. (g) USSPA ma2023symmetric. (h) ACL-SPC hong2023acl. (i) P2C cui2023p2c. (j) Ours.
• Figure 5: Qualitative results on the real-world dataset. The first raw represents the input incomplete point cloud, while the second to the last rows correspond to the results of PCN pcn, PCL2PCL pcl2plc, ACL-SPC hong2023acl, P2C cui2023p2c, and our method, respectively. (a)$\sim$(c) Car. (d)$\sim$(f) Chair. (g)$\sim$(i) Table.