PointSea: Point Cloud Completion via Self-structure Augmentation

TL;DR

PointSea tackles point cloud completion by learning from self-structured cues, avoiding reliance on extra color or viewpoint data. It introduces SVFNet to synthesize a global shape from cross-modal self-projections and a dual-path SDG to refine local details with adaptive feature fusion. The two-stage design—incompleteness-aware Structure Analysis and Similarity Alignment—paired with Path Selection yields robust performance across ShapeNet, PCN, and real-world scans, and extends to semantic scene completion. The approach demonstrates strong global understanding and local fidelity, with clear gains over existing methods and practical applicability to real-world 3D understanding tasks.

Abstract

Point cloud completion is a fundamental yet not well-solved problem in 3D vision. Current approaches often rely on 3D coordinate information and/or additional data (e.g., images and scanning viewpoints) to fill in missing parts. Unlike these methods, we explore self-structure augmentation and propose PointSea for global-to-local point cloud completion. In the global stage, consider how we inspect a defective region of a physical object, we may observe it from various perspectives for a better understanding. Inspired by this, PointSea augments data representation by leveraging self-projected depth images from multiple views. To reconstruct a compact global shape from the cross-modal input, we incorporate a feature fusion module to fuse features at both intra-view and inter-view levels. In the local stage, to reveal highly detailed structures, we introduce a point generator called the self-structure dual-generator. This generator integrates both learned shape priors and geometric self-similarities for shape refinement. Unlike existing efforts that apply a unified strategy for all points, our dual-path design adapts refinement strategies conditioned on the structural type of each point, addressing the specific incompleteness of each point. Comprehensive experiments on widely-used benchmarks demonstrate that PointSea effectively understands global shapes and generates local details from incomplete input, showing clear improvements over existing methods.

Figures (17)

• Figure 1: We explore Self-structure Augmentation, a novel strategy for high-quality global-to-local point cloud completion, dubbed PointSea. (a) In the global stage, PointSea understands incomplete shapes from self-projected multiple views. (b) In the local stage, PointSea collaborates both similar geometric similarities in input (red boxes) and learned shape priors (green boxes) for shape refinement. (c) PointSea shows clear improvements over its competitors, including SeedFormer zhou2022seedformer, AdaPoinTr 10232862, and SVDFormer Zhu_2023_ICCV.
• Figure 2: The original point cloud of large and complex objects is often incomplete due to many factors. Point cloud completion plays an essential role in extensive practical applications.
• Figure 3: The architecture of PointSea. SVFNet first generates a global shape from the cross-modal input. The coarse completion is then upsampled and refined with two SDGs.
• Figure 4: Illustration of feature fusion module. The features from cross-modal input are fused at both inter-view and intra-view levels.
• Figure 5: The architecture of SDG. The upper path represents Structure Analysis and the lower path represents Similarity Alignment. Each sub-network generates an offset feature which is then combined using a Path Selection module and used to regress into the coordinate offsets.