SPU-IMR: Self-supervised Arbitrary-scale Point Cloud Upsampling via Iterative Mask-recovery Network

TL;DR

This work reframes point cloud upsampling as global shape completion by masking and iteratively recovering patches in a self-supervised setting. The SPU-IMR framework leverages local patch masking, a visible-feature-guided iterative deformation block, and Earth Mover's Distance loss to produce accurate, uniform outputs. A key contribution is the multi-mask-recovery (MMR) strategy enabling test-time arbitrary-scale upsampling without retraining, demonstrated across synthetic and real-world datasets with competitive quantitative and strong qualitative results. The approach offers practical value for applications like autonomous driving and 3D reconstruction by delivering dense, uniformly distributed point clouds with a single training process.

Abstract

Point cloud upsampling aims to generate dense and uniformly distributed point sets from sparse point clouds. Existing point cloud upsampling methods typically approach the task as an interpolation problem. They achieve upsampling by performing local interpolation between point clouds or in the feature space, then regressing the interpolated points to appropriate positions. By contrast, our proposed method treats point cloud upsampling as a global shape completion problem. Specifically, our method first divides the point cloud into multiple patches. Then, a masking operation is applied to remove some patches, leaving visible point cloud patches. Finally, our custom-designed neural network iterative completes the missing sections of the point cloud through the visible parts. During testing, by selecting different mask sequences, we can restore various complete patches. A sufficiently dense upsampled point cloud can be obtained by merging all the completed patches. We demonstrate the superior performance of our method through both quantitative and qualitative experiments, showing overall superiority against both existing self-supervised and supervised methods.

Figures (7)

• Figure 1: Comparative illustrations of mainstream methods and our method in 2D. Take a point or a patch as an example: (a) Feature-space-based interpolation methods interpolate points in feature space. (b) Point-cloud-based interpolation methods directly interpolate points around the input. (c) Our proposed method predicts the masked point cloud patch's local shape and topological changes by leveraging the visible patches.
• Figure 2: Illustration of the proposed SPU-IMR method. First, we obtain the center points and point cloud patches using FPS and KNN. We then remove some point cloud patches based on a given mask ratio, leaving only the visible patches. Guided by encoding the visible patch features and the center point position embedding, learnable generated point cloud patches are iteratively transformed and optimized in the iterative deformation block module, completing the removed point cloud patches. Finally, we use the EMD between all output point cloud patches and the real point cloud patches as the loss function.
• Figure 3: Illustration of the proposed multi-mask-recovery module. We first divide the point cloud into patches and then mask them using the selected mask sequence. Afterward, we import these patches into the Iterative Mask-recovery module. All output results will be merged with $\mathcal{P}_{in}$. Finally, after removing outliers through the Outliers Removal module, the desired upsampling point cloud can be obtained through FPS.
• Figure 4: $4 \times$ point cloud upsampling results of chair and airplane, with input size of $1024$. 'AS' indicates arbitrary-scale methods and 'SS' indicates self-supervised methods. Please zoom in for better viewing.
• Figure 5: Results with different upsampling ratios of eight.