Efficient Point Clouds Upsampling via Flow Matching

TL;DR

Efficient Point Clouds Upsampling via Flow Matching (PUFM) treats upsampling as learning the optimal transport from sparse to dense point clouds rather than a noise-driven diffusion task. It combines midpoint interpolation to bridge density gaps with an Earth Mover's Distance (EMD) based pre-alignment to reduce permutation ambiguity, and then trains a velocity-field model via flow matching to map between distributions, enabling fast sampling with high fidelity. Quantitative results on synthetic datasets and qualitative assessments on RGB-D and LiDAR data show PUFM delivers superior upsampling quality with fewer sampling steps, while maintaining robustness to noise and generalizing to real-world scenes. This approach offers practical gains for 3D reconstruction and downstream perception tasks by delivering efficient, accurate, and scalable point cloud densification.

Abstract

Diffusion models are a powerful framework for tackling ill-posed problems, with recent advancements extending their use to point cloud upsampling. Despite their potential, existing diffusion models struggle with inefficiencies as they map Gaussian noise to real point clouds, overlooking the geometric information inherent in sparse point clouds. To address these inefficiencies, we propose PUFM, a flow matching approach to directly map sparse point clouds to their high-fidelity dense counterparts. Our method first employs midpoint interpolation to sparse point clouds, resolving the density mismatch between sparse and dense point clouds. Since point clouds are unordered representations, we introduce a pre-alignment method based on Earth Mover's Distance (EMD) optimization to ensure coherent interpolation between sparse and dense point clouds, which enables a more stable learning path in flow matching. Experiments on synthetic datasets demonstrate that our method delivers superior upsampling quality but with fewer sampling steps. Further experiments on ScanNet and KITTI also show that our approach generalizes well on RGB-D point clouds and LiDAR point clouds, making it more practical for real-world applications.

Figures (8)

• Figure 1: Convergence comparison among different distribution mapping paths. The diffusion model PUDM demonstrates slow convergence since it starts from noise distribution. Our proposed PUFM learns flow matching from sparse to dense point clouds. Pre-alignment is applied to minimize the learning ambiguity at the early stage, resulting in a more efficient upsampling process.
• Figure 2: Illustration of PUFM. It processes point clouds as patches (yellow circles), and learns $D_\theta$ to map the distribution from sparse to dense patches. It contains forward alignment and interpolation (a) and backward restoration and sampling (b). In (a), the model first pre-aligns the sparse (initially upsampled by midpoint interpolation) and dense point clouds and then randomly picks "noisy" points as $x_{\alpha_i}$ with the time step $\alpha_i$. In (b), the model optimizes the distribution flow to transform arbitrary sparse data into dense data effectively. During the sampling process, the model directly learns to densify the sparse points without further alignment. Note: the rotated sparse point clouds with 180 degrees illustrate the effect of misalignment between sparse and dense point clouds.
• Figure 3: A toy example on flow matching for point cloud transformation. Without pre-alignment, the model converges slower as it first transforms the source point clouds to a set of dispersed clusters and then gradually matches to the target point clouds. In contrast, the pre-alignment exhibits more efficient and consistent transformation.
• Figure 4: Visual comparison of different methods on 4$\times$ upsampling. We zoom in on the red box regions to highlight the point cloud upsampling differences. We also show the mesh reconstruction, and ours produces evenly distributed point clouds and smooth meshes.
• Figure 5: Visulization of arbitrary point cloud upsampling.