GaussianPU: A Hybrid 2D-3D Upsampling Framework for Enhancing Color Point Clouds via 3D Gaussian Splatting

TL;DR

GaussianPU addresses the challenge of upsampling dense colored point clouds for robotics by avoiding patch-based segmentation and enabling large-scale processing on consumer GPUs. It presents a 2D-3D hybrid framework that combines $3D$ Gaussian Splatting with a dual-scale rendered-image restoration network, augmented by Gaussian interpolation to controllably increase point counts by a factor $R$. Key innovations include fixed-point-count constraints, a uniform Gaussian scale, and joint color-geometry losses within the 3DGS upsampling module, yielding improved perceptual and geometric quality. On the WPC dataset, GaussianPU delivers substantial gains in IW-SSIM and related metrics, producing millions of colored points suitable for autonomous navigation and manipulation on standard GPUs.

Abstract

Dense colored point clouds enhance visual perception and are of significant value in various robotic applications. However, existing learning-based point cloud upsampling methods are constrained by computational resources and batch processing strategies, which often require subdividing point clouds into smaller patches, leading to distortions that degrade perceptual quality. To address this challenge, we propose a novel 2D-3D hybrid colored point cloud upsampling framework (GaussianPU) based on 3D Gaussian Splatting (3DGS) for robotic perception. This approach leverages 3DGS to bridge 3D point clouds with their 2D rendered images in robot vision systems. A dual scale rendered image restoration network transforms sparse point cloud renderings into dense representations, which are then input into 3DGS along with precise robot camera poses and interpolated sparse point clouds to reconstruct dense 3D point clouds. We have made a series of enhancements to the vanilla 3DGS, enabling precise control over the number of points and significantly boosting the quality of the upsampled point cloud for robotic scene understanding. Our framework supports processing entire point clouds on a single consumer-grade GPU, such as the NVIDIA GeForce RTX 3090, eliminating the need for segmentation and thus producing high-quality, dense colored point clouds with millions of points for robot navigation and manipulation tasks. Extensive experimental results on generating million-level point cloud data validate the effectiveness of our method, substantially improving the quality of colored point clouds and demonstrating significant potential for applications involving large-scale point clouds in autonomous robotics and human-robot interaction scenarios.

Figures (5)

• Figure 1: The comparison between previous point cloud upsampling methods and ours. (Up) Previous methods primarily relied on patch-based upsampling, processing point clouds patch by patch. (Down) In contrast, our approach can directly upsample the entire point cloud, thereby avoiding the degradation in quality that can arise from the patch-based approach.
• Figure 2: The framework of our proposed GaussianPU. In the data preparation stage, the sparse point cloud undergoes a rendering process followed by a restoration step to obtain dense rendered images. Simultaneously, the sparse point cloud is subjected to Gaussian interpolation to acquire a $R \times$ interpolated point cloud. The interpolated point cloud and dense rendered images obtained from the data preparation stage are fed into the 3DGS point cloud upsampling module along with the camera poses. Through iterative optimization, the 3DGS upsampling module generates a dense colored point cloud.
• Figure 3: Sparse point cloud rendering with different point sizes and their errors compared to dense point cloud rendering.
• Figure 4: Visualization of $4\times$ upsampling results. Our method achieves the best quality, exhibiting uniform and smooth surfaces with finer-grained detail representation. The upsampled point clouds show significant improvements in quality compared to the input sparse point clouds.
• Figure 5: Geometric visualization of $4\times$ upsampling results. It is evident that our proposed method is capable of generating denser point clouds with higher uniformity and precision compared to other approaches.