Sparse Point Cloud Patches Rendering via Splitting 2D Gaussians

TL;DR

This work tackles photo-realistic rendering from sparse point clouds by predicting 2D Gaussians directly from the input and rasterizing them for rendering. It introduces an entire-patch architecture with two identical 2D Gaussian prediction modules and a splitting decoder to densify Gaussians, enabling robust generalization across categories and effective handling of sparse data. The approach achieves state-of-the-art render quality with rapid, refinement-free rendering, and demonstrates strong generalization on cross-dataset evaluations, while offering transparent ablations and speed advantages. The model’s ability to render high-fidelity images from sparse data promises practical impact for real-time visualization, virtual reality, and autonomous systems where dense point clouds are unavailable.

Abstract

Current learning-based methods predict NeRF or 3D Gaussians from point clouds to achieve photo-realistic rendering but still depend on categorical priors, dense point clouds, or additional refinements. Hence, we introduce a novel point cloud rendering method by predicting 2D Gaussians from point clouds. Our method incorporates two identical modules with an entire-patch architecture enabling the network to be generalized to multiple datasets. The module normalizes and initializes the Gaussians utilizing the point cloud information including normals, colors and distances. Then, splitting decoders are employed to refine the initial Gaussians by duplicating them and predicting more accurate results, making our methodology effectively accommodate sparse point clouds as well. Once trained, our approach exhibits direct generalization to point clouds across different categories. The predicted Gaussians are employed directly for rendering without additional refinement on the rendered images, retaining the benefits of 2D Gaussians. We conduct extensive experiments on various datasets, and the results demonstrate the superiority and generalization of our method, which achieves SOTA performance. The code is available at https://github.com/murcherful/GauPCRender}{https://github.com/murcherful/GauPCRender.

Figures (8)

• Figure 1: Our method excels in rendering intricate, photo-realistic images from point clouds of different categories. In this instance, our model is trained on the Car category, utilizing 20K points as input.
• Figure 2: Overview of our proposed method. Our method predicts 2D Gaussians for point cloud rendering, employing an entire-patch architecture (bottom) and the 2D Gaussian Prediction Module (top left) with splitting decoders (top right).
• Figure 3: The illustration of our initialization approach, where the left side of each picture is a 2D schematic diagram and the right side is the rendered images of Gaussians. (a) The estimated normals of the point cloud. (b) Randomly Initialization. (c) Our Initialization. (d) Predicted 2D Gaussians.
• Figure 4: The evaluation of our method, TriVol and PFGS trained with different point numbers on Car category. The legend in the lower right corner indicates different methods.
• Figure 5: The rendering results of our method and previous methods on different categories. From top to bottom: scene, car, chair, shoe and human body.