Low Latency Point Cloud Rendering with Learned Splatting

TL;DR

This work trains a generic neural network to estimate 3D elliptical Gaussians from arbitrary point clouds and uses differentiable surface splatting to render smooth texture and surface normal for arbitrary views, and enables real-time rendering of dynamic point cloud.

Abstract

Point cloud is a critical 3D representation with many emerging applications. Because of the point sparsity and irregularity, high-quality rendering of point clouds is challenging and often requires complex computations to recover the continuous surface representation. On the other hand, to avoid visual discomfort, the motion-to-photon latency has to be very short, under 10 ms. Existing rendering solutions lack in either quality or speed. To tackle these challenges, we present a framework that unlocks interactive, free-viewing and high-fidelity point cloud rendering. We train a generic neural network to estimate 3D elliptical Gaussians from arbitrary point clouds and use differentiable surface splatting to render smooth texture and surface normal for arbitrary views. Our approach does not require per-scene optimization, and enable real-time rendering of dynamic point cloud. Experimental results demonstrate the proposed solution enjoys superior visual quality and speed, as well as generalizability to different scene content and robustness to compression artifacts. The code is available at https://github.com/huzi96/gaussian-pcloud-render .

Figures (13)

• Figure 1: Rendering by estimating elliptical parameters from point cloud using sparse 3D convolutional neural network.
• Figure 2: The UNet-like architecture with 3D convolutions. The network takes a point cloud as input and predicts the elliptical parameters and surface normal for each point.
• Figure 3: Average PSNR of rendered views from THuman 2.0 compact point clouds compressed to different bit-rates by G-PCC.
• Figure 4: Comparison of rendering results of a point cloud (1.7M points) in the BlendedMVS dataset. Quality metrics are shown in format (PSNR, MS-SSIM) and calculated from the rasterization results of the ground truth mesh. The insets visualize local details with $3\times$ zooming.
• Figure 5: Rendering results in the compact setting of a point cloud in the Thuman 2.0 Dataset. The first row shows the rendered RGB views. The second row shows the surface normal. The insets visualize local details with $4\times$ zooming.