NARVis: Neural Accelerated Rendering for Real-Time Scientific Point Cloud Visualization

TL;DR

NAR addresses the challenge of real-time, high-fidelity visualization of massive scientific point clouds by learning a neural post-processing pass that augments a fast, multi-stream rasterizer. It couples a high-performance MSR with a U-Net-based renderer trained offline to imitate a chosen high-quality renderer (Gaussian Splatting), enabling real-time rendering from the original PC data with view-dependent effects. Demonstrations on Hurricane, Storms, and Morro Bay show competitive fidelity (PSNR/SSIM) at interactive frame rates (e.g., >126 fps on large datasets) and modest memory footprints (~12 GB), with strong generalization to related PCs and meaningful ablations of components and data streams. The work offers a scalable, end-to-end framework for immersive scientific visualization, suitable for XR and large-scale analytics, by balancing rendering speed, memory use, and visual fidelity through learnable post-processing.

Abstract

Exploring scientific datasets with billions of samples in real-time visualization presents a challenge - balancing high-fidelity rendering with speed. This work introduces a novel renderer - Neural Accelerated Renderer (NAR), that uses the neural deferred rendering framework to visualize large-scale scientific point cloud data. NAR augments a real-time point cloud rendering pipeline with high-quality neural post-processing, making the approach ideal for interactive visualization at scale. Specifically, we train a neural network to learn the point cloud geometry from a high-performance multi-stream rasterizer and capture the desired postprocessing effects from a conventional high-quality renderer. We demonstrate the effectiveness of NAR by visualizing complex multidimensional Lagrangian flow fields and photometric scans of a large terrain and compare the renderings against the state-of-the-art high-quality renderers. Through extensive evaluation, we demonstrate that NAR prioritizes speed and scalability while retaining high visual fidelity. We achieve competitive frame rates of $>$ 126 fps for interactive rendering of $>$ 350M points (i.e., an effective throughput of $>$ 44 billion points per second) using $\sim$12 GB of memory on RTX 2080 Ti GPU. Furthermore, we show that NAR is generalizable across different point clouds with similar visualization needs and the desired post-processing effects could be obtained with substantial high quality even at lower resolutions of the original point cloud, further reducing the memory requirements.

Figures (11)

• Figure 1: We present Neural Accelerated Renderer (NAR) for high-quality visualization of large scientific point clouds with low motion-to-photon latency. We show that NAR can visualize a variety of scientific use cases, such as visualizing (a) static vector fields (Hurricane), (b) particle trajectories (Storms), and (c) photometric terrain scans (Morro Bay). (d) NAR renders high-quality images comparable to the rendering quality of conventional high-quality renderers (such as the Gaussian Splatting (GSplat) renderer) with significantly lower latency and memory footprints. (Note: the axes are in log-scale and the bubble sizes are proportional to the labeled number of points.)
• Figure 2: NAR Overview. (a) We train NAR with the large high-resolution scientific PC data and a preferred renderer (Gaussian Splatting forward renderer kerbl3Dgaussians in our case) for learning geometry and appearance respectively. $I_O$ and $I_R$ represent the rendering from the conventional renderer and NAR respectively. (b) We only use the original (or modified) PC to render real-time high-quality renderings based on the user's viewing directions for inference.
• Figure 3: Details of the NAR. We take the input view and the PC to render view-dependent PC features with an MSR and pass these features to our rendering network, a U-Net ronneberger2015u, to generate final renderings. Before passing to the U-Net, we down scale and apply a $1\times1$-Convolution (retaining the number of channels) to the multi-stream features and concatenate ($\bigoplus$) them to the first U-Net blocks at the respective resolutions. (Green blocks: down scaled and $1\times1$ convolved multi-stream features, Pink blocks: U-Net block outputs.)
• Figure 4: Gaussian splatting rendering effects for PC visualization and training data creation.
• Figure 5: Qualitative comparison of different renderers. NPBG and NAR are trained on GSplat renderings. $^\dagger$GSplat and NPBG used 0.5× and 0.1× the original number of MorroBay points, respectively, due to memory constraints. More results in Section \ref{['app:addres']} of the supplementary.