ViSNeRF: Efficient Multidimensional Neural Radiance Field Representation for Visualization Synthesis of Dynamic Volumetric Scenes

TL;DR

ViSNeRF introduces a multidimensional neural radiance field framework for visualization synthesis of dynamic volumetric scenes, addressing data I/O/storage bottlenecks by enabling 3D-aware rendering from sparse image samples and flexible parameter-space exploration. It factorizes both the spatial radiance field and the parameter dimensions, employing a hybrid explicit-implicit representation with a dual MLP decoder, SH-encoded viewing directions, and CP-based CP decomposition for scalable multi-parameter control. The approach achieves superior quality across static and dynamic datasets (PSNR/SSIM/LPIPS), requires substantially fewer training images than prior methods, and supports high-resolution outputs (1024×1024) with interactive exploration, albeit with training-time constraints. The combination of coarse-to-fine optimization, ray/voxel skipping, and compact factorization makes ViSNeRF a practical and scalable tool for visualization synthesis in large-scale simulations, with code available publicly.

Abstract

Domain scientists often face I/O and storage challenges when keeping raw data from large-scale simulations. Saving visualization images, albeit practical, is limited to preselected viewpoints, transfer functions, and simulation parameters. Recent advances in scientific visualization leverage deep learning techniques for visualization synthesis by offering effective ways to infer unseen visualizations when only image samples are given during training. However, due to the lack of 3D geometry awareness, existing methods typically require many training images and significant learning time to generate novel visualizations faithfully. To address these limitations, we propose ViSNeRF, a novel 3D-aware approach for visualization synthesis using neural radiance fields. Leveraging a multidimensional radiance field representation, ViSNeRF efficiently reconstructs visualizations of dynamic volumetric scenes from a sparse set of labeled image samples with flexible parameter exploration over transfer functions, isovalues, timesteps, or simulation parameters. Through qualitative and quantitative comparative evaluation, we demonstrate ViSNeRF's superior performance over several representative baseline methods, positioning it as the state-of-the-art solution. The code is available at https://github.com/JCBreath/ViSNeRF.

Figures (21)

• Figure 1: Overview of ViSNeRF using the Nyx dataset as an example of a dynamic volumetric scene. Features are sampled from spatial and parameter feature grids based on the camera ray's sampling position and scene parameters. These features are processed by the decoder to generate density and color values, which are then used in volume rendering to visualize the scene frame from the chosen camera view.
• Figure 1: Novel view synthesis of DVR images for static scenes. (a) to (h): InSituNet, CoordNet, StyleGAN2, EG3D, NeRF, 3DGS, Instant-NGP, TensoRF, ViSNeRF, and GT. Top to bottom: vortex, five jets, Tangaroa, and supernova.
• Figure 2: Inferred five jets (timestep) DVR images generated by K-Planes, HexPlane, and ViSNeRF using 462 views to train the dynamic scene with a full 360-degree view.
• Figure 2: Novel view synthesis of IR images for static scenes. (a) to (h): InSituNet, CoordNet, StyleGAN2, EG3D, NeRF, 3DGS, Instant-NGP, TensoRF, ViSNeRF, and GT. Top to bottom: vortex, five jets, Tangaroa, and supernova.
• Figure 3: Inferred DVR images of the Tangaroa dataset using 42 views to train the full 360-degree view. Zoom-in and difference images are provided for better comparison.