Adaptively Placed Multi-Grid Scene Representation Networks for Large-Scale Data Visualization

TL;DR

This work addresses the lack of adaptive parameter placement and scalability in scene representation networks (SRNs) for large-scale scientific visualization. It introduces Adaptively Placed Multi-Grid SRN (APMGSRN), which uses multiple learnable feature grids with transformation matrices to focus capacity where reconstruction error is high, and a domain-decomposition training/inference strategy to fit volumes too large for a single GPU. A differentiable feature-density loss based on flat-top Gaussians guides grid placement, and a lightweight decoder yields accurate scalar fields with efficient decoding. The authors also provide an open-source neural volume renderer for plug-and-play visualization with PyTorch SRNs and demonstrate improved reconstruction quality and scalable training on volumes up to 450 GB, with interactive rendering capabilities. While compression performance lags behind some dedicated compressors, the approach enables high-quality neural representations and interactive visualization for very large scientific datasets, with clear paths for future optimizations and extensions.

Abstract

Scene representation networks (SRNs) have been recently proposed for compression and visualization of scientific data. However, state-of-the-art SRNs do not adapt the allocation of available network parameters to the complex features found in scientific data, leading to a loss in reconstruction quality. We address this shortcoming with an adaptively placed multi-grid SRN (APMGSRN) and propose a domain decomposition training and inference technique for accelerated parallel training on multi-GPU systems. We also release an open-source neural volume rendering application that allows plug-and-play rendering with any PyTorch-based SRN. Our proposed APMGSRN architecture uses multiple spatially adaptive feature grids that learn where to be placed within the domain to dynamically allocate more neural network resources where error is high in the volume, improving state-of-the-art reconstruction accuracy of SRNs for scientific data without requiring expensive octree refining, pruning, and traversal like previous adaptive models. In our domain decomposition approach for representing large-scale data, we train an set of APMGSRNs in parallel on separate bricks of the volume to reduce training time while avoiding overhead necessary for an out-of-core solution for volumes too large to fit in GPU memory. After training, the lightweight SRNs are used for realtime neural volume rendering in our open-source renderer, where arbitrary view angles and transfer functions can be explored. A copy of this paper, all code, all models used in our experiments, and all supplemental materials and videos are available at https://github.com/skywolf829/APMGSRN.

Figures (9)

• Figure 1: Examples of our adaptively placed feature grids fitting to volumes during training. The feature grids find volume-specific regions to cover to maximize reconstruction accuracy. Video of evolution of grids during training available in supplemental materials.
• Figure 2: Comparison of our APMGSRN with other state-of-the-art models fVSRN Weiss22_fvsrn and NGP Muller22_ngp. The $\oplus$ operator represents concatenation.
• Figure 3: An overview of the APMGSRN architecture. In (1), a query coordinate is transformed into each of the $M$ grid's local coordinate systems learned by the transformation matrices, where grid local extents are assumed to be $[-1, 1]^3$. (2), the local coordinates are used to trilinearly interpolate within each feature grid (of resolution $D \times H \times W$) to obtain the corresponding feature for each grid. If the coordinate is out of bounds, zeros are returned instead. The resulting $M \cdot C$ features are concatenated into a feature vector. Steps (1) and (2) are considered the encoding process. In step (3), the feature vector is decoded in a shallow MLP for the final output value $f(x)$.
• Figure 4: Examples of 3 grid initializations and the effect on final grid locations and data reconstruction PSNR. Underlying data volume rendered depicts ground truth data for clarity of grid locations with respect to the raw data.
• Figure 5: Comparison of APMGSRN with TTHRESH and SZ for compressing the isotropic volume ($1024^3$). Our "compression" time is the time it takes to train the model, and our decompression time is the time it takes to query each point in the volume.