Make the Fastest Faster: Importance Mask Synthesis for Interactive Volume Visualization using Reconstruction Neural Networks

TL;DR

This work tackles the latency bottleneck of interactive volume visualization by pairing reconstruction neural networks (RecNNs) with a learned importance masking strategy. It introduces two neural components: IML Net, which learns an Importance Mask (IM) from a compact embedding, and IMS Net, which synthesizes IMs directly from novel view parameters, all wired through differentiable compaction/decompaction layers to support multiple sampling patterns. The approach is demonstrated with EnhanceNet (super-resolution) and FoVolNet (foveated rendering) across four volumetric datasets, achieving substantial rendering-latency reductions while maintaining comparable PSNR/SSIM to full-pixel reconstructions. Importantly, IML Net can be trained independently of the downstream RecNN, enabling practical deployment on off-the-shelf pretrained networks and accelerating interactive visualization without lengthy retraining.

Abstract

Visualizing a large-scale volumetric dataset with high resolution is challenging due to the substantial computational time and space complexity. Recent deep learning-based image inpainting methods significantly improve rendering latency by reconstructing a high-resolution image for visualization in constant time on GPU from a partially rendered image where only a portion of pixels go through the expensive rendering pipeline. However, existing solutions need to render every pixel of either a predefined regular sampling pattern or an irregular sample pattern predicted from a low-resolution image rendering. Both methods require a significant amount of expensive pixel-level rendering. In this work, we provide Importance Mask Learning (IML) and Synthesis (IMS) networks, which are the first attempts to directly synthesize important regions of the regular sampling pattern from the user's view parameters, to further minimize the number of pixels to render by jointly considering the dataset, user behavior, and the downstream reconstruction neural network. Our solution is a unified framework to handle various types of inpainting methods through the proposed differentiable compaction/decompaction layers. Experiments show our method can further improve the overall rendering latency of state-of-the-art volume visualization methods using reconstruction neural network for free when rendering scientific volumetric datasets. Our method can also directly optimize the off-the-shelf pre-trained reconstruction neural networks without elongated retraining.

Figures (24)

• Figure 1: Typical pipeline of interactive volume visualization using RecNN.
• Figure 2: Demonstration of compaction and decompaction for super-resolution RecNN and foveated rendering RecNN.
• Figure 3: Network architecture of IML Net. IML Net works on a compact 2D embedding domain (C-Image) with resolution of $n\times n$, which is much smaller than the full resolution output with a resolution of $m\times m$ ($m \gg n$).
• Figure 4: Network architecture of the encoder and decoder in IML Net. The input image resolution is $n\times n$.
• Figure 5: Network architecture of IMS Net.