Volume Encoding Gaussians: Transfer Function-Agnostic 3D Gaussians for Volume Rendering

TL;DR

Volume Encoding Gaussians (VEG) tackles the challenge of visualizing massive HPC volumes by learning a compact, transfer-function-agnostic 3D Gaussian representation that encodes only scalar fields. Through opacity-guided, differentiable rendering with Blinn-Phong lighting, VEG decouples visual appearance from data, enabling interactive visualization with arbitrary color/opacity mappings at render time. VEG demonstrates superior generalization to unseen transfer functions and substantially smaller memory footprints and training times compared with state-of-the-art 3D Gaussian Splatting approaches. The approach enables efficient, interactive volume visualization across structured and unstructured data and opens avenues for time-varying data and scalable training in HPC contexts.

Abstract

Visualizing the large-scale datasets output by HPC resources presents a difficult challenge, as the memory and compute power required become prohibitively expensive for end user systems. Novel view synthesis techniques can address this by producing a small, interactive model of the data, requiring only a set of training images to learn from. While these models allow accessible visualization of large data and complex scenes, they do not provide the interactions needed for scientific volumes, as they do not support interactive selection of transfer functions and lighting parameters. To address this, we introduce Volume Encoding Gaussians (VEG), a 3D Gaussian-based representation for volume visualization that supports arbitrary color and opacity mappings. Unlike prior 3D Gaussian Splatting (3DGS) methods that store color and opacity for each Gaussian, VEG decouple the visual appearance from the data representation by encoding only scalar values, enabling transfer function-agnostic rendering of 3DGS models. To ensure complete scalar field coverage, we introduce an opacity-guided training strategy, using differentiable rendering with multiple transfer functions to optimize our data representation. This allows VEG to preserve fine features across the full scalar range of a dataset while remaining independent of any specific transfer function. Across a diverse set of volume datasets, we demonstrate that our method outperforms the state-of-the-art on transfer functions unseen during training, while requiring a fraction of the memory and training time.

Figures (7)

• Figure 1: Diagram of our training pipeline, where operation flow is depicted with black arrows and gradient backpropagation is depicted with blue. We conduct image-space training, in which our differentiable rendering algorithm is used to optimize Volume Encoding Gaussians from loss computed against ground truth images. Each training step, we render our VEG by opacity/color mapping with a given transfer function, applying lighting effects with the Blinn-Phong model, projection into 2D with a given camera view, then rasterization and blending of the 2D colored primitives. We then compare the rendered output against shaded reference images created from volume rendering with an emission-absorption model. After computing loss, gradients are backpropagated through to the scaling, rotation, position, scalar values, weight and lighting parameters of the VEG for optimization. Gradients are also used for densification and pruning of the VEG, in which the density of 3D Gaussians in space is adaptively controlled.
• Figure 2: Diagram of our initialization strategy to create VEG from volume data. Our approach begins by deleting all connectivity information, leaving only scalar-valued data points. Next, we extend these data points with 3D scaling and rotation matrices, so that each point is a 3D Gaussian with attached data value. These are initialized to make each initial 3D Gaussian a sphere with size relative to the density of other points around it, with lower density creating larger Gaussians.
• Figure 3: Gaussian density comparison of VEG models trained using either a linear opacity map, an inverse linear opacity map, or our multiple opacity map method using three steps on the RBL dataset. Reference shows volume rendering with a pale rainbow colormap, where high values are red and low values are pale blue. Gaussian centers are painted using deep blue for each of the trained models, on top of the reference image, using SuperSplat'ssupersplat 3D Gaussian editor. Selected areas of high value are highlighted with an orange box, and areas of low value are highlighted with a yellow box. Notice training with the linear transfer function (which gives high opacity to high values) leads to greater Gaussian density in areas of high values and very low density in areas of low values, while the model trained with the inverse linear map does the opposite. Our multiple opacity map method leads to equal density for both areas of the function range.
• Figure 4: Interactive VEG renderer adapted from Tang et al. ivrgs_2025. Because VEG models are trained to generalize to arbitrary transfer functions, we replace their per-basic-set color and opacity sliders with selection of colormaps and an interactive opacity map creation tool.
• Figure 5: Qualitative comparison of our method with iVR-GS on the Vortex and Chameleon datasets. For each dataset and method, we show results from the tests run in Table \ref{['tab:evaluation_results']}, along with difference images showing perceivable difference in the CIE LUV color space between reconstructions and ground truth. Difference images use the Matplotlib 'jet' colormap with linear opacity map, shown at top, to highlight areas of low error (purple to blue), medium error (cyan to green), and high error (yellow to red). These results showcase how VEG models outperform iVR-GS composed models for unseen color and opacity mappings. For example, see "Broad Opacity" for chameleon, where the iVR-GS model incorrectly assigns opacity to large portions of the volume that are supposed to be transparent.