iVR-GS: Inverse Volume Rendering for Explorable Visualization via Editable 3D Gaussian Splatting

TL;DR

The paper tackles the high cost of real-time volume visualization by introducing iVR-GS, an inverse volume rendering framework based on editable 3D Gaussian splats. It uses multiple basic NVS models—each tied to a disjoint transfer-function (TF) range—and composes them into a full VolVis scene with editable Gaussian primitives that support color, opacity, and lighting edits via Blinn-Phong shading. A two-stage training regime (base 3DGS geometry followed by editable Gaussian optimization) plus vector quantization enables compact, composable representations and real-time rendering, with inverse volume exploration that can match a target reference image. Compared to Plenoxels, CCNeRF, and base 3DGS, iVR-GS delivers superior reconstruction quality, enables interactive scene editing, and achieves notable compression, offering practical pathways for explorable visualization on standard hardware.

Abstract

In volume visualization, users can interactively explore the three-dimensional data by specifying color and opacity mappings in the transfer function (TF) or adjusting lighting parameters, facilitating meaningful interpretation of the underlying structure. However, rendering large-scale volumes demands powerful GPUs and high-speed memory access for real-time performance. While existing novel view synthesis (NVS) methods offer faster rendering speeds with lower hardware requirements, the visible parts of a reconstructed scene are fixed and constrained by preset TF settings, significantly limiting user exploration. This paper introduces inverse volume rendering via Gaussian splatting (iVR-GS), an innovative NVS method that reduces the rendering cost while enabling scene editing for interactive volume exploration. Specifically, we compose multiple iVR-GS models associated with basic TFs covering disjoint visible parts to make the entire volumetric scene visible. Each basic model contains a collection of 3D editable Gaussians, where each Gaussian is a 3D spatial point that supports real-time scene rendering and editing. We demonstrate the superior reconstruction quality and composability of iVR-GS against other NVS solutions (Plenoxels, CCNeRF, and base 3DGS) on various volume datasets. The code is available at https://github.com/TouKaienn/iVR-GS.

Figures (15)

• Figure 1: The workflow of iVR-GS. (a) For each basic scene corresponding to one basic TF with a unique visible part, we render a collection of multi-view images as the training dataset of this scene. (b) Then, we optimize a basic iVR-GS model associated with each basic scene with editable Gaussians to enable scene editing. (c) Next, we compose multiple basic iVR-GS models into the composed iVR-GS model to make the entire VolVis scene visible. The composed iVR-GS supports (d) iterative real-time scene editing and (e) inverse volume exploration.
• Figure 1: The distribution of 42, 92, and 162 training views. The black pyramids denote the camera poses for capturing training images, and the red dots represent the camera positions for generating testing images.
• Figure 2: Comparing the reconstruction quality of iVR-GS trained with different numbers of multi-view images. The difference image shows the pixel-wise perceivable difference (purple/blue to cyan/green to yellow/red indicates low to medium to high) in the CIE LUV color space. Top and bottom: complex and simple scenes.
• Figure 2: Comparing relighting results of iVR-GS and LERP. Left to right: the light source moves from bottom to top.
• Figure 3: The screenshot of iVR-GS interface showing rendering results of the combustion dataset with ten basic TFs. (a) Rendering result of the iVR-GS. (b) Rendering options for selecting rendering modes, saving images, etc. (c) Color editors and sliders for color and opacity adjustment. (d) Light editing panel for lighting effect adjustment.