StyleRF-VolVis: Style Transfer of Neural Radiance Fields for Expressive Volume Visualization

TL;DR

StyleRF-VolVis tackles the problem of expressive style transfer in volume visualization by introducing a three-stage NeRF-based pipeline that separates content (geometry) from style (appearance). It integrates a base NeRF for geometry, a Palette Color Network for photorealistic editing, and an Unrestricted Color Network trained via knowledge distillation for non-photorealistic editing, enabling region-wise style transfer from arbitrary references while preserving multi-view consistency. The work demonstrates strong qualitative and quantitative results against image-, video-, and NeRF-based baselines, showing improved cross-view consistency and flexible NPSE capabilities, with a GUI for interactive editing. This approach advances visualization synthesis by offering high-quality, consistent, and flexible 3D stylization of volumetric data, with potential extensions to dynamic scenes, multivariate volumes, and natural-language interfaces.

Abstract

In volume visualization, visualization synthesis has attracted much attention due to its ability to generate novel visualizations without following the conventional rendering pipeline. However, existing solutions based on generative adversarial networks often require many training images and take significant training time. Still, issues such as low quality, consistency, and flexibility persist. This paper introduces StyleRF-VolVis, an innovative style transfer framework for expressive volume visualization (VolVis) via neural radiance field (NeRF). The expressiveness of StyleRF-VolVis is upheld by its ability to accurately separate the underlying scene geometry (i.e., content) and color appearance (i.e., style), conveniently modify color, opacity, and lighting of the original rendering while maintaining visual content consistency across the views, and effectively transfer arbitrary styles from reference images to the reconstructed 3D scene. To achieve these, we design a base NeRF model for scene geometry extraction, a palette color network to classify regions of the radiance field for photorealistic editing, and an unrestricted color network to lift the color palette constraint via knowledge distillation for non-photorealistic editing. We demonstrate the superior quality, consistency, and flexibility of StyleRF-VolVis by experimenting with various volume rendering scenes and reference images and comparing StyleRF-VolVis against other image-based (AdaIN), video-based (ReReVST), and NeRF-based (ARF and SNeRF) style rendering solutions.

Figures (19)

• Figure 1: The workflow of StyleRF-VolVis. (a) Given a collection of multi-view images, we first optimize a base NeRF model for accurate density representation. (b) Then, we train a PCN for PSE, allowing color, opacity, and lighting changes. (c) Next, we utilize KD to optimize a UCN with no color palette constraint. Optimized with the stylization loss, the trained UCN can produce NPSE results.
• Figure 1: Applying various opacity or lighting PSEs to the stylized NPSE scenes.
• Figure 2: The PCN architecture. The final predicted color is obtained by summing the view-dependent specular color output from the lighting MLP and the view-independent diffuse color output from the palette MLP. After training, palette weights $\omega$ are used for classifying points within the RF.
• Figure 2: Comparison of PCN rendering results under different $\lambda_\delta$ on the combustion dataset.
• Figure 3: Classification and rendering results without and with palette refinement. Top: classification results for each palette color. Bottom: rendering results when considering all palette colors.