Text-based Transfer Function Design for Semantic Volume Rendering

TL;DR

This work proposes a novel approach that leverages recent advancements in language-vision models to bridge the semantic gap between transfer function values and features of interest within the volume, and generates transfer functions that yield volume-rendered images closely matching the user’s intent.

Abstract

Transfer function design is crucial in volume rendering, as it directly influences the visual representation and interpretation of volumetric data. However, creating effective transfer functions that align with users' visual objectives is often challenging due to the complex parameter space and the semantic gap between transfer function values and features of interest within the volume. In this work, we propose a novel approach that leverages recent advancements in language-vision models to bridge this semantic gap. By employing a fully differentiable rendering pipeline and an image-based loss function guided by language descriptions, our method generates transfer functions that yield volume-rendered images closely matching the user's intent. We demonstrate the effectiveness of our approach in creating meaningful transfer functions from simple descriptions, empowering users to intuitively express their desired visual outcomes with minimal effort. This advancement streamlines the transfer function design process and makes volume rendering more accessible to a wider range of users.

Figures (7)

• Figure 1: We show an overview of T2TF. Given a TF, we volume render from a camera view to produce an image and subsequently obtain a CLIP-based encoding of the image. We then compare the image with a CLIP-based encoding of the user's prompt, expressive of their visualization objective, with negative sampled prompts. We form a contrastive loss between the user's prompt and negative prompts, giving us a way to update the TF via gradient-based optimization.
• Figure 2: T2TF results for a variety of volumes (scanned data, numerical simulations) and prompts (general vs. detailed descriptions).
• Figure 3: Volume-rendered images of the visible male dataset. In (a), the generic prompt results in the transfer functions that show an overview of features in the volume. Then, in (b)-(d), different opacity transfer functions highlight the skin and skull. This example illustrates the capability of T2TF to adjust opacity transfer functions
• Figure 4: Transfer functions and images comparing the T2TF with contrastive loss and maximizing CLIP score. With contrastive loss, the resulting transfer function can produce images closer to desired prompts.
• Figure 5: We observe that Beta prior consistently gives a clear volume rendering with correct opacity and fewer visual artifacts. Optimizing without a density prior often leads to noisy ambience and high transparency. Density prior from Dream Field or Entropy prior yields more transparent volume or fails to optimize.