Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hybrid Foveated Path Tracing with Peripheral Gaussians for Immersive Anatomy

Constantin Kleinbeck, Luisa Theelke, Hannah Schieber, Ulrich Eck, Rüdiger von Eisenhart-Rothe, Daniel Roth

TL;DR

This paper addresses the challenge of interactive, high-fidelity immersive visualization of volumetric medical data in VR. It introduces a hybrid rendering pipeline that streams foveated path tracing for the central region while rapidly regenerating a peripheral Gaussian Splatting model, with depth-guided reprojection to mask latency. Key contributions include an initialization approach that yields a ~10k-Gaussian peripheral model in under a second, continual refinement driven by foveal renders, and a reprojection mechanism that stabilizes integration of path-traced content with the peripheral cloud. Evaluations demonstrate improved foveal detail over real-time path tracing and competitive peripheral fidelity relative to precomputed Gaussians, enabling interactive tasks (e.g., transfer function changes) with minimal preprocessing. The approach holds practical significance for accessible, high-quality immersive medical visualization on consumer VR hardware by balancing quality, speed, and interactivity.

Abstract

Volumetric medical imaging offers great potential for understanding complex pathologies. Yet, traditional 2D slices provide little support for interpreting spatial relationships, forcing users to mentally reconstruct anatomy into three dimensions. Direct volumetric path tracing and VR rendering can improve perception but are computationally expensive, while precomputed representations, like Gaussian Splatting, require planning ahead. Both approaches limit interactive use. We propose a hybrid rendering approach for high-quality, interactive, and immersive anatomical visualization. Our method combines streamed foveated path tracing with a lightweight Gaussian Splatting approximation of the periphery. The peripheral model generation is optimized with volume data and continuously refined using foveal renderings, enabling interactive updates. Depth-guided reprojection further improves robustness to latency and allows users to balance fidelity with refresh rate. We compare our method against direct path tracing and Gaussian Splatting. Our results highlight how their combination can preserve strengths in visual quality while re-generating the peripheral model in under a second, eliminating extensive preprocessing and approximations. This opens new options for interactive medical visualization.

Hybrid Foveated Path Tracing with Peripheral Gaussians for Immersive Anatomy

TL;DR

This paper addresses the challenge of interactive, high-fidelity immersive visualization of volumetric medical data in VR. It introduces a hybrid rendering pipeline that streams foveated path tracing for the central region while rapidly regenerating a peripheral Gaussian Splatting model, with depth-guided reprojection to mask latency. Key contributions include an initialization approach that yields a ~10k-Gaussian peripheral model in under a second, continual refinement driven by foveal renders, and a reprojection mechanism that stabilizes integration of path-traced content with the peripheral cloud. Evaluations demonstrate improved foveal detail over real-time path tracing and competitive peripheral fidelity relative to precomputed Gaussians, enabling interactive tasks (e.g., transfer function changes) with minimal preprocessing. The approach holds practical significance for accessible, high-quality immersive medical visualization on consumer VR hardware by balancing quality, speed, and interactivity.

Abstract

Volumetric medical imaging offers great potential for understanding complex pathologies. Yet, traditional 2D slices provide little support for interpreting spatial relationships, forcing users to mentally reconstruct anatomy into three dimensions. Direct volumetric path tracing and VR rendering can improve perception but are computationally expensive, while precomputed representations, like Gaussian Splatting, require planning ahead. Both approaches limit interactive use. We propose a hybrid rendering approach for high-quality, interactive, and immersive anatomical visualization. Our method combines streamed foveated path tracing with a lightweight Gaussian Splatting approximation of the periphery. The peripheral model generation is optimized with volume data and continuously refined using foveal renderings, enabling interactive updates. Depth-guided reprojection further improves robustness to latency and allows users to balance fidelity with refresh rate. We compare our method against direct path tracing and Gaussian Splatting. Our results highlight how their combination can preserve strengths in visual quality while re-generating the peripheral model in under a second, eliminating extensive preprocessing and approximations. This opens new options for interactive medical visualization.
Paper Structure (22 sections, 2 equations, 5 figures, 3 tables)

This paper contains 22 sections, 2 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Volumetric Path Tracing
  4. Novel View Synthesis
  5. Immersive Rendering
  6. Research Gap
  7. System Design
  8. Method
  9. Volumetric Path Tracing
  10. Periphery Model Training
  11. Real-time Viewer
  12. Interconnections
  13. Evaluation
  14. Datasets
  15. Peripheral Model
  16. ...and 7 more sections

Figures (5)

  • Figure 1: System design showing the three main components and the two application flows between them. The initial model construction is always executed before interactive viewing, which uses and improves the initial model with the same images originally rendered for the real-time viewer.
  • Figure 2: Composition process of the foveated and peripheral region, as performed in the real-time viewer. A shows peripheral Gaussians, B how Gaussians are discarded in a downscaled frustum shape to enable correct transparency in the rendered views. C shows how the foveal view is composited at depth while being blended at the edges of the foveal view. Steps B and C show slightly angled views, with the drawn frustum highlighting the original camera view. In C, gaussians are downscaled for visual clarity in this image.
  • Figure 3: Visual overview of peripheral model initialization progress. Shown are two scenes, initialized from 12 images with 8 spp. We further include the quality after continual training with 512 additional views with 4 spp each.
  • Figure 4: Comparison of multiple initialization numbers of views and samples per pixel. The time on the x-axis includes the time to render the views and generate the initial point cloud. The render end time is indicated by the vertical lines on the x-axis. This means that every plotted point represents the visual quality that users would see at that point in time in the viewer, without transmission and loading overhead. Each line contains average and maximum p10/p90 values from different datasets and random views, representing typical and best/worst quality bands.
  • Figure 5: Visual comparison of the resulting quality and baselines (Ground Truth - GT). The upper row shows high-quality settings, as in \ref{['tab:quality-comparison']}, the lower row shows lower quality. Our approach retains more detail compared to gs and path tracing, and suffers fewer artifacts.