Interactive Holographic Visualization for 3D Facial Avatar

TL;DR

The paper addresses the need for realistic, shareable 3D visualizations for pain-assessment training by proposing a real-time holographic avatar pipeline. It combines 3D Gaussian Splatting with light-field shading to render a FLAME-based facial avatar on a multi-view Looking Glass display, guided by a Transformer-based generative model that fuses multi-modal cues to produce interactive feedback. Key contributions include a proof-of-concept pipeline linking multimodal input to holographic rendering, a fast 3D Gaussian Splatting avatar with a FLAME prior, and a calibrated multi-view display pipeline enabling group interaction. The results demonstrate improvements in generated non-verbal facial responsiveness across criteria of appropriateness, diversity, realism, and synchrony, indicating potential for scalable, headset-free medical training.

Abstract

Traditional methods for visualizing dynamic human expressions, particularly in medical training, often rely on flat-screen displays or static mannequins, which have proven inefficient for realistic simulation. In response, we propose a platform that leverages a 3D interactive facial avatar capable of displaying non-verbal feedback, including pain signals. This avatar is projected onto a stereoscopic, view-dependent 3D display, offering a more immersive and realistic simulated patient experience for pain assessment practice. However, there is no existing solution that dynamically predicts and projects interactive 3D facial avatars in real-time. To overcome this, we emphasize the need for a 3D display projection system that can project the facial avatar holographically, allowing users to interact with the avatar from any viewpoint. By incorporating 3D Gaussian Splatting (3DGS) and real-time view-dependent calibration, we significantly improve the training environment for accurate pain recognition and assessment.

Figures (3)

• Figure 1: 3D Holographic Avatar Projection Overview. A novel pipeline combining 3D Gaussian Splatting and light-field shading enables real-time, photo-realistic facial avatar hologram animations, streamed to the 3D Looking Glass display. The OpenGL-based light-field shading program uses FLAME expression parameters with its 3D Gaussian primitive correspondence and the display calibration parameters to render a multi-view quilt that adapts to the audience's viewpoint.
• Figure 2: Holographic rendering pipeline overview. Our system consists of a blend-shape model for the 3D Gaussian Splatting Avatar with a FLAME prior. Given an expression parameter $\psi_m$ at frame $\#m$, the pipeline linearly blends pre-trained blend shapes ${B_K}$ into an accumulated $B^*_m$ primitive parameters. An iterative procedure rasterizes this 3D facial avatar from various viewpoints to reproduce a light field given an input viewpoint. The final output is a shaded quilt comprised by multiple discrete views that visualizes the expected 3D hologram on the target 3D display.
• Figure 3: 3D FLAME output features could be used to reconstruct 3D facial mesh model that works as prior for photorealistic 3D Gaussian Splatting facial avatar.