Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OFERA: Blendshape-driven 3D Gaussian Control for Occluded Facial Expression to Realistic Avatars in VR

Seokhwan Yang, Boram Yoon, Seoyoung Kang, Hail Song, Woontack Woo

TL;DR

OFERA tackles the challenge of expressing VR headset users’ occluded faces by leveraging blendshape signals and a three-part pipeline: Blendshape Distribution Alignment (BDA) to normalize headset inputs, an Expression Parameter Mapper (EPM) to nonlinearly map to FLAME-based expression parameters, and a Mapper-integrated Avatar (MiA) to train avatars that honor the EPM output distribution. The end-to-end system enables real-time, photorealistic Gaussian head avatars in VR with a pipeline that senses, maps, updates, and renders at around 20 ms latency. Quantitative results show EPM outperforms baseline mappings in parameter and vertex accuracy, while a user study demonstrates enhanced embodiment, plausibility, and realism of expressions under occlusion. The work advances telepresence in VR by providing a practical, privacy-preserving, blendshape-driven pathway to expressive avatar rendering in immersive environments.

Abstract

We propose OFERA, a novel framework for real-time expression control of photorealistic Gaussian head avatars for VR headset users. Existing approaches attempt to recover occluded facial expressions using additional sensors or internal cameras, but sensor-based methods increase device weight and discomfort, while camera-based methods raise privacy concerns and suffer from limited access to raw data. To overcome these limitations, we leverage the blendshape signals provided by commercial VR headsets as expression inputs. Our framework consists of three key components: (1) Blendshape Distribution Alignment (BDA), which applies linear regression to align the headset-provided blendshape distribution to a canonical input space; (2) an Expression Parameter Mapper (EPM) that maps the aligned blendshape signals into an expression parameter space for controlling Gaussian head avatars; and (3) a Mapper-integrated Avatar (MiA) that incorporates EPM into the avatar learning process to ensure distributional consistency. Furthermore, OFERA establishes an end-to-end pipeline that senses and maps expressions, updates Gaussian avatars, and renders them in real-time within VR environments. We show that EPM outperforms existing mapping methods on quantitative metrics, and we demonstrate through a user study that the full OFERA framework enhances expression fidelity while preserving avatar realism. By enabling real-time and photorealistic avatar expression control, OFERA significantly improves telepresence in VR communication. A project page is available at https://ysshwan147.github.io/projects/ofera/.

OFERA: Blendshape-driven 3D Gaussian Control for Occluded Facial Expression to Realistic Avatars in VR

TL;DR

OFERA tackles the challenge of expressing VR headset users’ occluded faces by leveraging blendshape signals and a three-part pipeline: Blendshape Distribution Alignment (BDA) to normalize headset inputs, an Expression Parameter Mapper (EPM) to nonlinearly map to FLAME-based expression parameters, and a Mapper-integrated Avatar (MiA) to train avatars that honor the EPM output distribution. The end-to-end system enables real-time, photorealistic Gaussian head avatars in VR with a pipeline that senses, maps, updates, and renders at around 20 ms latency. Quantitative results show EPM outperforms baseline mappings in parameter and vertex accuracy, while a user study demonstrates enhanced embodiment, plausibility, and realism of expressions under occlusion. The work advances telepresence in VR by providing a practical, privacy-preserving, blendshape-driven pathway to expressive avatar rendering in immersive environments.

Abstract

We propose OFERA, a novel framework for real-time expression control of photorealistic Gaussian head avatars for VR headset users. Existing approaches attempt to recover occluded facial expressions using additional sensors or internal cameras, but sensor-based methods increase device weight and discomfort, while camera-based methods raise privacy concerns and suffer from limited access to raw data. To overcome these limitations, we leverage the blendshape signals provided by commercial VR headsets as expression inputs. Our framework consists of three key components: (1) Blendshape Distribution Alignment (BDA), which applies linear regression to align the headset-provided blendshape distribution to a canonical input space; (2) an Expression Parameter Mapper (EPM) that maps the aligned blendshape signals into an expression parameter space for controlling Gaussian head avatars; and (3) a Mapper-integrated Avatar (MiA) that incorporates EPM into the avatar learning process to ensure distributional consistency. Furthermore, OFERA establishes an end-to-end pipeline that senses and maps expressions, updates Gaussian avatars, and renders them in real-time within VR environments. We show that EPM outperforms existing mapping methods on quantitative metrics, and we demonstrate through a user study that the full OFERA framework enhances expression fidelity while preserving avatar realism. By enabling real-time and photorealistic avatar expression control, OFERA significantly improves telepresence in VR communication. A project page is available at https://ysshwan147.github.io/projects/ofera/.
Paper Structure (19 sections, 7 equations, 5 figures, 1 table)

This paper contains 19 sections, 7 equations, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Avatars in VR Communication
  4. Head Avatar Reconstruction
  5. Facial Expression Capture with VR Headsets
  6. Method
  7. Blendshape Distribution Alignment (BDA)
  8. Expression Parameter Mapper (EPM)
  9. Mapper-integrated Avatar (MiA)
  10. End-to-End Real-Time Gaussian Avatar System in VR
  11. Experiment
  12. Experimental Setup
  13. User Study
  14. Results
  15. Expression Mapping Evaluation
  16. ...and 4 more sections

Figures (5)

  • Figure 1: Overview of OFERA, which operates in online and offline phases. (a) In the online phase, headset blendshapes are processed through BDA, EPM, and MiA to drive a photorealistic 3D Gaussian avatar in real-time. (b) The offline phase involves fitting BDA (\ref{['sec_bda']}) and training EPM (\ref{['sec_epm']}) and MiA (\ref{['sec_mia']}) to ensure accurate expression mapping and consistent avatar rendering across headsets.
  • Figure 2: Architecture of the proposed EPM model. The network takes blendshape parameters ($BS_{train}$) as input and produces predicted target parameters ($\hat{Q}_{train}$) as output during training. It consists of an initial fully connected (FC) layer, followed by four residual blocks (each with two FC layers and an intermediate dropout layer), and a final FC layer. All FC layers are followed by batch normalization and ReLU activation, except for the last output FC layer. Each hidden FC layer has 128 units.
  • Figure 3: Overview of the two-day user study for facial data collection and VR-based evaluation (Day 1 and Day 2). (a) Facial expression recording for data collection and preparation; (b) VR setup for the evaluation session; and (c) an example study task in which participants controlled avatar facial expressions in VR using visual references. Avatar control and post-task questionnaires were repeated for each experimental condition (×5).
  • Figure 4: Qualitative comparison of vertex-wise reconstruction errors visualized as heatmaps. Matrix-based and linear mapping baselines show large errors around expressive regions such as the mouth and eyes, whereas our EPM significantly reduces these localized errors, leading to more faithful reproduction of facial expressions.
  • Figure 5: Results for (a)--(a-2) Virtual Embodiment Questionnaire and subscales; (b)--(b-2) VEQ+ and the subscales associated with Self-Identification; (c) Virtual Human Plausibility Questionnaire; (d) Facial Animation Realism.