VRGaussianAvatar: Integrating 3D Gaussian Avatars into VR

TL;DR

VRGaussianAvatar tackles the challenge of real-time, photorealistic full-body VR avatars controllable with only HMD signals. It combines a VR Frontend that infers full-body pose from head/hand tracking with a GA Backend that renders a single-image–reconstructed 3D Gaussian avatar, using Binocular Batching to render left and right eyes in one batched pass. The system demonstrates improved perceived appearance similarity, embodiment, and plausibility over image- and video-based mesh baselines in a within-subject study, and achieves real-time performance at high resolutions. This work shows that 3D Gaussian Splatting can be effectively integrated into practical VR pipelines, enabling immersive, socially capable avatars without external trackers.

Abstract

We present VRGaussianAvatar, an integrated system that enables real-time full-body 3D Gaussian Splatting (3DGS) avatars in virtual reality using only head-mounted display (HMD) tracking signals. The system adopts a parallel pipeline with a VR Frontend and a GA Backend. The VR Frontend uses inverse kinematics to estimate full-body pose and streams the resulting pose along with stereo camera parameters to the backend. The GA Backend stereoscopically renders a 3DGS avatar reconstructed from a single image. To improve stereo rendering efficiency, we introduce Binocular Batching, which jointly processes left and right eye views in a single batched pass to reduce redundant computation and support high-resolution VR displays. We evaluate VRGaussianAvatar with quantitative performance tests and a within-subject user study against image- and video-based mesh avatar baselines. Results show that VRGaussianAvatar sustains interactive VR performance and yields higher perceived appearance similarity, embodiment, and plausibility. Project page and source code are available at https://vrgaussianavatar.github.io.

Figures (7)

• Figure 1: System diagram of the proposed method, VRGaussianAvatar. Avatar reconstruction phase (left): Given a single input image, the Gaussian Avatar Module reconstructs a 3D full-body avatar. Runtime process (right): The VR Frontend and GA Backend operate in parallel to animate and render the avatar in real time. The VR Frontend estimates a SMPL-X–compatible full-body pose from head 6DoF and hand-tracking signals using the IK Module. The GA Backend animates and renders the reconstructed Gaussian avatar, and applies the proposed Binocular Batching with both world-to-camera matrices ($\textbf{T}^C_{W,L}, \textbf{T}^C_{W,R}$) to enable efficient real-time stereoscopic rendering.
• Figure 2: Baseline conditions. (A) video-based reconstruction: Implemented following prior works rcsmplzhang2025fatesong2025fasttexturetransferxr. (B) image-based reconstruction: Implemented following prior works liu2024texdreamercai2023smpler. (C) Runtime process: The IK Module estimates a full-body pose from head 6DoF and hand-tracking signals to animate the avatar in real time.
• Figure 3: (A) User study setup with participants performing poses from pre-recorded voice instructions. (B) Avatar type factor with two levels: self (left, resembling the participant) and other (right, resembling another person).
• Figure 4: Qualitative comparison of VRGaussianAvatar with baseline avatar representations under identical HMD-only control. Compared to baselines, VRGaussianAvatar typically preserves sharper appearance details in close-up regions, including head/face (sky-blue boxes) and clothing (orange boxes). All methods are rendered stereoscopically in real time, enabling interactive avatar control in VR.
• Figure 5: Results for (A)--(A-3) Virtual Embodiment Questionnaire and its subscales; (B)--(B-3) VEQ+ and its subscales. (A and M: significant main effect of avatar type and method, respectively; I: significant interaction effect between avatar type and method)