VRSplat: Fast and Robust Gaussian Splatting for Virtual Reality

TL;DR

VRSplat targets VR-specific weaknesses of 3D Gaussian Splatting by unifying Mini-Splatting with StopThePop and Optimal Projection, complemented by a single-pass foveated rasterizer and a fine-tuning step. The method eliminates popping and projection-induced distortions while maintaining real-time VR framerates (≥72 FPS) on consumer hardware, validated through a 25-participant user study and VR-focused image metrics. The results demonstrate robust, artifact-free rendering at high resolution and wide FOV, with substantial performance gains realized via single-pass foveation and efficient tiling. This work enables practical, high-quality real-time VR using Gaussian splats, offering a scalable path for immersive experiences on head-mounted displays.

Abstract

3D Gaussian Splatting (3DGS) has rapidly become a leading technique for novel-view synthesis, providing exceptional performance through efficient software-based GPU rasterization. Its versatility enables real-time applications, including on mobile and lower-powered devices. However, 3DGS faces key challenges in virtual reality (VR): (1) temporal artifacts, such as popping during head movements, (2) projection-based distortions that result in disturbing and view-inconsistent floaters, and (3) reduced framerates when rendering large numbers of Gaussians, falling below the critical threshold for VR. Compared to desktop environments, these issues are drastically amplified by large field-of-view, constant head movements, and high resolution of head-mounted displays (HMDs). In this work, we introduce VRSplat: we combine and extend several recent advancements in 3DGS to address challenges of VR holistically. We show how the ideas of Mini-Splatting, StopThePop, and Optimal Projection can complement each other, by modifying the individual techniques and core 3DGS rasterizer. Additionally, we propose an efficient foveated rasterizer that handles focus and peripheral areas in a single GPU launch, avoiding redundant computations and improving GPU utilization. Our method also incorporates a fine-tuning step that optimizes Gaussian parameters based on StopThePop depth evaluations and Optimal Projection. We validate our method through a controlled user study with 25 participants, showing a strong preference for VRSplat over other configurations of Mini-Splatting. VRSplat is the first, systematically evaluated 3DGS approach capable of supporting modern VR applications, achieving 72+ FPS while eliminating popping and stereo-disrupting floaters.

Figures (11)

• Figure 1: Normal FOV (top) and VR (bottom) views of Mini-Splatting fang2024minisplatting and ours. Increasing the field-of-view reveals shortcomings of current methods in VR applications: large Gaussians when projected with the local affine approximation can cause cloud-like artifacts and elongated Gaussians (i.e., "god rays"). VRSplat eliminates the artifacts while meeting the performance demands of smooth VR experiences.
• Figure 2: An overview of our pipeline for high-fidelity VR rendering: Starting from an efficient baseline, such as Mini-Splatting fang2024minisplatting, we fine-tune the Gaussians using our artifact-free differentiable rasterizer to arrive at a 3DGS model capable of VR rendering without artifacts. We thus remain largely baseline-agnostic, and the overhead for our fine-tuning is only a small fraction of the initial baseline reconstruction.
• Figure 3: Left: StopThePop computes the maximum contribution within a tile directly in screen space. Right: As we employ Optimal Projection huang2024optimal, we find the maximum by reasoning about the maximum on the per-Gaussian optimal plane, leveraging the fact that tiles form general quadrilaterals. Subsequently, projecting this maximum back to screen space yields the maximum contribution within a tile anon2024fastandrobust.
• Figure 4: We design a single-pass foveated rendering routine with visibility culling. (a) Depiction of our chosen tile configurations for our single-pass foveated rendering: we split the center region into tiles of size $16{\times}16$, and use $32{\times}32$ tiles in the periphery; hybrid tiles in-between (dotted) are treated as $16{\times}16$ tiles but the values of groups of $2{\times}2$ pixels are averaged and blended with the individual pixel values, based on the continuous blending mask (left). (b) We sample the visibility mask from OpenXR to first compute a mask of all visible tiles, from which we construct its summed-area table. We then compute the exact number of visible tiles covered by the rectangular extent of a 2D Gaussian splat from the summed-area values of its corner points.
• Figure 5: User Study results: our method is preferred by users across all tested modalities including quality, artifacts and preference. Due to the prevalence of popping artifacts in Mini-Splatting ($z$), users strongly prefer our method. In Mini-Splatting (Dist), projection errors cause the preference for our method.