EAST: Environment-Aware Stylized Transition Along the Reality-Virtuality Continuum

TL;DR

The paper addresses the disruption of VR immersion during real-world interruptions by introducing EAST, a framework that blends interruptions into stylized VR using style-transferred 3D Gaussian Splatting and a differentiable rasterizer. By leveraging AdaIN-based style transfer on a 3D Gaussian scene representation, EAST maintains aesthetic coherence across the reality-virtuality continuum and supports real-time rendering on VR HMDs. Quantitative results show superior structural similarity, perceptual fidelity, and detail preservation compared with baselines, while a user study demonstrates enhanced cognitive comfort and immersion with stylized transitions. A PICO 4 Pro case study confirms practical viability, including hardware-aware optimizations that keep frame rates high and latency low.

Abstract

In the Virtual Reality (VR) gaming industry, maintaining immersion during real-world interruptions remains a challenge, particularly during transitions along the reality-virtuality continuum (RVC). Existing methods tend to rely on digital replicas or simple visual transitions, neglecting to address the aesthetic discontinuities between real and virtual environments, especially in highly stylized VR games. This paper introduces the Environment-Aware Stylized Transition (EAST) framework, which employs a novel style-transferred 3D Gaussian Splatting (3DGS) technique to transfer real-world interruptions into the virtual environment with seamless aesthetic consistency. Rather than merely transforming the real world into game-like visuals, EAST minimizes the disruptive impact of interruptions by integrating real-world elements within the framework. Qualitative user studies demonstrate significant enhancements in cognitive comfort and emotional continuity during transitions, while quantitative experiments highlight EAST's ability to maintain visual coherence across diverse VR styles.

Figures (5)

• Figure 1: The implem ented environment-aware stylized transition with seamless aesthetic consistency. From left to right: participants' first-person perspective of the the game scene; a real-world scenario with an interruption; the transition from VR to the real world with aesthetic consistency.
• Figure 2: Overview of our environment-aware stylized transition (EAST) approach. Real-world interruptions are seamlessly integrated into VR by stylizing objects to match the game’s aesthetics, maintaining immersion and minimizing disruption.
• Figure 3: The pipeline of our method for achieving style transfer in the reconstructed 3D environment.
• Figure 4: Sample images from the game-style dataset. These screenshots, collected from various VR and streamable games, represent the diverse styles used in our experiments to evaluate transitions between game aesthetics and real-world environments.
• Figure 5: Demonstration of the style transfer results. Left: images of the real-world environment; Right: game-aware style transfer results based on the game scene. These screenshots, collected from various VR and streamable games, represent the diverse styles used in our experiments to evaluate transitions between game aesthetics and real-world environments.