VR-GS: A Physical Dynamics-Aware Interactive Gaussian Splatting System in Virtual Reality

TL;DR

VR-GS presents a physics-aware interactive Gaussian Splatting system for real-time VR content manipulation. By coupling explicit Gaussian primitives with an XPBD-based deformation cage and a novel two-level embedding, it obtains realistic dynamics while maintaining rendering efficiency. The framework includes segmentation, 2D inpainting-guided 3D inpainting, and dynamic shadow casting to enhance realism, with VR demonstrations and a user study showing improved immersion for physics-based interactions. This work advances interactive, physically plausible 3D content creation in VR and XR pipelines, enabling more intuitive editing and manipulation of complex scenes.

Abstract

As consumer Virtual Reality (VR) and Mixed Reality (MR) technologies gain momentum, there's a growing focus on the development of engagements with 3D virtual content. Unfortunately, traditional techniques for content creation, editing, and interaction within these virtual spaces are fraught with difficulties. They tend to be not only engineering-intensive but also require extensive expertise, which adds to the frustration and inefficiency in virtual object manipulation. Our proposed VR-GS system represents a leap forward in human-centered 3D content interaction, offering a seamless and intuitive user experience. By developing a physical dynamics-aware interactive Gaussian Splatting in a Virtual Reality setting, and constructing a highly efficient two-level embedding strategy alongside deformable body simulations, VR-GS ensures real-time execution with highly realistic dynamic responses. The components of our Virtual Reality system are designed for high efficiency and effectiveness, starting from detailed scene reconstruction and object segmentation, advancing through multi-view image in-painting, and extending to interactive physics-based editing. The system also incorporates real-time deformation embedding and dynamic shadow casting, ensuring a comprehensive and engaging virtual experience.Our project page is available at: https://yingjiang96.github.io/VR-GS/.

Figures (13)

• Figure 1: VR-GS is an interactive system designed to integrate 3D gs and pbd for generating a real-time interactive experience. Beginning with multi-view images, the pipeline combines scene reconstruction, segmentation, and inpainting using Gaussian kernels. These kernels form the foundation for VR-GS's utilization of the sparse volumetric data structure VDB, facilitating bounding mesh reconstruction and subsequent tetrahedralization. VR-GS further harnesses a novel two-level Gaussian embedding, pbd, collision detection, and shadow casting techniques, all converging to deliver a captivating and immersive user experience.
• Figure 2: Our two-level embedding effectively resolves spiky artifacts. Each Gaussian kernel is embedded into a local tetrahedron. The vertices of the local tetrahedron are independently embedded into the global mesh.
• Figure 3: Two-level Embedding Evaluation. Our two-level embedding alleviates the commonly seen spiky artifacts for deformed GS kernels.
• Figure 4: Inpainting Evaluation.gs struggles to reconstruct occluded surfaces. By leveraging LAMA suvorov2022resolution, we produce 2D inpainted results that guide the 3D scene inpainting, enhancing the realism of the 3D representation.
• Figure 5: Shadow Map Evaluation. gs traditionally models shadows as static surface textures. Our approach, employing a shadow map, generates dynamic shadows for a more immersive experience.