GS-Verse: Mesh-based Gaussian Splatting for Physics-aware Interaction in Virtual Reality

TL;DR

GS-Verse introduces a mesh-based Gaussian Splatting framework that tightly couples a surface mesh with Gaussian primitives to enable physics-aware interaction in VR. By replacing proxy cages with geometry-aligned splats anchored to mesh faces and integrating with standard physics engines, GS-Verse achieves more realistic deformations and interactions, while remaining engine-agnostic and content-reusable. A two-task user study with 18 participants across three scenes demonstrates that GS-Verse significantly improves naturalness in stretching and provides consistent performance across twisting and shaking, with high usability and low workload. The approach combines a mesh-driven GS representation, segmentation, and mesh-to-GS parameterization to deliver a unified, efficient, and scalable VR editing and interaction toolkit with strong practical impact for immersive content creation.

Abstract

As the demand for immersive 3D content grows, the need for intuitive and efficient interaction methods becomes paramount. Current techniques for physically manipulating 3D content within Virtual Reality (VR) often face significant limitations, including reliance on engineering-intensive processes and simplified geometric representations, such as tetrahedral cages, which can compromise visual fidelity and physical accuracy. In this paper, we introduce GS-Verse (Gaussian Splatting for Virtual Environment Rendering and Scene Editing), a novel method designed to overcome these challenges by directly integrating an object's mesh with a Gaussian Splatting (GS) representation. Our approach enables more precise surface approximation, leading to highly realistic deformations and interactions. By leveraging existing 3D mesh assets, GS-Verse facilitates seamless content reuse and simplifies the development workflow. Moreover, our system is designed to be physics-engine-agnostic, granting developers robust deployment flexibility. This versatile architecture delivers a highly realistic, adaptable, and intuitive approach to interactive 3D manipulation. We rigorously validate our method against the current state-of-the-art technique that couples VR with GS in a comparative user study involving 18 participants. Specifically, we demonstrate that our approach is statistically significantly better for physics-aware stretching manipulation and is also more consistent in other physics-based manipulations like twisting and shaking. Further evaluation across various interactions and scenes confirms that our method consistently delivers high and reliable performance, showing its potential as a plausible alternative to existing methods.

Figures (11)

• Figure 1: The three scenes used in our study: (top) dark room, (middle) toy room, and (bottom) garden as seen from VR. The three columns on the right present a handful from a large range of possible physics-aware 3D object manipulations (e.g., moving, swinging, stretching, pulling, twisting, shaking, crushing, tipping).
• Figure 2: Overview of our proposed method, GS-Verse. The approach enables real-time interaction in VR by generating mesh-based Gaussian Splatting assets. The processing pipeline begins with multiview image scene reconstruction, followed by mesh extraction using SuGaR or Trellis. A subsequent segmentation step optimizes the scene by dividing it into dynamic and static components. Thanks to mesh-based parameterization via GaMeS, the resulting representations can be seamlessly integrated into physics-aware engines such as Unity, enabling efficient and physically consistent VR interactions.
• Figure 3: Visual example of a participant performing the requested manipulations during the first, closed-ended task: stretch, twist, shake in VR-GS (first row) and GS-Verse method (second row).
• Figure 4: Artifacts during maximum object stretching from the user perspective. In VR-GS (top row), overstretching caused very large splats that overlapped the object. In addition, users were able to detach individual splats, making the interaction unsuccessful. In contrast, our GS-Verse (bottom row) remained robust even under extreme stretching, showing minimal artifacts.
• Figure 5: Comparison of VR-GS and GS-Verse across three visualization stages: the simulation mesh (left), the original render (middle), and the render inside VR (right). Unlike VR-GS, which relies on a cage mesh, GS-Verse employs a more geometry-accurate simulation mesh.