VRScout: Towards Real-Time, Autonomous Testing of Virtual Reality Games

TL;DR

VRScout tackles scalable automated testing for VR games by learning from human demonstrations with an Action Chunking Transformer, enabling multi-step action sequences and dynamic horizon adaptation to balance reactivity and precision. The approach achieves data-efficient learning and real-time inference (60 FPS) on consumer GPUs across diverse titles, including Beat Saber and Pistol Whip, with as little as 3.5 hours of demonstrations. It demonstrates expert-level performance in fast-paced VR contexts and discusses extensions to multimodal sensing and hybrid learning to broaden applicability. The work offers a practical, open-source framework for automated VR quality assurance and safety auditing.

Abstract

Virtual Reality (VR) has rapidly become a mainstream platform for gaming and interactive experiences, yet ensuring the quality, safety, and appropriateness of VR content remains a pressing challenge. Traditional human-based quality assurance is labor-intensive and cannot scale with the industry's rapid growth. While automated testing has been applied to traditional 2D and 3D games, extending it to VR introduces unique difficulties due to high-dimensional sensory inputs and strict real-time performance requirements. We present VRScout, a deep learning-based agent capable of autonomously navigating VR environments and interacting with virtual objects in a human-like and real-time manner. VRScout learns from human demonstrations using an enhanced Action Chunking Transformer that predicts multi-step action sequences. This enables our agent to capture higher-level strategies and generalize across diverse environments. To balance responsiveness and precision, we introduce a dynamically adjustable sliding horizon that adapts the agent's temporal context at runtime. We evaluate VRScout on commercial VR titles and show that it achieves expert-level performance with only limited training data, while maintaining real-time inference at 60 FPS on consumer-grade hardware. These results position VRScout as a practical and scalable framework for automated VR game testing, with direct applications in both quality assurance and safety auditing.

Figures (4)

• Figure 1: Architecture of the VR agent.
• Figure 2: Temporal Ensembling with Dynamic Sliding Window
• Figure 3: Training curve plot (L1 loss vs. iterations).
• Figure 4: Screenshots for each VR title: Beat Saber, Pistol Whip, SuperHot VR.