SIM2VR: Towards Automated Biomechanical Testing in VR

TL;DR

SIM2VR integrates biomechanical user simulations directly into the Unity VR application, closing the loop between the simulated user and the target VR environment to produce ecologically valid, forward-looking predictions. By extending UitB with a Unity-specific asset, OpenXR input/output alignment, and a synchronized data exchange, SIM2VR enables training muscle-actuated models inside the exact app humans use, rather than relying on re-implemented replicas. The paper demonstrates this approach on a fast-paced Whac-A-Mole variant and a bimanual VR Beats Kit, showing that the simulated users can predict differences in performance, effort, and strategies comparable to human data and can reveal design-induced behavioral changes. While achieving this alignment substantially improves predictive validity, the authors acknowledge limits related to perception, cognition, reward design, and sensory feedback, and call for continued methodological advances and community involvement to broaden applicability across VR tasks. Overall, SIM2VR offers a practical pathway toward automated biomechanical testing in VR, potentially accelerating design iteration and reducing reliance on costly user studies.

Abstract

Automated biomechanical testing has great potential for the development of VR applications, as initial insights into user behaviour can be gained in silico early in the design process. In particular, it allows prediction of user movements and ergonomic variables, such as fatigue, prior to conducting user studies. However, there is a fundamental disconnect between simulators hosting state-of-the-art biomechanical user models and simulators used to develop and run VR applications. Existing user simulators often struggle to capture the intricacies of real-world VR applications, reducing ecological validity of user predictions. In this paper, we introduce SIM2VR, a system that aligns user simulation with a given VR application by establishing a continuous closed loop between the two processes. This, for the first time, enables training simulated users directly in the same VR application that real users interact with. We demonstrate that SIM2VR can predict differences in user performance, ergonomics and strategies in a fast-paced, dynamic arcade game. In order to expand the scope of automated biomechanical testing beyond simple visuomotor tasks, advances in cognitive models and reward function design will be needed.

Figures (9)

• Figure 1: To obtain valid and reliable predictions of interactive user behaviour in VR, a simulated user needs to perceive and control exactly the same VR application as real users. Our sim2vr system addresses this by establishing a closed loop between the simulation process and the desired VR application, ensuring that the same input and output signals are generated as with real hardware. The resulting time-continuous integration allows to train and evaluate a biomechanical user simulation directly in a given VR environment.
• Figure 2: With sim2vr, simulated users have access to exactly the same visual output as humans. The top figure depicts how real users perceive a VR environment (in this case, the Whac-A-Mole game from Section \ref{['sec:whac-a-mole']}), while the bottom figure shows the downsampled RGB-D image that the simulated user perceives.
• Figure 3: sim2vr enables efficient data transfer between the simulated user (implemented in UitB) and the VR application (implemented in Unity). This is achieved by establishing a ZeroMQ connection between the generic UnityEnv task module and the SIM2VR Asset, both provided by sim2vr. The UnityEnv equips the biomechanical model with VR controllers and an HMD. Visual perception of the VR application is received through the UnityHeadset vision module, based on the rendered HMD images sent to the UnityEnv. The SIM2VR Asset also allows application-dependent rewards, such as game scores, to be defined directly in the VR application using the RLEnv Unity script. The rewards can be used by the simulated user to improve their control policy while interacting with the VR application.
• Figure 4: a) In the VR Beats game, the player controls two lightsabers and has to cut through boxes that are moving towards them. b) Still frames from evaluations of trained simulated users playing the VR Beats game, about to hit an incoming box. Top: A simulated user trained with a high effort cost weight ($0.05$) has learned to keep their arms down and rely on wrist movements. Bottom: A simulated user trained with a low effort cost weight ($0.001$) moves their arms considerably more, sometimes resulting in implausible postures.
• Figure 5: (a) In Whac-A-Mole, targets appear randomly at one of nine fixed positions and must be hit with a hammer within one second to score a point. The transparent targets are shown for visualisation purposes only and are not visible during the game. (b) The game allows for three different placements of the target area.