Robo-Saber: Generating and Simulating Virtual Reality Players

TL;DR

Robo-Saber demonstrates promise in synthesizing rich gameplay data for predictive applications and enabling a physics-based whole-body VR playtesting agent.

Abstract

We present the first motion generation system for playtesting virtual reality (VR) games. Our player model generates VR headset and handheld controller movements from in-game object arrangements, guided by style exemplars and aligned to maximize simulated gameplay score. We train on the large BOXRR-23 dataset and apply our framework on the popular VR game Beat Saber. The resulting model Robo-Saber produces skilled gameplay and captures diverse player behaviors, mirroring the skill levels and movement patterns specified by input style exemplars. Robo-Saber demonstrates promise in synthesizing rich gameplay data for predictive applications and enabling a physics-based whole-body VR playtesting agent.

Figures (14)

• Figure 1: Overview of our generative model architecture, extending Categorical Codebook Matching (CCM, starke2024categorical, \ref{['sec:method']}) with Transformer encoder models $\mathcal{E}^\text{style}$ and $\mathcal{E}^\text{game}$, as well as a modified loss function based on Jensen-Shannon divergence.
• Figure 2: Comparing Robo-Saber's performance to that of human players. Robo-Saber trajectories are produced by using $N_\text{ref}=5$ segments from elite (top 5%) players. Left: Human and Robo-Saber TS score distributions across all held-out maps are shown as raincloud plots. Right: Robo-Saber's performance relative to human players is quantified as score percentiles. To compute the percentiles, for each difficulty level, we select the 100 most-played maps (400 total) and compare Robo-Saber's score against human scores in each map. The number of human scores available in each map is visualized by opaqueness (max=388, min=3) The performance statistics are summarized as mean $\pm$ standard error. The reference-aware model with 5 reference segments from top players ($N_\text{ref}=5$) outperforms humans on average.
• Figure 3: Sampling-based candidate trajectory selection improves performance compared to using deterministic Argmax selection for GS-VAE during inference. The percentiles are evaluated on the same held-out maps as \ref{['fig:robo-vs-humans-scatter']} with $N_\text{ref}=5$ and $\mathbf{x}^{:N_\text{ref}}$ sampled from elite (top 5%) players. The TS score percentiles are summarized as mean $\pm$ standard error for each configuration.
• Figure 4: Quantifying Robo-Saber's ability to produce trajectories consistent with the style reference. The oracle player classifier's top-$k$ accuracy measures how well the generated $3p$ trajectories are recognized. Adding style reference segments clearly improves the recognizability.
• Figure 5: Quantifying Robo-Saber variants' calibration to the skill levels of human players, measured by the Pearson correlation ($r$) between Robo-Saber and human players' performance on held-out maps. Left: Densities of points comparing robot (predicted) scores against ground truth human scores. The reference-aware Robo-Saber with $N_\text{ref}=5$ achieves a strong correlation of $r=0.789$. Right: Distributions of score differences between humans and Robo-Saber variants.