Versatile User Identification in Extended Reality using Pretrained Similarity-Learning

TL;DR

This article developed a similarity-learning model and pretrained it on the "Who Is Alyx?" dataset and pretrained it on a dedicated set of users for model validation and final evaluation, showing superior performance.

Abstract

Various machine learning approaches have proven to be useful for user verification and identification based on motion data in eXtended Reality (XR). However, their real-world application still faces significant challenges concerning versatility, i.e., in terms of extensibility and generalization capability. This article presents a solution that is both extensible to new users without expensive retraining, and that generalizes well across different sessions, devices, and user tasks. To this end, we developed a similarity-learning model and pretrained it on the "Who Is Alyx?" dataset. This dataset features a wide array of tasks and hence motions from users playing the VR game "Half-Life: Alyx". In contrast to previous works, we used a dedicated set of users for model validation and final evaluation. Furthermore, we extended this evaluation using an independent dataset that features completely different users, tasks, and three different XR devices. In comparison with a traditional classification-learning baseline, our model shows superior performance, especially in scenarios with limited enrollment data. The pretraining process allows immediate deployment in a diverse range of XR applications while maintaining high versatility. Looking ahead, our approach paves the way for easy integration of pretrained motion-based identification models in production XR systems.

Figures (6)

• Figure 1: Who is the real Marc? This Social VR scene taken from Lin et al. Lin2023 demonstrates how a versatile identification model can be used to reveal impostors and, hence, foster trust in SocialVR applications: two users claim to be 'Marc', but, as the identification system indicates, only one is genuine.
• Figure 2: Data split of the 63 users for the similarity-learning (SL) and classification-learning (CL) models; the SL model was pretrained and validated with 36 users; the remaining 27 users were used for evaluation of SL and CL.
• Figure 3: Results for our experiments for the proposed similarity-learning method, as well as the baseline classification-learning model. We also visualize the difference between both models to highlight performance gaps.
• Figure 4: Detailed view of the 5-minute columns from \ref{['fig:dml_model_sequence_accuracy']}; the enrollment procedure is repeated 5 times for each model and duration, each repetition denoted with a circle, stars denote the means; shaded areas indicate 95% confidence intervals (bootstrapped).
• Figure 5: Training of the similarity-learning model with fewer subjects in training data set; reported is $Acc_{\text{enr}: all}^{\text{use}: 1}$ on the test dataset (with all 27 test subjects in each instance).