Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Deep Networks for Detecting User Familiarity with VR from Hand Interactions

Mingjun Li, Numan Zafar, Natasha Kholgade Banerjee, Sean Banerjee

TL;DR

This work tackles the problem of detecting a user's familiarity with VR from hand movements during a door-unlock task to enable on-demand training. It compares three deep architectures (MLP, FCN, PCT) trained on sliding windows of dominant-hand trajectories collected from 14 participants performing multiple 4-digit codes on a VR keypad. The best result reaches 88.03% accuracy (code 2648, window size 120, using PCT), with varying performance across codes and architectures, highlighting the influence of acclimatization and window length. The findings demonstrate feasibility of using VR movement data for familiarity detection and point to future work on more diverse tasks, multi-hand interactions, and on-demand training strategies to ease novice VR users into complex environments.

Abstract

As VR devices become more prevalent in the consumer space, VR applications are likely to be increasingly used by users unfamiliar with VR. Detecting the familiarity level of a user with VR as an interaction medium provides the potential of providing on-demand training for acclimatization and prevents the user from being burdened by the VR environment in accomplishing their tasks. In this work, we present preliminary results of using deep classifiers to conduct automatic detection of familiarity with VR by using hand tracking of the user as they interact with a numeric passcode entry panel to unlock a VR door. We use a VR door as we envision it to the first point of entry to collaborative virtual spaces, such as meeting rooms, offices, or clinics. Users who are unfamiliar with VR will have used their hands to open doors with passcode entry panels in the real world. Thus, while the user may not be familiar with VR, they would be familiar with the task of opening the door. Using a pilot dataset consisting of 7 users familiar with VR, and 7 not familiar with VR, we acquire highest accuracy of 88.03\% when 6 test users, 3 familiar and 3 not familiar, are evaluated with classifiers trained using data from the remaining 8 users. Our results indicate potential for using user movement data to detect familiarity for the simple yet important task of secure passcode-based access.

Evaluating Deep Networks for Detecting User Familiarity with VR from Hand Interactions

TL;DR

This work tackles the problem of detecting a user's familiarity with VR from hand movements during a door-unlock task to enable on-demand training. It compares three deep architectures (MLP, FCN, PCT) trained on sliding windows of dominant-hand trajectories collected from 14 participants performing multiple 4-digit codes on a VR keypad. The best result reaches 88.03% accuracy (code 2648, window size 120, using PCT), with varying performance across codes and architectures, highlighting the influence of acclimatization and window length. The findings demonstrate feasibility of using VR movement data for familiarity detection and point to future work on more diverse tasks, multi-hand interactions, and on-demand training strategies to ease novice VR users into complex environments.

Abstract

As VR devices become more prevalent in the consumer space, VR applications are likely to be increasingly used by users unfamiliar with VR. Detecting the familiarity level of a user with VR as an interaction medium provides the potential of providing on-demand training for acclimatization and prevents the user from being burdened by the VR environment in accomplishing their tasks. In this work, we present preliminary results of using deep classifiers to conduct automatic detection of familiarity with VR by using hand tracking of the user as they interact with a numeric passcode entry panel to unlock a VR door. We use a VR door as we envision it to the first point of entry to collaborative virtual spaces, such as meeting rooms, offices, or clinics. Users who are unfamiliar with VR will have used their hands to open doors with passcode entry panels in the real world. Thus, while the user may not be familiar with VR, they would be familiar with the task of opening the door. Using a pilot dataset consisting of 7 users familiar with VR, and 7 not familiar with VR, we acquire highest accuracy of 88.03\% when 6 test users, 3 familiar and 3 not familiar, are evaluated with classifiers trained using data from the remaining 8 users. Our results indicate potential for using user movement data to detect familiarity for the simple yet important task of secure passcode-based access.
Paper Structure (11 sections, 1 equation, 3 figures, 2 tables)

This paper contains 11 sections, 1 equation, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. VR Door Unlock Application
  3. Dataset
  4. Experiments
  5. Multi-Layer Perceptron
  6. Fully Convolutional Network
  7. Point Cloud Transformer
  8. Training and Test Split
  9. Loss Function
  10. Results
  11. Conclusion

Figures (3)

  • Figure 1: Left: Close-up view of numeric panel in VR Door Unlock application. The panel is placed on a VR door. Right: Participant interacts with the panel by entering a numeric passcode to unlock the door.
  • Figure 2: 10-session trajectories from each of the 4 passcode combinations used in this work from a participant without VR familiarity (top) and with VR familiarity (bottom). We observe that the user with no prior familiarity with VR show more variability in their interaction profile when compared to the user with prior familiarity.
  • Figure 3: ROC curves for the classifiers and number combinations analyzed.