Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Automated Testing of Prevalent 3D User Interactions in Virtual Reality Applications

Ruizhen Gu, José Miguel Rojas, Donghwan Shin

TL;DR

The paper tackles the challenge of automated testing for prevalent 3D VR interactions by introducing the Interaction Flow Graph (IFG) to model complex, multi-step interactions and guide test generation. It builds XRBench3D, a 10-scene VR benchmark with 456 interactions, and XRintTest, a model-based tester that statically constructs the IFG and dynamically explores scenes to activate interactions. Evaluation shows XRintTest achieves about 93% interaction flow coverage (fire, manipulate, and socket) and outperforms random exploration by roughly 12x in effectiveness and 6x in efficiency, while also detecting runtime and non-exception interaction issues and revealing design smells. The work offers a principled framework for VR interaction testing with practical implications for improving robustness and design quality in XR applications, and it outlines directions to extend this approach with learning-based techniques and broader XR modalities.

Abstract

Virtual Reality (VR) technologies offer immersive user experiences across various domains, but present unique testing challenges compared to traditional software. Existing VR testing approaches enable scene navigation and interaction activation, but lack the ability to automatically synthesise realistic 3D user inputs (e.g, grab and trigger actions via hand-held controllers). Automated testing that generates and executes such input remains an unresolved challenge. Furthermore, existing metrics fail to robustly capture diverse interaction coverage. This paper addresses these gaps through four key contributions. First, we empirically identify four prevalent interaction types in nine open-source VR projects: fire, manipulate, socket, and custom. Second, we introduce the Interaction Flow Graph, a novel abstraction that systematically models 3D user interactions by identifying targets, actions, and conditions. Third, we construct XRBench3D, a benchmark comprising ten VR scenes that encompass 456 distinct user interactions for evaluating VR interaction testing. Finally, we present XRintTest, an automated testing approach that leverages this graph for dynamic scene exploration and interaction execution. Evaluation on XRBench3D shows that XRintTest achieves great effectiveness, reaching 93% coverage of fire, manipulate and socket interactions across all scenes, and performing 12x more effectively and 6x more efficiently than random exploration. Moreover, XRintTest can detect runtime exceptions and non-exception interaction issues, including subtle configuration defects. In addition, the Interaction Flow Graph can reveal potential interaction design smells that may compromise intended functionality and hinder testing performance for VR applications.

Automated Testing of Prevalent 3D User Interactions in Virtual Reality Applications

TL;DR

The paper tackles the challenge of automated testing for prevalent 3D VR interactions by introducing the Interaction Flow Graph (IFG) to model complex, multi-step interactions and guide test generation. It builds XRBench3D, a 10-scene VR benchmark with 456 interactions, and XRintTest, a model-based tester that statically constructs the IFG and dynamically explores scenes to activate interactions. Evaluation shows XRintTest achieves about 93% interaction flow coverage (fire, manipulate, and socket) and outperforms random exploration by roughly 12x in effectiveness and 6x in efficiency, while also detecting runtime and non-exception interaction issues and revealing design smells. The work offers a principled framework for VR interaction testing with practical implications for improving robustness and design quality in XR applications, and it outlines directions to extend this approach with learning-based techniques and broader XR modalities.

Abstract

Virtual Reality (VR) technologies offer immersive user experiences across various domains, but present unique testing challenges compared to traditional software. Existing VR testing approaches enable scene navigation and interaction activation, but lack the ability to automatically synthesise realistic 3D user inputs (e.g, grab and trigger actions via hand-held controllers). Automated testing that generates and executes such input remains an unresolved challenge. Furthermore, existing metrics fail to robustly capture diverse interaction coverage. This paper addresses these gaps through four key contributions. First, we empirically identify four prevalent interaction types in nine open-source VR projects: fire, manipulate, socket, and custom. Second, we introduce the Interaction Flow Graph, a novel abstraction that systematically models 3D user interactions by identifying targets, actions, and conditions. Third, we construct XRBench3D, a benchmark comprising ten VR scenes that encompass 456 distinct user interactions for evaluating VR interaction testing. Finally, we present XRintTest, an automated testing approach that leverages this graph for dynamic scene exploration and interaction execution. Evaluation on XRBench3D shows that XRintTest achieves great effectiveness, reaching 93% coverage of fire, manipulate and socket interactions across all scenes, and performing 12x more effectively and 6x more efficiently than random exploration. Moreover, XRintTest can detect runtime exceptions and non-exception interaction issues, including subtle configuration defects. In addition, the Interaction Flow Graph can reveal potential interaction design smells that may compromise intended functionality and hinder testing performance for VR applications.
Paper Structure (44 sections, 6 figures, 2 tables)

This paper contains 44 sections, 6 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. VR Development with Unity
  4. XR Interaction Toolkit
  5. 3D Interaction
  6. GameObject Layers and Interaction Layers
  7. XR Scene Testing
  8. Model-based Testing
  9. Motivating Example
  10. Empirical Analysis of Prevalent VR Interactions
  11. Data Collection
  12. Results
  13. Discussions
  14. Unresponsive Interaction Issues
  15. Socket Interactions
  16. ...and 29 more sections

Figures (6)

  • Figure 1: Meta Quest Touch Pro Controller (Right)
  • Figure 2: Hierarchical Structure and Interaction Flows in VR Scene Modelling (Actions: $trigger_{inst}$/$grab_{inst}$ denote instant actions; $trigger_{const}$/$grab_{const}$ denote continuous actions; $mov_{fwd}$ indicates forward controller movement; $rot_{rt}$ indicate rightward controller rotation.)
  • Figure 3: Permutations of Socket Interactions (an example from Scene 10)
  • Figure 4: VR User Interaction Testing with XRintTest
  • Figure 5: Intersection of Colliders of the Target Interactable (Green Bounds) and Controller (Red Bounds)
  • ...and 1 more figures