Task Breakpoint Generation using Origin-Centric Graph in Virtual Reality Recordings for Adaptive Playback

TL;DR

The proposed task segmentation method provides a foundation for dynamically adjusting VR playback according to user proficiency and progress, with potential for extension into automatic timeline segmentation systems for diverse VR recordings.

Abstract

We propose a method for generating task breakpoints based on an Origin-Centric Graph (OCG) to segment goal-oriented activity recordings into task units for adaptive playback in Virtual Reality (VR) environments. With the development of Augmented Reality (AR)/VR head-mounted displays (HMDs), research on adaptive tutorials and authoring tools has become active, but existing task segmentation methods mainly rely on manual annotation or are restricted to 2D video which limits their applicability to 3D VR contexts. In our approach, assembly scenarios with clearly defined task boundaries are recorded using a structured spatio-temporal scene graph (STSG), and the OCG is employed to track changes in the central object and the formation of new groups, thereby generating task breakpoints automatically. A user study collected user-perceived task breakpoints to establish ground truth (GT), and comparison with the algorithm-detected breakpoints demonstrated high agreement and confirmed accuracy in supporting adaptive playback. The proposed task segmentation method provides a foundation for dynamically adjusting VR playback according to user proficiency and progress, with potential for extension into automatic timeline segmentation systems for diverse VR recordings.

Figures (4)

• Figure 1: An illustration of constructing an Origin-centric Graph by identifying the origin object from the fully assembled STSG. In (B), “O” denotes the origin object.
• Figure 2: Examples of task breakpoints. (A) Parts connect to the origin object; (B) the central object is updated; (C) a new group begins. Green boxes indicate connections identified as breakpoints. The dark blue circle labeled “O” represents the origin object, the blue circle labeled “C” indicates the central object, and light blue circles represent general objects. Pink solid lines show parts assembled at the current moment, while purple solid lines represent parts assembled just before the current moment.
• Figure 3: Visualization of the GT breakpoints, derived from user-identified breakpoints in drone and bicycle assembly, alongside the breakpoints identified by our algorithm, for both fine and coarse units.
• Figure 4: An illustration of the tasks subdivided into fine and coarse units using breakpoints identified by our algorithm. Dashed borders indicate screenshots of specific moments in tasks at the fine-level, while solid borders indicate screenshots of specific moments in tasks at the coarse-level (which also encompass smaller units).