Motion as Emotion: Detecting Affect and Cognitive Load from Free-Hand Gestures in VR

TL;DR

It is found that the affect and cognitive load induced by tasks are associated with significant differences in gesture features such as speed, distance and hand tension.

Abstract

Affect and cognitive load influence many user behaviors. In this paper, we propose Motion as Emotion, a novel method that utilizes fine differences in hand motion to recognise affect and cognitive load in virtual reality (VR). We conducted a study with 22 participants who used common free-hand gesture interactions to carry out tasks of varying difficulty in VR environments. We find that the affect and cognitive load induced by tasks are associated with significant differences in gesture features such as speed, distance and hand tension. Standard support vector classification (SVC) models could accurately predict two levels (low, high) of valence, arousal and cognitive load from these features. Our results demonstrate the potential of Motion as Emotion as an accurate and reliable method of inferring user affect and cognitive load from free-hand gestures, without needing any additional wearable sensors or modifications to a standard VR headset.

Figures (5)

• Figure 1: View of the virtual environment for each of the task conditions. Each task involves common interactions in VR, such as selection, swiping and realistic interactions with objects in the environment.
• Figure 2: Effect of task condition on induced arousal (left) and valence (right). Arousal is significantly higher in the challenging condition for all except the Slingshot task; Valence is significantly lower in the challenging condition for all except the Slingshot task.
• Figure 3: Effect of task condition on induced mental workload in the easy condition (gray) and challenging condition (blue).
• Figure 4: Illustration of detected hand keypoints during task performance. The RGB video and keypoints were used to manually characterize hand movements during the primary phases of gesture formation, including preparation (hands moving away from resting positions, typically on the lap), stroke (the movements used to indicate commands) and retraction (hands moving back towards resting positions).
• Figure 5: Illustrations of frequently observed versions of the same gesture: relaxed (left), and tense (right). Notably, gestures that look tense tend to be characterized by increased extension of the fingers.