Exploring Emotions in Multi-componential Space using Interactive VR Games

TL;DR

This work operationalised a data-driven approach using interactive Virtual Reality games and collected multimodal measures (self-reports, physiological and facial signals) from 39 participants to identify the unique contributions of each component to emotion differentiation.

Abstract

Emotion understanding is a complex process that involves multiple components. The ability to recognise emotions not only leads to new context awareness methods but also enhances system interaction's effectiveness by perceiving and expressing emotions. Despite the attention to discrete and dimensional models, neuroscientific evidence supports those emotions as being complex and multi-faceted. One framework that resonated well with such findings is the Component Process Model (CPM), a theory that considers the complexity of emotions with five interconnected components: appraisal, expression, motivation, physiology and feeling. However, the relationship between CPM and discrete emotions has not yet been fully explored. Therefore, to better understand emotions underlying processes, we operationalised a data-driven approach using interactive Virtual Reality (VR) games and collected multimodal measures (self-reports, physiological and facial signals) from 39 participants. We used Machine Learning (ML) methods to identify the unique contributions of each component to emotion differentiation. Our results showed the role of different components in emotion differentiation, with the model including all components demonstrating the most significant contribution. Moreover, we found that at least five dimensions are needed to represent the variation of emotions in our dataset. These findings also have implications for using VR environments in emotion research and highlight the role of physiological signals in emotion recognition within such environments.

Figures (7)

• Figure 1: Simple Component Process Model (CPM) based on RN21.
• Figure 2: A participant playing VR games while wearing wearable devices, providing subjective assessments.
• Figure 3: Participant's average intensity ratings for discrete emotions by VR games. The size of the bubble is proportional to the average emotional intensity. For better representation of the difference across games, the games were sorted based on the average rating of Joy emotion.
• Figure 4: Results of the componential hierarchical clustering of discrete emotion terms. Two distinct clusters of negative (red cluster) and positive (green cluster) emotions with meaningful sub-clusters can be observed.
• Figure 5: The correlation matrix displays the average intensity of three facial EMG expressions ("smile_emg", "frown_emg", "eyebrowraise_emg") in black font and the GEW emotions in blue font. Positive correlations are indicated by a (+) symbol, and a (-) symbol indicates negative correlations. Asterisks indicate the statistical significance of the results at a p-value of: *p<0.05, **p<0.01, ***p<0.001.