egoEMOTION: Egocentric Vision and Physiological Signals for Emotion and Personality Recognition in Real-World Tasks

TL;DR

The paper addresses the gap in egocentric vision benchmarks by incorporating internal affect and personality. It introduces egoEMOTION, the first multimodal dataset combining egocentric visual data with physiological signals and dense self-reports across induced and naturalistic tasks. It defines three benchmarks for continuous affect, discrete emotion, and personality inference, and shows that head-mounted eye-tracking and motion signals can outperform traditional physiological sensors in real-world emotion prediction. The work highlights potential applications in affect-aware human-computer interaction and emphasizes ethical data sharing and reproducibility for future research.

Abstract

Understanding affect is central to anticipating human behavior, yet current egocentric vision benchmarks largely ignore the person's emotional states that shape their decisions and actions. Existing tasks in egocentric perception focus on physical activities, hand-object interactions, and attention modeling - assuming neutral affect and uniform personality. This limits the ability of vision systems to capture key internal drivers of behavior. In this paper, we present egoEMOTION, the first dataset that couples egocentric visual and physiological signals with dense self-reports of emotion and personality across controlled and real-world scenarios. Our dataset includes over 50 hours of recordings from 43 participants, captured using Meta's Project Aria glasses. Each session provides synchronized eye-tracking video, headmounted photoplethysmography, inertial motion data, and physiological baselines for reference. Participants completed emotion-elicitation tasks and naturalistic activities while self-reporting their affective state using the Circumplex Model and Mikels' Wheel as well as their personality via the Big Five model. We define three benchmark tasks: (1) continuous affect classification (valence, arousal, dominance); (2) discrete emotion classification; and (3) trait-level personality inference. We show that a classical learning-based method, as a simple baseline in real-world affect prediction, produces better estimates from signals captured on egocentric vision systems than processing physiological signals. Our dataset establishes emotion and personality as core dimensions in egocentric perception and opens new directions in affect-driven modeling of behavior, intent, and interaction.

Figures (15)

• Figure 1: egoEMOTION is a multimodal emotion and personality recognition dataset that captures participants' facial, eye-tracking, egocentric, and physiological signals during induced video stimuli and naturalistic real-world activities. Participants reported their emotions via emoti-SAM emoti-sam and a weighted Mikels' Wheel mikelswheel, and their personality using the Big Five model costa_bigfive.
• Figure 2: Data collection from the egocentric glasses and physiological sensors during each task with their associated self-reports. Further information about the study protocol is available in Appendix \ref{['app:study_protocol']}.
• Figure 3: Participant self-reports across tasks. (a) Mean arousal-valence ratings. (b) Proportions of discrete emotions reported in Sessions A and B. (c) Boxplots of Big Five personality trait scores.
• Figure 4: Pearson correlations between self-reports.(a) continuous self-ratings and personality scores (b) discrete emotions and personality scores (c) discrete emotions and continuous self-ratings.
• Figure 5: Overview of the experimental protocol. The experiment consisted of two sessions. In session A, participants watched 9 video clips, with a 40 s washout between clips and a 5 s video of a cross preceding each clip. In session B, participants performed 7 real-world tasks. Each task was spaced by a 1-min washout clip. Two questionnaires, corresponding to the emoti-SAM emoti-sam and a weighted Mikels' Wheel mikelswheel were answered after each emotional stimulus.