Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Gems: Group Emotion Profiling Through Multimodal Situational Understanding

Anubhav Kataria, Surbhi Madan, Shreya Ghosh, Tom Gedeon, Abhinav Dhall

TL;DR

This work tackles the challenge of understanding emotions across individuals, groups, and events in multi-person videos. It introduces VGAF-GEMS, a densely annotated benchmark, and the GEMS framework that combines a Swin-transformer–based individual emotion encoder with multimodal contextual descriptions from an MLLM, fused through S3Attention to yield predictions at the individual, group, and situational levels, including valence and arousal. The dataset comprises $4{,}183$ videos ($2{,}661$ train, $766$ val, $756$ test) with per-frame emotion annotations and event-level labels, enabling robust evaluation of emotional compositionality over time. Results show zero-shot video-LLM performance is weak, but S3Attention substantially improves multitask predictions, underscoring the importance of temporal-contextual fusion for holistic group emotion profiling and motivating further research with this challenging benchmark.

Abstract

Understanding individual, group and event level emotions along with contextual information is crucial for analyzing a multi-person social situation. To achieve this, we frame emotion comprehension as the task of predicting fine-grained individual emotion to coarse grained group and event level emotion. We introduce GEMS that leverages a multimodal swin-transformer and S3Attention based architecture, which processes an input scene, group members, and context information to generate joint predictions. Existing multi-person emotion related benchmarks mainly focus on atomic interactions primarily based on emotion perception over time and group level. To this end, we extend and propose VGAF-GEMS to provide more fine grained and holistic analysis on top of existing group level annotation of VGAF dataset. GEMS aims to predict basic discrete and continuous emotions (including valence and arousal) as well as individual, group and event level perceived emotions. Our benchmarking effort links individual, group and situational emotional responses holistically. The quantitative and qualitative comparisons with adapted state-of-the-art models demonstrate the effectiveness of GEMS framework on VGAF-GEMS benchmarking. We believe that it will pave the way of further research. The code and data is available at: https://github.com/katariaak579/GEMS

Gems: Group Emotion Profiling Through Multimodal Situational Understanding

TL;DR

This work tackles the challenge of understanding emotions across individuals, groups, and events in multi-person videos. It introduces VGAF-GEMS, a densely annotated benchmark, and the GEMS framework that combines a Swin-transformer–based individual emotion encoder with multimodal contextual descriptions from an MLLM, fused through S3Attention to yield predictions at the individual, group, and situational levels, including valence and arousal. The dataset comprises videos ( train, val, test) with per-frame emotion annotations and event-level labels, enabling robust evaluation of emotional compositionality over time. Results show zero-shot video-LLM performance is weak, but S3Attention substantially improves multitask predictions, underscoring the importance of temporal-contextual fusion for holistic group emotion profiling and motivating further research with this challenging benchmark.

Abstract

Understanding individual, group and event level emotions along with contextual information is crucial for analyzing a multi-person social situation. To achieve this, we frame emotion comprehension as the task of predicting fine-grained individual emotion to coarse grained group and event level emotion. We introduce GEMS that leverages a multimodal swin-transformer and S3Attention based architecture, which processes an input scene, group members, and context information to generate joint predictions. Existing multi-person emotion related benchmarks mainly focus on atomic interactions primarily based on emotion perception over time and group level. To this end, we extend and propose VGAF-GEMS to provide more fine grained and holistic analysis on top of existing group level annotation of VGAF dataset. GEMS aims to predict basic discrete and continuous emotions (including valence and arousal) as well as individual, group and event level perceived emotions. Our benchmarking effort links individual, group and situational emotional responses holistically. The quantitative and qualitative comparisons with adapted state-of-the-art models demonstrate the effectiveness of GEMS framework on VGAF-GEMS benchmarking. We believe that it will pave the way of further research. The code and data is available at: https://github.com/katariaak579/GEMS

Paper Structure

This paper contains 6 sections, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. VGAF-GEMS
  4. Benchmark and Metrics
  5. Results and Discussion
  6. Conclusion

Figures (4)

  • Figure 1: GEMS. A brief overview of GEMS framework. Given any of the defined situations (eg: Celebration-party, Fighting), the framework aims to link participant's individual emotion, group emotion, situational emotion.
  • Figure 2: Data statistics for the VGAF-GEMS dataset. The figure presents the class-wise distribution of individual emotions, group emotions, and situational contexts in the proposed dataset. Note, Quar: Quarrelling, CasGat: Casual Gathering, GrpAct: Group Activities and CelPar:Celebration Party. The y-axis indicates the number of samples corresponding to each class.
  • Figure 3: GEMS Overview. Given a video input, GEMS parse individual level emotion embedding via emotion encoder, and contextual embedding via MLLM. contextual and individual level information is fused by S3 Attention mechanism to get group level and situation level information.
  • Figure 4: Qualitative analysis of the GEMS framework. Here, GT: Ground Truth and Predicted: Predicted from GEMS.