Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Emolysis: A Multimodal Open-Source Group Emotion Analysis and Visualization Toolkit

Shreya Ghosh, Zhixi Cai, Parul Gupta, Garima Sharma, Abhinav Dhall, Munawar Hayat, Tom Gedeon

TL;DR

Emolysis addresses the need for interactive, multimodal group emotion analysis by mapping video input to per-person emotion, valence, and arousal in real time. The toolkit integrates visual, audio, and linguistic streams (EfficientNet-based visual with MTCNN; TRILL audio; Whisper+RoBERTa linguistic) and exposes a web-based GUI with OS-independent deployment. It supports English and Mandarin, segment-wise processing, and a flexible backend (PyTorch + FastAPI) designed for easy extension and cross-dataset labeling via mapping across AffectNet, CMU-MOSEI, and ARBEE. The authors demonstrate qualitative generalizability on a separate video set and emphasize privacy-preserving runtime inference with consent. Emolysis has potential to enable empathetic design in HCI applications such as lectures, crowd management, and event analysis through an accessible, open-source platform.

Abstract

Automatic group emotion recognition plays an important role in understanding complex human-human interaction. This paper introduces, Emolysis, a Python-based, standalone open-source group emotion analysis toolkit for use in different social situations upon getting consent from the users. Given any input video, Emolysis processes synchronized multimodal input and maps it to group level emotion, valence and arousal. Additionally, the toolkit supports major mobile and desktop platforms (Android, iOS, Windows). The Emolysis platform also comes with an intuitive graphical user interface that allows users to select different modalities and target persons for more fine-grained emotion analysis. Emolysis is freely available for academic research and encourages application developers to extend it to application specific environments on top of the existing system. We believe that the extension mechanism is quite straightforward. Our code models and interface are available at https://github.com/ControlNet/emolysis.

Emolysis: A Multimodal Open-Source Group Emotion Analysis and Visualization Toolkit

TL;DR

Emolysis addresses the need for interactive, multimodal group emotion analysis by mapping video input to per-person emotion, valence, and arousal in real time. The toolkit integrates visual, audio, and linguistic streams (EfficientNet-based visual with MTCNN; TRILL audio; Whisper+RoBERTa linguistic) and exposes a web-based GUI with OS-independent deployment. It supports English and Mandarin, segment-wise processing, and a flexible backend (PyTorch + FastAPI) designed for easy extension and cross-dataset labeling via mapping across AffectNet, CMU-MOSEI, and ARBEE. The authors demonstrate qualitative generalizability on a separate video set and emphasize privacy-preserving runtime inference with consent. Emolysis has potential to enable empathetic design in HCI applications such as lectures, crowd management, and event analysis through an accessible, open-source platform.

Abstract

Automatic group emotion recognition plays an important role in understanding complex human-human interaction. This paper introduces, Emolysis, a Python-based, standalone open-source group emotion analysis toolkit for use in different social situations upon getting consent from the users. Given any input video, Emolysis processes synchronized multimodal input and maps it to group level emotion, valence and arousal. Additionally, the toolkit supports major mobile and desktop platforms (Android, iOS, Windows). The Emolysis platform also comes with an intuitive graphical user interface that allows users to select different modalities and target persons for more fine-grained emotion analysis. Emolysis is freely available for academic research and encourages application developers to extend it to application specific environments on top of the existing system. We believe that the extension mechanism is quite straightforward. Our code models and interface are available at https://github.com/ControlNet/emolysis.
Paper Structure (4 sections, 2 figures)

This paper contains 4 sections, 2 figures.

Table of Contents

  1. Introduction and Background
  2. Emolysis: Software Description
  3. Performance
  4. Conclusion

Figures (2)

  • Figure 1: Emolysis Overview. Emolysis is an interactive, multimodal group emotion analysis and visualization toolkit. The screenshot represents an example scenario from a video feed where the user can select a subset of people as well as the modality used.
  • Figure 2: Emolysis Workflow. The workflow of the Emolysis pipeline. Emolysis tool allows the user to interact with the interface in terms of modality as well as subset of person selection.