The AffectToolbox: Affect Analysis for Everyone

TL;DR

The paper addresses the lack of accessible, end-to-end affective computing tools that do not require programming expertise. It introduces the AffectToolbox, a GUI-based, open-source platform with modular pipelines for face, pose, voice, and sentiment analysis, plus an event-driven fusion that outputs continuous PAD scores in real time. Key contributions include ready-to-use modality models, a queue-based architecture for easy extensibility, and broadcasting capabilities to integrate with other applications. The work aims to democratize affective computing for research and HCI by lowering technical barriers and enabling rapid prototyping across disciplines.

Abstract

In the field of affective computing, where research continually advances at a rapid pace, the demand for user-friendly tools has become increasingly apparent. In this paper, we present the AffectToolbox, a novel software system that aims to support researchers in developing affect-sensitive studies and prototypes. The proposed system addresses the challenges posed by existing frameworks, which often require profound programming knowledge and cater primarily to power-users or skilled developers. Aiming to facilitate ease of use, the AffectToolbox requires no programming knowledge and offers its functionality to reliably analyze the affective state of users through an accessible graphical user interface. The architecture encompasses a variety of models for emotion recognition on multiple affective channels and modalities, as well as an elaborate fusion system to merge multi-modal assessments into a unified result. The entire system is open-sourced and will be publicly available to ensure easy integration into more complex applications through a well-structured, Python-based code base - therefore marking a substantial contribution toward advancing affective computing research and fostering a more collaborative and inclusive environment within this interdisciplinary field.

Figures (6)

• Figure 1: The modular architecture of the AffectToolbox (Section \ref{['sec:system']}). In its current state, audiovisual sensory devices (e.g. webcams) provide easy means to generate all considered data streams (i.e. audio, transcript, video and skeleton data). So called activity checks trigger the machine learning based analysis of respective modalities (Section \ref{['sec:modalities']}). The uni-modal results of applied affect recognition models are represented by a subset of pleasure, arousal and/or dominance scores. These unimodal emotional cues are the input for an event-driven fusion algorithm (Section \ref{['sec:fusion']}), which deduces a coherent affective state, represented in the continuous PAD emotional space (Section \ref{['sec:emo']}).
• Figure 2: User interacting with a virtual agent, whose emotional behaviour is taking real-time input from the AffectToolbox into consideration.
• Figure 3: The three dimensional PAD model of affect. Continuous pleasure, arousal and dominance scores describe affective states, from which discrete emotion labels can be derived.
• Figure 4: Based on empirical studies, a dominance level can be derived from facial expressions described in the valence/arousal space or inferred emotional labels.
• Figure 5: Audio signal of a person's normal speech leading into an affective burst of laughter. The characteristics of the signal parts can be well differentiated. Further processing of the signal carves out detailed differences and enables the categorization of more subtle affective states.