xTrace: A Facial Expressive Behaviour Analysis Tool for Continuous Affect Recognition
Mani Kumar Tellamekala, Shashank Jaiswal, Thomas Smith, Timur Alamev, Gary McKeown, Anthony Brown, Michel Valstar
TL;DR
xTrace tackles robust, real-time continuous affect recognition from in-the-wild face videos by using interpretable Low-Level Descriptors and AU intensities to predict valence and arousal with uncertainty estimates. The system comprises uncertainty-aware landmark detection, AU intensity estimation, and temporal VA regression, trained on a large \\$\sim\$450k video corpus and benchmarked against leading tools, achieving high CCC scores (e.g., \\$CCC\approx0.86\\$) and SEWA improvements (~7.1%) while maintaining cross-modal robustness to occlusions and non-frontal poses. A key contribution is sampling-free predictive uncertainty (epistemic, aleatoric, cumulative) that informs downstream decisions, alongside an on-device, edge-friendly architecture capable of real-time performance (\\≥30 FPS). The work demonstrates the practicality of continuous VA analysis for real-world applications such as mental health, driver monitoring, and social robotics, and provides a scalable approach to cover broad regions of the 2D emotion space. \\$
Abstract
Recognising expressive behaviours in face videos is a long-standing challenge in Affective Computing. Despite significant advancements in recent years, it still remains a challenge to build a robust and reliable system for naturalistic and in-the-wild facial expressive behaviour analysis in real time. This paper addresses two key challenges in building such a system: (1). The paucity of large-scale labelled facial affect video datasets with extensive coverage of the 2D emotion space, and (2). The difficulty of extracting facial video features that are discriminative, interpretable, robust, and computationally efficient. Toward addressing these challenges, this work introduces xTrace, a robust tool for facial expressive behaviour analysis and predicting continuous values of dimensional emotions, namely valence and arousal, from in-the-wild face videos. To address challenge (1), the proposed affect recognition model is trained on the largest facial affect video data set, containing $\sim$450k videos that cover most emotion zones in the dimensional emotion space, making xTrace highly versatile in analysing a wide spectrum of naturalistic expressive behaviours. To address challenge (2), xTrace uses facial affect descriptors that are not only explainable, but can also achieve a high degree of accuracy and robustness with low computational complexity. The key components of xTrace are benchmarked against three existing tools: MediaPipe, OpenFace, and Augsburg Affect Toolbox. On an in-the-wild benchmarking set composed of $\sim$50k videos, xTrace achieves 0.86 mean Concordance Correlation Coefficient (CCC) and on the SEWA test set it achieves 0.75 mean CCC, outperforming existing SOTA by $\sim$7.1\%.