Emotion Detection through Body Gesture and Face
Haoyang Liu
TL;DR
Emotion Detection through Body Gesture and Face addresses the limitation of facial-only emotion recognition by leveraging body gestures via OpenPose on Aff-Wild2 and DFEW to classify seven basic emotions and estimate valence/arousal. The approach uses a bifurcated architecture: CNN backbones (ResNet/DenseNet) for discrete emotion classification and feed-forward networks for regression of valence/arousal, enabling multimodal analysis. The study demonstrates that CNN-based models generally outperform ANN for both classification and regression, with DenseNet/DenseNet121 yielding strong performance; however, keypoint reliability and data redundancy present challenges. The work contributes to affective computing by integrating pose-based features with deep learning to improve emotion understanding in real-world settings, with potential applications in mental health, education, and autonomous systems.
Abstract
The project leverages advanced machine and deep learning techniques to address the challenge of emotion recognition by focusing on non-facial cues, specifically hands, body gestures, and gestures. Traditional emotion recognition systems mainly rely on facial expression analysis and often ignore the rich emotional information conveyed through body language. To bridge this gap, this method leverages the Aff-Wild2 and DFEW databases to train and evaluate a model capable of recognizing seven basic emotions (angry, disgust, fear, happiness, sadness, surprise, and neutral) and estimating valence and continuous scales wakeup descriptor. Leverage OpenPose for pose estimation to extract detailed body posture and posture features from images and videos. These features serve as input to state-of-the-art neural network architectures, including ResNet, and ANN for emotion classification, and fully connected layers for valence arousal regression analysis. This bifurcation strategy can solve classification and regression problems in the field of emotion recognition. The project aims to contribute to the field of affective computing by enhancing the ability of machines to interpret and respond to human emotions in a more comprehensive and nuanced way. By integrating multimodal data and cutting-edge computational models, I aspire to develop a system that not only enriches human-computer interaction but also has potential applications in areas as diverse as mental health support, educational technology, and autonomous vehicle systems.