Automated UX Insights from User Research Videos by Integrating Facial Emotion and Text Sentiment

TL;DR

This work presents a proof-of-concept for automated UX insight extraction from user research videos by integrating facial emotion recognition, speech-to-text, and text-based sentiment analysis. It combines a ViT-based FER pipeline (selected from 10 pretrained models via cross-dataset evaluation), OpenAI Whisper for ASR, and a GoEmotions-based text classifier, with temporal alignment and fusion to synthesize multimodal signals. A case study demonstrates end-to-end processing, including data output (emotion_probabilities.csv, speech_probabilities.csv), and GPT-4o-driven insight generation through PandasAI, yielding time-stamped, actionable UX insights and anomaly detection. The study highlights the practical potential of multimodal emotion analytics for UX evaluation, while addressing ethical considerations and outlining directions for real-time deployment, speaker-aware analysis, and broader domain applications.

Abstract

Emotion recognition technology has been studied from the past decade. With its growing importance and applications such as customer service, medical, education, etc., this research study aims to explore its potential and importance in the field of User experience evaluation. Recognizing and keeping track of user emotions in user research video is important to understand user needs and expectations from a service/product. Little research has been done that focuses on automating emotion extraction from a video where more than one modality has been incorporated in the field of UX. The study aims at implementing different modalities such as facial emotion recognition, speech-to-text and text-based emotion recognition for capturing emotional nuances from a user research video and extract meaningful actionable insights. For selection of facial emotion recognition model, 10 pre-trained models were evaluated on three benchmark datasets i.e. FER-2013, AffectNet and CK+, selecting the model with most generalization ability. To extract speech and convert to text, OpenAI's Whisper model was implemented and finally the emotions from text were recognized using a pre-trained model available at HuggingFace website having an evaluation accuracy more than 95%. The study also integrates the gathered data using temporal alignment and fusion for deeper and contextual insights. The study further demonstrates a way of automating data analysis through PandasAI Python library where OpenAI's GPT-4o model was implemented along with a discussion on other possible solutions. This study is an attempt to demonstrate a proof of concept where automated meaningful insights are extracted from a video based on user emotions.

Figures (6)

• Figure 1: Project architecture.
• Figure 2: FER model workflow
• Figure 3: Overall framework of STT and Text-based Emotion detection.
• Figure 4: Normalized values of all emotions detected at each frame in Testing_video1
• Figure 5: Emotions detected from participant's speech over time from Testing/_video1