Free Energy-Based Modeling of Emotional Dynamics in Video Advertisements

TL;DR

This work addresses how to quantify viewer emotions in video advertisements without external signals by deploying a free energy principle (FEP) based framework. It uses scene-level expressive features extracted from multimodal content and a hidden Markov model to compute FE components—$D_{\mathrm{KL}}$, $\mathrm{BS}$, and $\mathrm{UN}$—representing pleasantness, surprise, and habituation, respectively, then aggregates them into video-level indices (peak, end, skew, decay) for robust interpretation. Across a large Japanese ad dataset, the approach reveals consistent associations between expression elements and FE metrics, identifies three distinct emotional patterns, and demonstrates robustness to hyperparameter choices with reasonable cross-genre/duration generalizability. The method offers an interpretable, scalable tool for advertising analysis and design, with potential extensions to subjective validation, behavioral outcomes, and AI-generated content evaluation.

Abstract

Emotional responses during advertising video viewing are recognized as essential for understanding media effects because they have influenced attention, memory, and purchase intention. To establish a methodological basis for explainable emotion estimation without relying on external information such as physiological signals or subjective ratings, we have quantified "pleasantness," "surprise," and "habituation" solely from scene-level expression features of advertising videos, drawing on the free energy(FE) principle, which has provided a unified account of perception, learning, and behavior. In this framework, Kullback-Leibler divergence (KLD) has captured prediction error, Bayesian surprise (BS) has captured belief updates, and uncertainty (UN) has reflected prior ambiguity, and together they have formed the core components of FE. Using 1,059 15 s food video advertisements, the experiments have shown that KLD has reflected "pleasantness" associated with brand presentation, BS has captured "surprise" arising from informational complexity, and UN has reflected "surprise" driven by uncertainty in element types and spatial arrangements, as well as by the variability and quantity of presented elements. This study also identified three characteristic emotional patterns, namely uncertain stimulus, sustained high emotion, and momentary peak and decay, demonstrating the usefulness of the proposed method. Robustness across nine hyperparameter settings and generalization tests with six types of Japanese advertising videos (three genres and two durations) confirmed that these tendencies remained stable. This work can be extended by integrating a wider range of expression elements and validating the approach through subjective ratings, ultimately guiding the development of technologies that can support the creation of more engaging advertising videos.

Figures (12)

• Figure 1: Overview of the proposed processing flow.
• Figure 2: Graphical model of the proposed HMM.
• Figure 3: Correlation between FE metrics and advertising features in video scenes.
• Figure 4: Mean FE profiles for the food cluster structure
• Figure 5: Representative example scenes and FE metrics from each cluster.