Encoding Emotion Through Self-Supervised Eye Movement Reconstruction

TL;DR

This work tackles predicting emotion from eye movement in naturalistic, low-resolution video, addressing limitations of high-end gaze data. It presents GLASS, a transformer-based encoder-decoder trained in a self-supervised manner to reconstruct future eye movement, followed by finetuning an emotion head on downstream tasks. The two experiments show GLASS improves VAD regression and behavior classification, with findings that longer gaze sequences and longer pretraining forecast horizons enhance performance, and that pretraining quality correlates with downstream results. The approach enables scalable affective analysis from large, unlabeled, real-world video data, using a low-cost eye-tracking signal captured in natural settings.

Abstract

The relationship between emotional expression and eye movement is well-documented, with literature establishing gaze patterns are reliable indicators of emotion. However, most studies utilize specialized, high-resolution eye-tracking equipment, limiting the potential reach of findings. We investigate how eye movement can be used to predict multimodal markers of emotional expression from naturalistic, low-resolution videos. We utilize a collection of video interviews from the USC Shoah Foundation's Visual History Archive with Holocaust survivors as they recount their experiences in the Auschwitz concentration camp. Inspired by pretraining methods on language models, we develop a novel gaze detection model that uses self-supervised eye movement reconstruction that can effectively leverage unlabeled video. We use this model's encoder embeddings to fine-tune models on two downstream tasks related to emotional expression. The first is aligning eye movement with directional emotion estimates from speech. The second task is using eye gaze as a predictor of three momentary manifestations of emotional behaviors: laughing, crying/sobbing, and sighing. We find our new model is predictive of emotion outcomes and observe a positive correlation between pretraining performance and emotion processing performance for both experiments. We conclude self-supervised eye movement reconstruction is an effective method for encoding the affective signal they carry.

Figures (5)

• Figure 1: Example frames of a sample interview with a Holocaust survivor from the VOICES dataset.
• Figure 2: Architecture of GLASS. Pretraining starts with an encoder-decoder eye gaze prediction task and for downstream prediction the decoder is replaced with an emotion model head.
• Figure 3: Kernel density estimates (KDEs) of emotional state distributions as a function of the z-score of their Euclidean distance from the dataset mean, comparing our original audio model with our modified sampling. VAD indicates valence, arousal, and dominance.
• Figure 4: Performance of GLASS on the self-supervised task of subsequent gaze prediction on a held-out validation set. Three different conditions for GLASS are tested: model size (top), number of input seconds received (middle), and number of subsequent seconds predicted (bottom). The predict-previous baseline predicts the next frame to be the current frame after teacher-forcing. All GLASS models predict autoregressively without teacher-forcing.
• Figure 5: Validation gaze correlation after GLASS's self-supervised pretraining vs. Exp. 1 and Exp. 2 metrics. We present negative MAE as lower MAE is more desirable. Point shapes correspond to input time windows.