End-to-End Facial Expression Detection in Long Videos

TL;DR

The paper tackles the problem of end-to-end facial expression detection in long videos by jointly optimizing spotting (onset/offset) and recognition (emotion category) within a single network, FEDN. FEDN employs an attention-driven backbone (segment and sliding window attention), a temporal feature pyramid (neck), and decoupled task heads to simultaneously locate expressions and classify them, while using a 1D Distance IoU loss and binary cross-entropy for training. Empirical results on CASME$^2$ and CASME$^3$ show state-of-the-art performance in both spotting and detection, with notable gains over cascaded or single-task baselines, validating the benefits of joint optimization and facial-specific feature learning. The work reduces error propagation from sequential pipelines, lowers computational cost by eschewing optical flow, and provides a robust framework for robust expression detection in long videos with practical implications for psychology, neuroscience, and human-computer interaction.

Abstract

Facial expression detection involves two interrelated tasks: spotting, which identifies the onset and offset of expressions, and recognition, which classifies them into emotional categories. Most existing methods treat these tasks separately using a two-step training pipelines. A spotting model first detects expression intervals. A recognition model then classifies the detected segments. However, this sequential approach leads to error propagation, inefficient feature learning, and suboptimal performance due to the lack of joint optimization of the two tasks. We propose FEDN, an end-to-end Facial Expression Detection Network that jointly optimizes spotting and recognition. Our model introduces a novel attention-based feature extraction module, incorporating segment attention and sliding window attention to improve facial feature learning. By unifying two tasks within a single network, we greatly reduce error propagation and enhance overall performance. Experiments on CASME}^2 and CASME^3 demonstrate state-of-the-art accuracy for both spotting and detection, underscoring the benefits of joint optimization for robust facial expression detection in long videos.

Figures (6)

• Figure 1: Overview of our facial expression detection network (FEDN). The model consists of three main components: backbone, neck, and head. The backbone introduces two novel attention-based modules-segment attention and fusion and sliding window attention-to enhance feature extraction from facial segments. The neck constructs a feature pyramid with four levels of different temporal resolutions, while the head generates bounding boxes at each level, representing detected expressions with temporal locations and confidence scores. The pipeline integrates bounding boxes from all sliding windows in the video and applies NMS to remove overlapping boxes, producing the final outputs. (($h, w, c$)=image dimensions, $s$=segment number of a sliding window, $f$=frame number of a segment, ($d, d_1, d_2$)=hidden layer dimensions), $C$=number of classes).
• Figure 2: AP vs. IoU curves for spotting and detection in CASME$^2$, comparing our method against STR-based and cascaded baselines.
• Figure 3: Detection AP (self-reported) across IoU intervals.
• Figure 4: Framework variations comparing our design with alternatives, including the addition of a confidence branch and the use of a coupled head.
• Figure 5: Confusion matrix of CASME$^3$.