Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Video Joint-Embedding Predictive Architectures for Facial Expression Recognition

Lennart Eing, Cristina Luna-Jiménez, Silvan Mertes, Elisabeth André

TL;DR

This work shows that pixel-level reconstruction pretexts are not required for effective facial expression recognition. By leveraging a pre-trained Video Joint-Embedding Predictive Architecture (V-JEPA) encoder frozen during FER training and using an attentive probing head, the authors achieve state-of-the-art results on RAVDESS and strong performance on CREMA-D, with notable cross-dataset generalization. The approach emphasizes embedding-based pretraining and predictive latent representations over pixel reconstructions, suggesting a practical pathway to robust FER with limited labeled data. The work includes release of code and highlights potential generalization to in-the-wild scenarios, albeit with caveats about dataset scope and label mapping.

Abstract

This paper introduces a novel application of Video Joint-Embedding Predictive Architectures (V-JEPAs) for Facial Expression Recognition (FER). Departing from conventional pre-training methods for video understanding that rely on pixel-level reconstructions, V-JEPAs learn by predicting embeddings of masked regions from the embeddings of unmasked regions. This enables the trained encoder to not capture irrelevant information about a given video like the color of a region of pixels in the background. Using a pre-trained V-JEPA video encoder, we train shallow classifiers using the RAVDESS and CREMA-D datasets, achieving state-of-the-art performance on RAVDESS and outperforming all other vision-based methods on CREMA-D (+1.48 WAR). Furthermore, cross-dataset evaluations reveal strong generalization capabilities, demonstrating the potential of purely embedding-based pre-training approaches to advance FER. We release our code at https://github.com/lennarteingunia/vjepa-for-fer.

Video Joint-Embedding Predictive Architectures for Facial Expression Recognition

TL;DR

This work shows that pixel-level reconstruction pretexts are not required for effective facial expression recognition. By leveraging a pre-trained Video Joint-Embedding Predictive Architecture (V-JEPA) encoder frozen during FER training and using an attentive probing head, the authors achieve state-of-the-art results on RAVDESS and strong performance on CREMA-D, with notable cross-dataset generalization. The approach emphasizes embedding-based pretraining and predictive latent representations over pixel reconstructions, suggesting a practical pathway to robust FER with limited labeled data. The work includes release of code and highlights potential generalization to in-the-wild scenarios, albeit with caveats about dataset scope and label mapping.

Abstract

This paper introduces a novel application of Video Joint-Embedding Predictive Architectures (V-JEPAs) for Facial Expression Recognition (FER). Departing from conventional pre-training methods for video understanding that rely on pixel-level reconstructions, V-JEPAs learn by predicting embeddings of masked regions from the embeddings of unmasked regions. This enables the trained encoder to not capture irrelevant information about a given video like the color of a region of pixels in the background. Using a pre-trained V-JEPA video encoder, we train shallow classifiers using the RAVDESS and CREMA-D datasets, achieving state-of-the-art performance on RAVDESS and outperforming all other vision-based methods on CREMA-D (+1.48 WAR). Furthermore, cross-dataset evaluations reveal strong generalization capabilities, demonstrating the potential of purely embedding-based pre-training approaches to advance FER. We release our code at https://github.com/lennarteingunia/vjepa-for-fer.
Paper Structure (17 sections, 2 equations, 6 figures, 4 tables)

This paper contains 17 sections, 2 equations, 6 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Literature
  3. Self-supervised FER methods
  4. Joint-Embedding Predictive Architectures
  5. Methodology
  6. vjepa
  7. Attentive Probing
  8. Classifying Whole Videos
  9. Experiments
  10. Training Setup
  11. Data
  12. Metrics
  13. Results and Discussion
  14. RAVDESS and CREMA-D
  15. Cross-Dataset Evaluation
  16. ...and 2 more sections

Figures (6)

  • Figure 1: General overview of a jepa.
  • Figure 2: Training Setup for vjepa (left) and Attentive Probing (right)
  • Figure 3: Confusion matrices of same-dataset experiments.
  • Figure 4: Average confusion matrixes of cross-dataset evaluations.
  • Figure 5: Visualization of the first two principal components (PCA) using global average pooling (left), and attentive probing (right). Embeddings were produced using the best model trained on CREMA-D during k-fold validation on the respective validation dataset.
  • ...and 1 more figures