Discrete Facial Encoding: : A Framework for Data-driven Facial Display Discovery

TL;DR

This work introduces Discrete Facial Encoding (DFE), an unsupervised framework that learns a compact, interpretable dictionary of facial expressions by disentangling identity and pose with 3D Morphable Models and encoding expression features via a Residual Vector-Quantized VAE. Expressions are represented as a sequence of discrete tokens, each visualizable as localized facial deformations, enabling interpretable templates and compositional decoding. Across expression fidelity, diversity, and downstream psychological tasks (stress, depression, personality), DFE consistently outperforms FACS-based and several self-supervised baselines while offering superior interpretability and token-wise visualization. The approach demonstrates strong potential for scalable affective computing applications, with limitations including dependence on 3DMM richness and static-image focus, suggesting avenues for temporal and multimodal extensions.

Abstract

Facial expression analysis is central to understanding human behavior, yet existing coding systems such as the Facial Action Coding System (FACS) are constrained by limited coverage and costly manual annotation. In this work, we introduce Discrete Facial Encoding (DFE), an unsupervised, data-driven alternative of compact and interpretable dictionary of facial expressions from 3D mesh sequences learned through a Residual Vector Quantized Variational Autoencoder (RVQ-VAE). Our approach first extracts identity-invariant expression features from images using a 3D Morphable Model (3DMM), effectively disentangling factors such as head pose and facial geometry. We then encode these features using an RVQ-VAE, producing a sequence of discrete tokens from a shared codebook, where each token captures a specific, reusable facial deformation pattern that contributes to the overall expression. Through extensive experiments, we demonstrate that Discrete Facial Encoding captures more precise facial behaviors than FACS and other facial encoding alternatives. We evaluate the utility of our representation across three high-level psychological tasks: stress detection, personality prediction, and depression detection. Using a simple Bag-of-Words model built on top of the learned tokens, our system consistently outperforms both FACS-based pipelines and strong image and video representation learning models such as Masked Autoencoders. Further analysis reveals that our representation covers a wider variety of facial displays, highlighting its potential as a scalable and effective alternative to FACS for psychological and affective computing applications.

Figures (9)

• Figure 1: Overview of our proposed expression coding framework. Given an input expression vector $\boldsymbol{\psi}$ extracted from a 3DMM model, a transformer-based encoder maps it into a latent representation. This representation is then quantized using Residual VQ to produce discrete expression tokens. A lightweight MLP decoder reconstructs the expression vector $\tilde{\boldsymbol{\psi}}$, preserving additive structure and interpretability.
• Figure 2: Some examples of deformation heatmap.
• Figure 3: Some expression templates discovered by our system.
• Figure 4: Example facial images and their corresponding token decompositions produced by our system.
• Figure 5: Qualitative retrieval examples comparing our token-based representation (DFE) with AU-based encoding.