Tiny is not small enough: High-quality, low-resource facial animation models through hybrid knowledge distillation

TL;DR

The paper tackles the challenge of producing high-quality, speech-driven 3D facial animations on resource-constrained devices by addressing the lack of large paired audio-animation data. It introduces a two-stage hybrid knowledge distillation framework that first applies heterogeneous KD from a large, transformer-based teacher to a CNN-based student, then uses the resulting student as a homogeneous teacher to train even smaller, faster models with feature supervision, all aided by pseudo-labeling from the teacher. The approach achieves dramatic model compression (up to 1000x smaller than the teacher) and latency reductions (down to 64 ms future-context, with many variants under 140 ms) while maintaining a high percentage of the teacher’s lip-sync quality, verified through quantitative metrics like PBM accuracy, MSE, and jitter, as well as qualitative assessments and a human user study. A real-time ensemble predictor further stabilizes outputs for on-device gameplay scenarios. The work demonstrates practical on-device facial animation suitable for games and real-time digital characters, and shows generalization across speakers, languages, and even alternative teacher models, underscoring the robustness and applicability of hybrid KD for low-resource animation pipelines.

Abstract

The training of high-quality, robust machine learning models for speech-driven 3D facial animation requires a large, diverse dataset of high-quality audio-animation pairs. To overcome the lack of such a dataset, recent work has introduced large pre-trained speech encoders that are robust to variations in the input audio and, therefore, enable the facial animation model to generalize across speakers, audio quality, and languages. However, the resulting facial animation models are prohibitively large and lend themselves only to offline inference on a dedicated machine. In this work, we explore on-device, real-time facial animation models in the context of game development. We overcome the lack of large datasets by using hybrid knowledge distillation with pseudo-labeling. Given a large audio dataset, we employ a high-performing teacher model to train very small student models. In contrast to the pre-trained speech encoders, our student models only consist of convolutional and fully-connected layers, removing the need for attention context or recurrent updates. In our experiments, we demonstrate that we can reduce the memory footprint to up to 3.4 MB and required future audio context to up to 81 ms while maintaining high-quality animations. This paves the way for on-device inference, an important step towards realistic, model-driven digital characters.

Figures (10)

• Figure 1: Heterogeneous Knowledge distillation: A teacher model with frozen parameters is used to generate animation data for a large audio corpus. This is used to train a small, less complex student model. As the teacher is based on transformers, it receives the entire audio file as input while the student only sees input audio windows centered around the current frame.
• Figure 2: Hybrid knowledge distillation: We use the supervising signal of the V2F teacher as well as a feature loss computed with the help of the small student that was trained in the heterogeneous KD step. We train two different conditions: smaller students (in row three) and students with reduced latency (in row four).
• Figure 3: Metrics visualization of PBM accuracy, reconstruction loss, and jitter. $\tilde{S}$ represents smoothed results using the real-time ensemble prediction method described in Section \ref{['sec:method:ensemble']}. The teacher is depicted in gray. Baselines are green and our models are blue. We use light and dark green for baseline models trained without and with KD. Light blue indicates our models trained with heterogeneous KD, medium blue indicates models trained with hybrid KD and dark blue are hybrid KD models with smoothing.
• Figure 4: A visualization of mouth poses given different sounds across several models. The focus is on /p/, /b/, and /m/ sounds and lip role during the $f$ sound. The teacher $T$ is given as a reference in the top row.
• Figure 5: Lip vertices over time for $S_0$ vs MFCC ($M_{KD}$) to visualize lip closure.