KeyFace: Expressive Audio-Driven Facial Animation for Long Sequences via KeyFrame Interpolation

TL;DR

KeyFace introduces a two-stage latent-diffusion framework to address long-duration audio-driven facial animation by generating low-frame-rate keyframes conditioned on audio and identity, then interpolating between them for temporal coherence. The model incorporates continuous emotion via valence and arousal and explicitly handles non-speech vocalizations, supported by new LipScore and NSV-accuracy metrics. Through a combination of latent diffusion, dual audio encoders, specialized losses, and tailored guidance, KeyFace achieves state-of-the-art results on long sequences and demonstrates robust emotion interpolation and NSV generation with strong subjective preferences. This approach significantly improves realism, continuity, and expressiveness for extended talking-face videos, enabling more natural and empathetic avatars in real-time and offline applications.

Abstract

Current audio-driven facial animation methods achieve impressive results for short videos but suffer from error accumulation and identity drift when extended to longer durations. Existing methods attempt to mitigate this through external spatial control, increasing long-term consistency but compromising the naturalness of motion. We propose KeyFace, a novel two-stage diffusion-based framework, to address these issues. In the first stage, keyframes are generated at a low frame rate, conditioned on audio input and an identity frame, to capture essential facial expressions and movements over extended periods of time. In the second stage, an interpolation model fills in the gaps between keyframes, ensuring smooth transitions and temporal coherence. To further enhance realism, we incorporate continuous emotion representations and handle a wide range of non-speech vocalizations (NSVs), such as laughter and sighs. We also introduce two new evaluation metrics for assessing lip synchronization and NSV generation. Experimental results show that KeyFace outperforms state-of-the-art methods in generating natural, coherent facial animations over extended durations, successfully encompassing NSVs and continuous emotions.

Figures (20)

• Figure 1: KeyFace generates long-term videos using a two-stage pipeline: first, keyframes are created as anchor points, then they are used by an interpolation model to produce smooth transitions.
• Figure 2: Overview of KeyFace's two-stage framework. The main architecture (a) is shared between the two stages, differing only in the conditioning inputs. A detailed view is provided in (b). In the keyframe generation stage, the model receives an identity frame $x_{\text{id}}$, repeated and concatenated with the noised video input to match the input dimensions. In the interpolation stage, the model is conditioned on two consecutive frames $z_{s}$ and $z_{e}$ from the keyframe sequence, interpolating the intermediate frames using a learned masked embedding $z_m$ and a binary mask $M$. Both stages incorporate audio embeddings $A_{wb}$ from WavLM and BEATs (c). We also use continuous emotion embeddings in the keyframe generation to produce facial animations that accurately convey both speech content and emotional expressions.
• Figure 3: We present sliding window FID with a 1-second window size for videos generated by different methods.
• Figure 4: We show KeyFace's ability to interpolate between several different emotions within the same video.
• Figure 5: We show the impact of guidance scale for identity and audio condition on FID and LipScore on HDTF hdtf.