Emotional Conversation: Empowering Talking Faces with Cohesive Expression, Gaze and Pose Generation

TL;DR

This paper tackles the challenge of generating emotionally expressive talking-face videos that coherently align expression, gaze, and head pose with emotion, beyond lip-sync alone. It introduces a two-stage framework that first predicts emotionally aligned facial cues from speech and emotion, then maps relocated landmarks to latent keypoints for high-quality synthesis using a pretrained generator. The approach combines auto-regressive cue synthesis (landmarks, gaze, pose) with a collaborative emotion classifier and gaze discretization, demonstrating significant improvements on the MEAD dataset over state-of-the-art methods. The method offers a scalable, cost-efficient path to realistic, emotionally aware talking faces for animation and AI-generated content.

Abstract

Vivid talking face generation holds immense potential applications across diverse multimedia domains, such as film and game production. While existing methods accurately synchronize lip movements with input audio, they typically ignore crucial alignments between emotion and facial cues, which include expression, gaze, and head pose. These alignments are indispensable for synthesizing realistic videos. To address these issues, we propose a two-stage audio-driven talking face generation framework that employs 3D facial landmarks as intermediate variables. This framework achieves collaborative alignment of expression, gaze, and pose with emotions through self-supervised learning. Specifically, we decompose this task into two key steps, namely speech-to-landmarks synthesis and landmarks-to-face generation. The first step focuses on simultaneously synthesizing emotionally aligned facial cues, including normalized landmarks that represent expressions, gaze, and head pose. These cues are subsequently reassembled into relocated facial landmarks. In the second step, these relocated landmarks are mapped to latent key points using self-supervised learning and then input into a pretrained model to create high-quality face images. Extensive experiments on the MEAD dataset demonstrate that our model significantly advances the state-of-the-art performance in both visual quality and emotional alignment.

Figures (7)

• Figure 1: Architecture of the proposed method, which performs emotional talking face generation in two steps. In Step 1, we innovatively achieved the simultaneous generation of facial cue sequences, including normalized landmarks, gaze, and head pose. These cues are then aligned with emotional labels via self-supervised learning. In Step 2, we utilize the emotionally aligned facial cues from Step 1 as inputs, employing a pre-trained model to produce vivid emotional talking face videos.
• Figure 2: Pipeline of gaze direction discretization.
• Figure 3: Qualitative comparison of generated normalized landmarks between our method and three other methods on the MEAD dataset.
• Figure 4: Comparison of left eye gaze distribution between our model and state-of-the-art models across different emotion categories.
• Figure 5: Visualization of the head pose sequences in the pitch, yaw, and roll directions under different emotions.