Context-aware Talking Face Video Generation

TL;DR

Context-aware Talking Face Video Generation tackles generating talking head videos conditioned on driving audio in scenarios with audience or environmental context. It introduces a Two-stage Cross-modal Control Pipeline (TCCP) that first converts context into explicit facial landmarks and then synthesizes the talking face video, guided by a Multi-modal MVControlNet diffusion model with both latent-based and video-based conditions. Experiments on TV-show data demonstrate improved audio-visual synchronization, higher fidelity, and better frame coherence compared to baselines, validating the effectiveness of explicit contextual conditioning. This approach enables more natural integration of talking heads into real scenes, with potential applications in digital humans, virtual avatars, and multi-person video generation.

Abstract

In this paper, we consider a novel and practical case for talking face video generation. Specifically, we focus on the scenarios involving multi-people interactions, where the talking context, such as audience or surroundings, is present. In these situations, the video generation should take the context into consideration in order to generate video content naturally aligned with driving audios and spatially coherent to the context. To achieve this, we provide a two-stage and cross-modal controllable video generation pipeline, taking facial landmarks as an explicit and compact control signal to bridge the driving audio, talking context and generated videos. Inside this pipeline, we devise a 3D video diffusion model, allowing for efficient contort of both spatial conditions (landmarks and context video), as well as audio condition for temporally coherent generation. The experimental results verify the advantage of the proposed method over other baselines in terms of audio-video synchronization, video fidelity and frame consistency.

Figures (7)

• Figure 1: (a) An illustration of our task setting. Given a context video and a driving audio, the task is to generate harmonious and consistent talking head video inside the masked region. (b) Our two-stage generation pipeline. Facial landmarks are adopted as an intermediate representation to enhance controllability.
• Figure 2: TCCP pipeline. Rows referring to pipeline stages: taking head facial landmark $\mathcal{L}mk$ generation and coherent talking face video $\mathcal{O}$ generation. Columns referring to temporal steps: First Frame and All Frames. Each step is built on MVControlNet, a two-branch diffusion-based model. The video diffusion branch takes in Video Diffusion Branch Input and Latent Based Condition. The control branch takes in Video-based Condition. Different types of inputs/outputs are marked by arrows in different colors.
• Figure 3: Model Architecture of multimodal controlled video generation network (MVControlNet). (a) The overall model architecture, consisting of two model branches. The Video Diffusion Branch predicts the diffusion noise added to the input at time step $t$ under the latent-based condition $c_l$. The Control Branch is a trainable copy of the video diffusion branch, which is used to add video-based condition $c_v$ to the video diffusion branch by zero convolution layer. (b) A detailed demonstration of Video Diffusion Block. 1D temporal convolutions and temporal self attention are stacked after spatial layers to enforce spatial-temporal coherence. (c) A brief illustration of encoding of latent-based condition $c_l$. Conditions of multiple modalities will be encoded separately, and then concatenated by frame.
• Figure 4: We compare our work with baselines in face generation quality. Our model can generate harmonious and consistent talking face with vivid facial expressions and diverse head poses.
• Figure 5: We compare our work with baselines considering full conversation scene quality. Scene details are given in the right column, and inconsistency can be observed near the shoulder in results generated by ControlNet and SadTalker.