Synergizing Motion and Appearance: Multi-Scale Compensatory Codebooks for Talking Head Video Generation

TL;DR

This work tackles realistic talking head video generation by addressing both motion accuracy and appearance fidelity from a one-shot source image. It introduces a unified framework that jointly learns multi-scale motion and appearance codebooks and employs transformer-based compensation to refine motion flows and intermediate appearance features across scales. The approach demonstrates substantial improvements over state-of-the-art methods on VoxCeleb1 and competitive results on CelebV-HQ, supported by comprehensive ablations that validate the benefits of multi-scale codebooks, code allocation, and joint learning. The results indicate strong potential for high-quality, identity-preserving talking head video generation with practical implications for real-time synthesis and related applications.

Abstract

Talking head video generation aims to generate a realistic talking head video that preserves the person's identity from a source image and the motion from a driving video. Despite the promising progress made in the field, it remains a challenging and critical problem to generate videos with accurate poses and fine-grained facial details simultaneously. Essentially, facial motion is often highly complex to model precisely, and the one-shot source face image cannot provide sufficient appearance guidance during generation due to dynamic pose changes. To tackle the problem, we propose to jointly learn motion and appearance codebooks and perform multi-scale codebook compensation to effectively refine both the facial motion conditions and appearance features for talking face image decoding. Specifically, the designed multi-scale motion and appearance codebooks are learned simultaneously in a unified framework to store representative global facial motion flow and appearance patterns. Then, we present a novel multi-scale motion and appearance compensation module, which utilizes a transformer-based codebook retrieval strategy to query complementary information from the two codebooks for joint motion and appearance compensation. The entire process produces motion flows of greater flexibility and appearance features with fewer distortions across different scales, resulting in a high-quality talking head video generation framework. Extensive experiments on various benchmarks validate the effectiveness of our approach and demonstrate superior generation results from both qualitative and quantitative perspectives when compared to state-of-the-art competitors.

Figures (13)

• Figure 1: Effect of motion and appearance compensation with jointly learned compensatory codebooks. Motion flows and warped source appearance features are refined by the complementary information retrieved from the motion and appearance codebooks, which jointly contribute to high-quality generated results.
• Figure 2: Overview of the framework. For each scale, multi-scale motion and appearance codebook compensation consists of two sub-modules. (i) Motion Codebook Compensation (MCC) compensates for a motion flow with the motion codebook. (ii) To refine the source facial feature warped by the compensated motion flow, Appearance Codebook Compensation (ACC) produces the compensated appearance feature with the appearance codebook for image decoding. These two sub-modules are employed for all scales. We learn the motion and appearance codebooks jointly with the whole framework.
• Figure 3: Illustration of motion codebook learning and compensation for scale $i$. We adopt a transformer structure $\mathcal{T}_M$ to compensate for the motion flow using the motion codebook. The proposed motion codebook is learned under the supervision of a reconstruction loss and a code-level loss.
• Figure 4: Illustration of appearance codebook learning and compensation for scale $i$. We utilize a transformer structure $\mathcal{T}_A$ to compensate for the warped features with the appearance codebook, adding more facial details to the feature map. Similar to the motion codebook, the designed appearance codebook is learned under the supervision of a code-level loss.
• Figure 5: Illustration of the code allocation scheme. We take the multi-scale motion codebook as an example.