LawDNet: Enhanced Audio-Driven Lip Synthesis via Local Affine Warping Deformation

TL;DR

LawDNet tackles the challenge of realistic audio driven lip synthesis by introducing Local Affine Warping Deformation, a sparse, region-specific warping mechanism guided by learnable keypoints and local affine parameters to capture non-linear lip motion. A coarse-grained grid and a dual-stream GAN discriminator (spatial and temporal) enforce both high-quality lip shapes and frame-to-frame coherence, while face frontalization and synchronization preprocessing stabilize performance across poses. Quantitative and qualitative evaluations on HDTF and the corrected Chinese Authentic Talking Head Dataset show LawDNet achieves superior lip dynamism and temporal stability with notable improvements in SSIM and LSE-D over state-of-the-art methods, and demonstrate robust generalization to TTS audio. These results suggest LawDNet offers a practical and scalable approach for high-fidelity lip synchronization in photorealistic avatars and virtual agents, with potential extensions to 3D lip reading and motion transfer scenarios.

Abstract

In the domain of photorealistic avatar generation, the fidelity of audio-driven lip motion synthesis is essential for realistic virtual interactions. Existing methods face two key challenges: a lack of vivacity due to limited diversity in generated lip poses and noticeable anamorphose motions caused by poor temporal coherence. To address these issues, we propose LawDNet, a novel deep-learning architecture enhancing lip synthesis through a Local Affine Warping Deformation mechanism. This mechanism models the intricate lip movements in response to the audio input by controllable non-linear warping fields. These fields consist of local affine transformations focused on abstract keypoints within deep feature maps, offering a novel universal paradigm for feature warping in networks. Additionally, LawDNet incorporates a dual-stream discriminator for improved frame-to-frame continuity and employs face normalization techniques to handle pose and scene variations. Extensive evaluations demonstrate LawDNet's superior robustness and lip movement dynamism performance compared to previous methods. The advancements presented in this paper, including the methodologies, training data, source codes, and pre-trained models, will be made accessible to the research community.

Figures (7)

• Figure 1: Audio-driven lip synthesis: This figure demonstrates the effect of generating new lip shapes in the source video according to the driven audio.
• Figure 2: Overview of the LawDNet framework: Data preprocessing aligns inputs via face frontalization and soft masking. Feature modulation, guided by audio and visual cues, uses keypoints and affine parameters for feature warping, generating lip-synced outputs via $G$. Dual discriminators $D_T$ and $D_S$ and multi-level losses ensure training stability and quality.
• Figure 3: Illustration of the data preprocessing.
• Figure 4: Illustrations of the local affine warping Deformation process applied to feature maps, with keypoints, radii ($\rho$), and affine transformation parameters ($S,R,T$) learned under the guidance of audio-visual features. Each module generates a distinct warping field $A^c|_p$ on the coarse-grained grid, effectuating the deformation of the feature maps across individual channels.
• Figure 5: Qualitative comparison with SOTAs. The left images are generated using the original audio from the source videos, with the goal of closely replicating the ground truth lip shapes. The right images display lip synchronization with Text-to-Speech (TTS) audio, evaluating the model's generalization capabilities. Note: 'MakeItTalk zhou2020makelttalk' and 'PC-AVS zhou2021pose' require additional head and eye motion input, hence the discrepancy in pose.