Exploring Talking Head Models With Adjacent Frame Prior for Speech-Preserving Facial Expression Manipulation

TL;DR

SPFEM editing often distorts lip synchronization due to the entanglement of expressions and mouth shapes. The authors propose THFEM, a two-stage framework that fuses SPFEM with AD-THG and augments it with adjacent-frame priors to generate sequences of frames with faithful lip movements. Extensive experiments on MEAD and supplementary datasets, plus a user study, show consistent improvements in FAD, LSE-D, and CSIM over strong baselines, validating the approach. The work advances practical speech-preserving facial expression manipulation with potential impact on film, virtual avatars, and digital humans.

Abstract

Speech-Preserving Facial Expression Manipulation (SPFEM) is an innovative technique aimed at altering facial expressions in images and videos while retaining the original mouth movements. Despite advancements, SPFEM still struggles with accurate lip synchronization due to the complex interplay between facial expressions and mouth shapes. Capitalizing on the advanced capabilities of audio-driven talking head generation (AD-THG) models in synthesizing precise lip movements, our research introduces a novel integration of these models with SPFEM. We present a new framework, Talking Head Facial Expression Manipulation (THFEM), which utilizes AD-THG models to generate frames with accurately synchronized lip movements from audio inputs and SPFEM-altered images. However, increasing the number of frames generated by AD-THG models tends to compromise the realism and expression fidelity of the images. To counter this, we develop an adjacent frame learning strategy that finetunes AD-THG models to predict sequences of consecutive frames. This strategy enables the models to incorporate information from neighboring frames, significantly improving image quality during testing. Our extensive experimental evaluations demonstrate that this framework effectively preserves mouth shapes during expression manipulations, highlighting the substantial benefits of integrating AD-THG with SPFEM.

Figures (18)

• Figure 2: An overall pipeline of the proposed THFEM that integrates SPFEM models with AD-THG models. Given a source image and a reference image, the SPFEM model first edits the source image to generate a facial image with the expression of the reference image. Subsequently, leveraging the output from SPFEM, the image encoder extracts identity information and pose encoder ascertains the head pose sequence, while the audio encoder derives mouth shape representations from the adjacent n frames of audio corresponding to the source image. Finally, the generator combines these three representations to predict n frame image sequence. (n is set to 5 in our experiments.)
• Figure : Source
• Figure : Reference
• Figure : Reference
• Figure : Source