Dimitra: Audio-driven Diffusion model for Expressive Talking Head Generation

TL;DR

Dimitra introduces a diffusion-based, audio-conditioned framework for expressive talking head generation that operates on a 3DMM representation to synthesize lip motion, facial expressions, and head pose from a single reference image and an audio sequence. The system uses a three-part pipeline (MMM, cMDT, Video Renderer) with multimodal audio conditioning (Wav2Vec features, phonemes, and transcripts) to produce realistic, dataset-agnostic videos. Quantitative and qualitative results on VoxCeleb2 and HDTF show superior landmark realism and competitive lip-sync, with user studies highlighting perceived realism. This work advances practical, expressive talking-head generation suitable for avatars and virtual assistants without requiring additional conditioning sequences.

Abstract

We propose Dimitra, a novel framework for audio-driven talking head generation, streamlined to learn lip motion, facial expression, as well as head pose motion. Specifically, we train a conditional Motion Diffusion Transformer (cMDT) by modeling facial motion sequences with 3D representation. We condition the cMDT with only two input signals, an audio-sequence, as well as a reference facial image. By extracting additional features directly from audio, Dimitra is able to increase quality and realism of generated videos. In particular, phoneme sequences contribute to the realism of lip motion, whereas text transcript to facial expression and head pose realism. Quantitative and qualitative experiments on two widely employed datasets, VoxCeleb2 and HDTF, showcase that Dimitra is able to outperform existing approaches for generating realistic talking heads imparting lip motion, facial expression, and head pose.

Figures (2)

• Figure 1: Dimitra pipeline. Dimitra comprises three main parts, a Motion Modeling Module (MMM), a Conditional Motion Diffusion Transformer (cMDT) and a Video Renderer. In the training stage, 3D meshes (3DMM) are extracted from a video by the MMM. They are used by the cMDT jointly with features extracted from an audio sequence, to noise then denoise the 3DMM sequence. In the inference stage, using an audio sequence and an identity 3DMM as condition, cMDT aims at generating a 3DMM sequence from Gaussian noise. Finally, the Video Renderer transforms the 3DMM sequence into a RGB video.
• Figure 2: Qualitative results. Examples of generated samples pertained to the VoxCeleb2 dataset on the left and pertained to HDTF on the right.