Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DEMO: Disentangled Motion Latent Flow Matching for Fine-Grained Controllable Talking Portrait Synthesis

Peiyin Chen, Zhuowei Yang, Hui Feng, Sheng Jiang, Rui Yan

TL;DR

The paper tackles the difficulty of achieving temporally coherent, finely controllable talking-head video synthesis driven by audio. It introduces DEMO, a two-stage framework with a Fine-Grained Controllable Motion Encoder that yields disentangled lip, eye gaze, and head pose representations, and an OT-based flow-matching module with a transformer vector-field predictor conditioned on audio. Key contributions include the FCME design for motion disentanglement and an OT flow-matching approach with frame-wise conditioning and temporal attention. Experimental results on MEAD, RAVDESS, and HDTF show state-of-the-art performance in video realism, lip–audio synchronization, and motion fidelity compared to diffusion- and non-diffusion baselines. The work proposes a principled paradigm for controllable talking-head synthesis that combines structured motion representations with flow-based trajectory generation, with broad implications for virtual communication and media production.

Abstract

Audio-driven talking-head generation has advanced rapidly with diffusion-based generative models, yet producing temporally coherent videos with fine-grained motion control remains challenging. We propose DEMO, a flow-matching generative framework for audio-driven talking-portrait video synthesis that delivers disentangled, high-fidelity control of lip motion, head pose, and eye gaze. The core contribution is a motion auto-encoder that builds a structured latent space in which motion factors are independently represented and approximately orthogonalized. On this disentangled motion space, we apply optimal-transport-based flow matching with a transformer predictor to generate temporally smooth motion trajectories conditioned on audio. Extensive experiments across multiple benchmarks show that DEMO outperforms prior methods in video realism, lip-audio synchronization, and motion fidelity. These results demonstrate that combining fine-grained motion disentanglement with flow-based generative modeling provides a powerful new paradigm for controllable talking-head video synthesis.

DEMO: Disentangled Motion Latent Flow Matching for Fine-Grained Controllable Talking Portrait Synthesis

TL;DR

The paper tackles the difficulty of achieving temporally coherent, finely controllable talking-head video synthesis driven by audio. It introduces DEMO, a two-stage framework with a Fine-Grained Controllable Motion Encoder that yields disentangled lip, eye gaze, and head pose representations, and an OT-based flow-matching module with a transformer vector-field predictor conditioned on audio. Key contributions include the FCME design for motion disentanglement and an OT flow-matching approach with frame-wise conditioning and temporal attention. Experimental results on MEAD, RAVDESS, and HDTF show state-of-the-art performance in video realism, lip–audio synchronization, and motion fidelity compared to diffusion- and non-diffusion baselines. The work proposes a principled paradigm for controllable talking-head synthesis that combines structured motion representations with flow-based trajectory generation, with broad implications for virtual communication and media production.

Abstract

Audio-driven talking-head generation has advanced rapidly with diffusion-based generative models, yet producing temporally coherent videos with fine-grained motion control remains challenging. We propose DEMO, a flow-matching generative framework for audio-driven talking-portrait video synthesis that delivers disentangled, high-fidelity control of lip motion, head pose, and eye gaze. The core contribution is a motion auto-encoder that builds a structured latent space in which motion factors are independently represented and approximately orthogonalized. On this disentangled motion space, we apply optimal-transport-based flow matching with a transformer predictor to generate temporally smooth motion trajectories conditioned on audio. Extensive experiments across multiple benchmarks show that DEMO outperforms prior methods in video realism, lip-audio synchronization, and motion fidelity. These results demonstrate that combining fine-grained motion disentanglement with flow-based generative modeling provides a powerful new paradigm for controllable talking-head video synthesis.

Paper Structure

This paper contains 4 sections, 6 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Experiments
  4. CONCLUSION

Figures (4)

  • Figure 1: Overview of the proposed DEMO framework for talking-head video generation. Given a source image (left) and a driving audio sequence, DEMO employs a Fine-Grained Controllable Motion Encoder (orange) to construct a disentangled motion representation that separates lip, head-pose, and eye movements. Audio embeddings enriched with emotion cues (blue) drive the motion evolution. A Vector Field Predictor with OT-based flow matching (green) refines noisy motion latents into temporally coherent trajectories, which are integrated by an ODE solver and finally decoded into high-fidelity, synchronized video frames (right).
  • Figure 2: The structure of our Fine-Grained Controllable Motion Encoder.
  • Figure 3: Qualitative comparison with existing approaches on RAVDESS/HDTF datasets.
  • Figure 4: Fine-grained motion control with DEMO. Given a source image, a driving signal and a driving audio sequence, the framework varies only one motion factor (eye gaze, head pose, or lip movement) while keeping the others fixed.