FLOAT: Generative Motion Latent Flow Matching for Audio-driven Talking Portrait

TL;DR

FLOAT introduces flow matching in a learned motion latent space to generate audio-driven talking portraits with temporally coherent motion and improved efficiency. By embedding motion in an orthogonal basis and using a transformer-based vector-field predictor conditioned on audio and emotion, FLOAT reduces sampling steps while maintaining high visual quality. The method demonstrates state-of-the-art results on HDTF and RAVDESS across multiple metrics and supports flexible at-test control, including lambda-based motion editing and emotion redirection. This approach offers a practical, efficient alternative to diffusion-based video synthesis for realistic talking portraits with emotion-aware dynamics.

Abstract

With the rapid advancement of diffusion-based generative models, portrait image animation has achieved remarkable results. However, it still faces challenges in temporally consistent video generation and fast sampling due to its iterative sampling nature. This paper presents FLOAT, an audio-driven talking portrait video generation method based on flow matching generative model. Instead of a pixel-based latent space, we take advantage of a learned orthogonal motion latent space, enabling efficient generation and editing of temporally consistent motion. To achieve this, we introduce a transformer-based vector field predictor with an effective frame-wise conditioning mechanism. Additionally, our method supports speech-driven emotion enhancement, enabling a natural incorporation of expressive motions. Extensive experiments demonstrate that our method outperforms state-of-the-art audio-driven talking portrait methods in terms of visual quality, motion fidelity, and efficiency.

Figures (26)

• Figure 1: Quantitative comparison results with state-of-the-art methods on HDTF hdtf / RAVDESS ravdess. The best result for each metric is in bold, and the second-best result is underlined. $^{\dagger}$: evaluated with raw $256\times256$ resolution outputs.
• Figure 2: Overview of FLOAT. We encode the source image $S\in\mathbb{R}^{3\times H \times W}$ into the latent with the explicit identity-motion decomposition $w_s = w_{s\to r} + w_{r \to s} \in \mathbb{R}^d$. Given audio segments $a^{-L':L} \in \mathbb{R}^{(L'+L) \times d_a}$ of the length $L' + L$ and the reference motion $w_{r\to s}$$\in$$\mathbb{R}^{d}$, and the speech-driven emotion label $w_e$$\in$$\mathbb{R}^7$, a flow matching transformer estimates the generating vector field $v_t(\varphi_t(x_0), \mathbf{c}_t; \theta) \in \mathbb{R}^{L \times d}$ from noisy motion latents, which is used to solve corresponding ODE and generates the motion latents $w_{r\to\hat{D}^{1:L}}$. Finally, the sequence of latents $w_{S \to \hat{D}^{1:L}} := (w_{S \to r} + w_{r \to \hat{D}^{l}})_{l=1}^{L}$ are decoded into the video $\hat{D}^{1:L} \in \mathbb{R}^{L \times 3 \times H \times W}$.
• Figure 3: Efficacy of $\mathcal{L}_{\text{comp-lp}}$ for fine-grained motion and fidelity.
• Figure 4: Frame-wise vector field predictor block at inference.
• Figure 5: Qualitative comparison results with state-of-the-art methods on HDTF hdtf / RAVDESS ravdess. Please refer to supplementary videos. Note that we additionally provide a video comparison with EMOemo and VASA-1vasa_1 using their video demonstration.