Faces that Speak: Jointly Synthesising Talking Face and Speech from Text

TL;DR

This paper tackles the problem of generating synchronized talking-face video and speech from text and a portrait by unifying text-driven talking-face generation with face-stylised TTS. It introduces an OT-CFM-based motion sampler coupled with an auto-encoder motion normaliser to produce realistic, pose-rich facial motions, and a motion-removal conditioning strategy that enables speaker-consistent voice across facial motion. The TTS is conditioned on identity features from the TFG model, enabling robust voice synthesis that preserves prosody, timbre, and accent while producing natural lip movements aligned with the generated speech. Experiments on LRS3, VoxCeleb2, and LRS2 demonstrate superior video quality and lip-sync, strong unseen-identity generalization, and clear advantages over cascade baselines, highlighting the practical potential of unified text-driven multimodal synthesis.

Abstract

The goal of this work is to simultaneously generate natural talking faces and speech outputs from text. We achieve this by integrating Talking Face Generation (TFG) and Text-to-Speech (TTS) systems into a unified framework. We address the main challenges of each task: (1) generating a range of head poses representative of real-world scenarios, and (2) ensuring voice consistency despite variations in facial motion for the same identity. To tackle these issues, we introduce a motion sampler based on conditional flow matching, which is capable of high-quality motion code generation in an efficient way. Moreover, we introduce a novel conditioning method for the TTS system, which utilises motion-removed features from the TFG model to yield uniform speech outputs. Our extensive experiments demonstrate that our method effectively creates natural-looking talking faces and speech that accurately match the input text. To our knowledge, this is the first effort to build a multimodal synthesis system that can generalise to unseen identities.

Figures (5)

• Figure 1: Our framework integrates Talking Face Generation (TFG) and Text-to-Speech (TTS) systems, generating synchronised natural speech and a talking face video from a single portrait and text input. Our model is capable of variational motion generation by conditioning the TFG model with the intermediate representations of the TTS model. The speech is conditioned using the identity features extracted in the TFG model to align with the input identity.
• Figure 2: Overall architecture of our framework. The TTS model receives identity representations from the TFG model, while the TFG model takes conditions for natural motion generation from the TTS model. These complementary elements enhance our model's capabilities in generating both speech and talking faces. The EMB block denotes an embedding operation. The grey dashed arrow represents a path used only during the training process, and the red arrows represent paths used only during the inference process.
• Figure 3: The architecture of the audio mapper. The condition denotes the concatenated feature of text embedding $\boldsymbol{e}_{t}$, upsampled text feature $\Tilde{f}_t$, and energy, which is a norm of $\Tilde{f}_t$.
• Figure 4: Qualitative Results. We compare our method with several baselines listed in \ref{['tab:results_lrs2']}. Our approach outperforms all the baselines in terms of generating natural facial motions, encompassing lip shape and head pose. MakeItTalk and SadTalker exhibit smaller variance in head poses, while Audio2Head fails to preserve the source identity. We emphasis that our TTSF system can generate sophisticated lip shapes, reflecting both linguistic and acoustic information from our TTS model.
• Figure 5: Speaker representation space of (a) Face-TTS and (b) Ours. Each colour represents a different speaker.