Ditto: Motion-Space Diffusion for Controllable Realtime Talking Head Synthesis

TL;DR

Ditto tackles the bottlenecks of diffusion-based talking head synthesis by designing a motion-space diffusion framework grounded in LivePortrait-derived motion representations. A Conditional Diffusion Transformer, guided by enhanced signals and a clear motion-to-semantics mapping, enables fine-grained control over facial motion while supporting streaming, low-latency inference. The method achieves superior controllability, lip-sync accuracy, and identity preservation on standard benchmarks, and demonstrates real-time performance suitable for interactive AI applications. The work also provides ablations and a practical pipeline for streaming inference, with code released for public use.

Abstract

Recent advances in diffusion models have endowed talking head synthesis with subtle expressions and vivid head movements, but have also led to slow inference speed and insufficient control over generated results. To address these issues, we propose Ditto, a diffusion-based talking head framework that enables fine-grained controls and real-time inference. Specifically, we utilize an off-the-shelf motion extractor and devise a diffusion transformer to generate representations in a specific motion space. We optimize the model architecture and training strategy to address the issues in generating motion representations, including insufficient disentanglement between motion and identity, and large internal discrepancies within the representation. Besides, we employ diverse conditional signals while establishing a mapping between motion representation and facial semantics, enabling control over the generation process and correction of the results. Moreover, we jointly optimize the holistic framework to enable streaming processing, real-time inference, and low first-frame delay, offering functionalities crucial for interactive applications such as AI assistants. Extensive experimental results demonstrate that Ditto generates compelling talking head videos and exhibits superiority in both controllability and real-time performance.

Figures (6)

• Figure 1: The summary of the proposed Ditto, where $\mathcal{T}$ is the DiT for motion generation, $\mathcal{H}$ is the HuBERT for audio feature extraction, $\mathcal{M}$ is the Motion Extractor, $\mathcal{F}$ is the Appearance Feature Extractor and $\mathcal{G}$ is the Face Renderer.
• Figure 2: Visualization of the architecture of the proposed conditional Diffusion Transformer, which generates compact motion representations based on various conditional signals.
• Figure 3: Examples of dedicated fine-grained control of subtle facial motion. The notation "15-0" in the top-left image denotes the application of an x-axis offset to the 15th keypoint. The central image depicts the rendered result when applying zero offsets across all dimensions.
• Figure 4: Generation results with portraits of different styles and scales. Fine-grained control over gaze, emotion, pose, etc.
• Figure 5: The qualitative comparison of our approach on two characters with different styles, poses, and hairstyles. For each character, we generate videos using the same audio input with each method and select frames at the same location for comparison. Blue arrows indicate the locations of artifacts, including inaccurate lip movements, disordered teeth, blur, etc.