SoulX-LiveTalk Technical Report

TL;DR

SoulX-LiveTalk presents a 14B diffusion transformer framework for real-time audio driven avatar synthesis that preserves bidirectional context within video chunks. It introduces Self-Correcting Bidirectional Distillation to reduce error accumulation and a multi-step retrospective mechanism to sustain infinite streaming, while a full-stack acceleration stack achieves sub-second startup and 32 FPS. The approach blends Latency-Aware Spatiotemporal Adaptation with efficient distillation, eliminating classifier-free guidance and enabling rapid training with minimal steps. Extensive quantitative and qualitative evaluations on TalkBench demonstrate improvements in visual fidelity, lip-sync, and temporal stability, surpassing several state-of-the-art baselines. The work offers a practical baseline with open-source potential and outlines future directions toward deploying high-fidelity diffusion-based avatars on consumer hardware through pruning and quantization.

Abstract

Deploying massive diffusion models for real-time, infinite-duration, audio-driven avatar generation presents a significant engineering challenge, primarily due to the conflict between computational load and strict latency constraints. Existing approaches often compromise visual fidelity by enforcing strictly unidirectional attention mechanisms or reducing model capacity. To address this problem, we introduce \textbf{SoulX-LiveTalk}, a 14B-parameter framework optimized for high-fidelity real-time streaming. Diverging from conventional unidirectional paradigms, we use a \textbf{Self-correcting Bidirectional Distillation} strategy that retains bidirectional attention within video chunks. This design preserves critical spatiotemporal correlations, significantly enhancing motion coherence and visual detail. To ensure stability during infinite generation, we incorporate a \textbf{Multi-step Retrospective Self-Correction Mechanism}, enabling the model to autonomously recover from accumulated errors and preventing collapse. Furthermore, we engineered a full-stack inference acceleration suite incorporating hybrid sequence parallelism, Parallel VAE, and kernel-level optimizations. Extensive evaluations confirm that SoulX-LiveTalk is the first 14B-scale system to achieve a \textbf{sub-second start-up latency (0.87s)} while reaching a real-time throughput of \textbf{32 FPS}, setting a new standard for high-fidelity interactive digital human synthesis.

Figures (5)

• Figure 2: Framework Overview. (a) Stage 1: Latency-Aware Spatiotemporal Adaptation, which adapts the model to operate effectively under lower spatial resolutions and shorter temporal frames to meet real-time constraints. (b) Stage 2: Self-Correcting Bidirectional Distillation, where the generator autoregressively synthesizes $k$ chunks conditioned on past motion frames, while the Real and Fake Score networks align data distributions through distillation losses.
• Figure 3: Visual quality comparison on 5-second video generation.Orange boxes highlight static hand poses in Ditto, while blue boxes highlight significant artifacts (e.g., hand distortion, over-exposure) in baselines. In contrast, SoulX-LiveTalk eliminates these artifacts, demonstrating superior structural integrity and detail fidelity.
• Figure 4: Qualitative evaluation of long-term stability. Comparison across $10$s to $1000$s reveals structural collapse in baselines (blue boxes) versus the sustained robustness of SoulX-LiveTalk (orange boxes), which preserves sharp details even after 1000 seconds of continuous generation.
• Figure 5: Qualitative comparison of lip-sync precision on Chinese pronunciations.Yellow dashed boxes indicate lip shape distortions in baseline methods for characters “上”, “突” and, “法” , whereas our method achieves high alignment with the Ground Truth (GT).
• Figure 6: Detailed inference latency breakdown on 8 NVIDIA H800 GPUs.