Real-time One-Step Diffusion-based Expressive Portrait Videos Generation

TL;DR

This work tackles the slow inference of diffusion-based portrait videos by introducing OSA-LCM, a one-step latent consistency framework. It comprises two stages: Stage 1 trains an adversarial latent consistency model (Adv-LCM) with a temporal-audio discriminator to achieve high-quality results in few steps, and Stage 2 applies editing fine-tuned training (EFT) to bridge the temporal gap for one-step generation, enabling rolling sampling. The approach yields substantial speedups (over 10x, up to 20x in some setups) while maintaining video fidelity and lip-sync quality comparable to multi-step baselines, outperforming existing open-source portrait video diffusion models. The method demonstrates robustness across reference images, audio types, and even anime data, making real-time expressive portrait video generation practical on high-end hardware. The practical impact is a viable, real-time diffusion-based avatar system suitable for interactive applications and streaming.

Abstract

Latent diffusion models have made great strides in generating expressive portrait videos with accurate lip-sync and natural motion from a single reference image and audio input. However, these models are far from real-time, often requiring many sampling steps that take minutes to generate even one second of video-significantly limiting practical use. We introduce OSA-LCM (One-Step Avatar Latent Consistency Model), paving the way for real-time diffusion-based avatars. Our method achieves comparable video quality to existing methods but requires only one sampling step, making it more than 10x faster. To accomplish this, we propose a novel avatar discriminator design that guides lip-audio consistency and motion expressiveness to enhance video quality in limited sampling steps. Additionally, we employ a second-stage training architecture using an editing fine-tuned method (EFT), transforming video generation into an editing task during training to effectively address the temporal gap challenge in single-step generation. Experiments demonstrate that OSA-LCM outperforms existing open-source portrait video generation models while operating more efficiently with a single sampling step.

Figures (13)

• Figure 1: One-step generation results for our proposed one-step avatar latent consistency models (OSA-LCM). We can generate high-quality and diverse resolutions (512x512, 512x896, 896x512) with a single sampling step.
• Figure 2: Model architecture for our base model and proposed discriminator. The architecture of the base model refers to the EMO.
• Figure 3: Different sampling steps for vanilla LCM trained via only consistency distillation loss.
• Figure 4: Different sampling steps for Adv-LCM and OSA-LCM. Adv-LCM will also generate blurry videos.
• Figure 5: Two stages training of our OSA-LCM. In the second stage, we propose a novel training scheme that uses the noising data of past frames to predict future frames. Both two stages use adversarial loss and consistency loss.