Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SkyReels-Audio: Omni Audio-Conditioned Talking Portraits in Video Diffusion Transformers

Zhengcong Fei, Hao Jiang, Di Qiu, Baoxuan Gu, Youqiang Zhang, Jiahua Wang, Jialin Bai, Debang Li, Mingyuan Fan, Guibin Chen, Yahui Zhou

TL;DR

SkyReels-Audio proposes a unified omni audio-conditioned talking portrait system built on pretrained Video Diffusion Transformers, enabling multilinear conditioning from audio, text, images, and video with infinite-length generation. It introduces a hybrid learning strategy, a facial-region-aware loss, and an audio-guided sampling scheme to improve lip-sync and identity retention. A Bidirectional Latent Fusion mechanism ensures temporal coherence for long videos, while TeaCache and USP accelerate inference. A dedicated data pipeline curates synchronized audio–video–text triplets, and comprehensive experiments show superior lip-sync accuracy, identity consistency, and natural facial dynamics across diverse identities and conditions.

Abstract

The generation and editing of audio-conditioned talking portraits guided by multimodal inputs, including text, images, and videos, remains under explored. In this paper, we present SkyReels-Audio, a unified framework for synthesizing high-fidelity and temporally coherent talking portrait videos. Built upon pretrained video diffusion transformers, our framework supports infinite-length generation and editing, while enabling diverse and controllable conditioning through multimodal inputs. We employ a hybrid curriculum learning strategy to progressively align audio with facial motion, enabling fine-grained multimodal control over long video sequences. To enhance local facial coherence, we introduce a facial mask loss and an audio-guided classifier-free guidance mechanism. A sliding-window denoising approach further fuses latent representations across temporal segments, ensuring visual fidelity and temporal consistency across extended durations and diverse identities. More importantly, we construct a dedicated data pipeline for curating high-quality triplets consisting of synchronized audio, video, and textual descriptions. Comprehensive benchmark evaluations show that SkyReels-Audio achieves superior performance in lip-sync accuracy, identity consistency, and realistic facial dynamics, particularly under complex and challenging conditions.

SkyReels-Audio: Omni Audio-Conditioned Talking Portraits in Video Diffusion Transformers

TL;DR

SkyReels-Audio proposes a unified omni audio-conditioned talking portrait system built on pretrained Video Diffusion Transformers, enabling multilinear conditioning from audio, text, images, and video with infinite-length generation. It introduces a hybrid learning strategy, a facial-region-aware loss, and an audio-guided sampling scheme to improve lip-sync and identity retention. A Bidirectional Latent Fusion mechanism ensures temporal coherence for long videos, while TeaCache and USP accelerate inference. A dedicated data pipeline curates synchronized audio–video–text triplets, and comprehensive experiments show superior lip-sync accuracy, identity consistency, and natural facial dynamics across diverse identities and conditions.

Abstract

The generation and editing of audio-conditioned talking portraits guided by multimodal inputs, including text, images, and videos, remains under explored. In this paper, we present SkyReels-Audio, a unified framework for synthesizing high-fidelity and temporally coherent talking portrait videos. Built upon pretrained video diffusion transformers, our framework supports infinite-length generation and editing, while enabling diverse and controllable conditioning through multimodal inputs. We employ a hybrid curriculum learning strategy to progressively align audio with facial motion, enabling fine-grained multimodal control over long video sequences. To enhance local facial coherence, we introduce a facial mask loss and an audio-guided classifier-free guidance mechanism. A sliding-window denoising approach further fuses latent representations across temporal segments, ensuring visual fidelity and temporal consistency across extended durations and diverse identities. More importantly, we construct a dedicated data pipeline for curating high-quality triplets consisting of synchronized audio, video, and textual descriptions. Comprehensive benchmark evaluations show that SkyReels-Audio achieves superior performance in lip-sync accuracy, identity consistency, and realistic facial dynamics, particularly under complex and challenging conditions.

Paper Structure

This paper contains 31 sections, 5 equations, 8 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Method
  3. Preliminaries
  4. Flow Matching.
  5. Video Diffusion Transformers.
  6. Model Architecture
  7. Hybrid Learning Strategy
  8. Inference Optimization
  9. Audio CFG.
  10. Infinite Video Generation via Bidirectional Latent Fusion.
  11. Hybrid Inference Strategy
  12. Model Acceleration.
  13. Data Pipeline
  14. Experiments
  15. Experimental Settings
  16. ...and 16 more sections

Figures (8)

  • Figure 1: Given a portrait image, text, or video along with audio input, SkyReels-Audio can generate and edit portraits with strong identity consistency, expressive facial and natural body dynamics. In addition, SkyReels-Audio support infinite video generation based on various multi-modal controllable clues.
  • Figure 2: Overview of SkyReels-Audio. Whisper encodes resampled audio and fuse video tokens with cross-attention layers. Image and video controls are joint featured with VAE before combine with input noise to provide a video identity and environment priors.
  • Figure 3: Illustration of BLF. BLF is a tuning-free overlapping sliding window strategy, performing bidirectional fusion of the latents within adjacent windows in the same denoising step.
  • Figure 4: Data Processing Pipeline. This is a data funnel to filter high-quality video data.
  • Figure 5: Qualitative comparisons with other audio-driven talking portrait methods. Our approach produce more accurate lip synchronization with naturalness.
  • ...and 3 more figures