Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EchoTorrent: Towards Swift, Sustained, and Streaming Multi-Modal Video Generation

Rang Meng, Weipeng Wu, Yingjie Yin, Yuming Li, Chenguang Ma

TL;DR

EchoTorrent addresses the challenge of real-time, streaming, multi-modal video generation by introducing a four-part framework that jointly reduces inference passes and preserves temporal fidelity. It combines Multi-Teacher Training, Adaptive CFG Calibration Distribution Matching Distillation (ACC-DMD), Hybrid Long Tail Forcing, and a VAE Decoder Refiner within a hybrid causal-bidirectional architecture to suppress spatial blurring, temporal drift, and lip-sync degradation. Key contributions include a SFT-then-RL teacher ensemble, spatiotemporal CFG scheduling, tail-frame alignment for long-horizon rollouts, and pixel-domain decoder refinement, yielding few-pass, long-duration, audio-driven avatar generation with sustained identity and audio-visual alignment. Empirical results show EchoTorrent achieving state-of-the-art or competitive performance on quality, synchronization, and efficiency metrics, maintaining steady throughput for extended sequences and across multiple scenarios.

Abstract

Recent multi-modal video generation models have achieved high visual quality, but their prohibitive latency and limited temporal stability hinder real-time deployment. Streaming inference exacerbates these issues, leading to pronounced multimodal degradation, such as spatial blurring, temporal drift, and lip desynchronization, which creates an unresolved efficiency-performance trade-off. To this end, we propose EchoTorrent, a novel schema with a fourfold design: (1) Multi-Teacher Training fine-tunes a pre-trained model on distinct preference domains to obtain specialized domain experts, which sequentially transfer domain-specific knowledge to a student model; (2) Adaptive CFG Calibration (ACC-DMD), which calibrates the audio CFG augmentation errors in DMD via a phased spatiotemporal schedule, eliminating redundant CFG computations and enabling single-pass inference per step; (3) Hybrid Long Tail Forcing, which enforces alignment exclusively on tail frames during long-horizon self-rollout training via a causal-bidirectional hybrid architecture, effectively mitigates spatiotemporal degradation in streaming mode while enhancing fidelity to reference frames; and (4) VAE Decoder Refiner through pixel-domain optimization of the VAE decoder to recover high-frequency details while circumventing latent-space ambiguities. Extensive experiments and analysis demonstrate that EchoTorrent achieves few-pass autoregressive generation with substantially extended temporal consistency, identity preservation, and audio-lip synchronization.

EchoTorrent: Towards Swift, Sustained, and Streaming Multi-Modal Video Generation

TL;DR

EchoTorrent addresses the challenge of real-time, streaming, multi-modal video generation by introducing a four-part framework that jointly reduces inference passes and preserves temporal fidelity. It combines Multi-Teacher Training, Adaptive CFG Calibration Distribution Matching Distillation (ACC-DMD), Hybrid Long Tail Forcing, and a VAE Decoder Refiner within a hybrid causal-bidirectional architecture to suppress spatial blurring, temporal drift, and lip-sync degradation. Key contributions include a SFT-then-RL teacher ensemble, spatiotemporal CFG scheduling, tail-frame alignment for long-horizon rollouts, and pixel-domain decoder refinement, yielding few-pass, long-duration, audio-driven avatar generation with sustained identity and audio-visual alignment. Empirical results show EchoTorrent achieving state-of-the-art or competitive performance on quality, synchronization, and efficiency metrics, maintaining steady throughput for extended sequences and across multiple scenarios.

Abstract

Recent multi-modal video generation models have achieved high visual quality, but their prohibitive latency and limited temporal stability hinder real-time deployment. Streaming inference exacerbates these issues, leading to pronounced multimodal degradation, such as spatial blurring, temporal drift, and lip desynchronization, which creates an unresolved efficiency-performance trade-off. To this end, we propose EchoTorrent, a novel schema with a fourfold design: (1) Multi-Teacher Training fine-tunes a pre-trained model on distinct preference domains to obtain specialized domain experts, which sequentially transfer domain-specific knowledge to a student model; (2) Adaptive CFG Calibration (ACC-DMD), which calibrates the audio CFG augmentation errors in DMD via a phased spatiotemporal schedule, eliminating redundant CFG computations and enabling single-pass inference per step; (3) Hybrid Long Tail Forcing, which enforces alignment exclusively on tail frames during long-horizon self-rollout training via a causal-bidirectional hybrid architecture, effectively mitigates spatiotemporal degradation in streaming mode while enhancing fidelity to reference frames; and (4) VAE Decoder Refiner through pixel-domain optimization of the VAE decoder to recover high-frequency details while circumventing latent-space ambiguities. Extensive experiments and analysis demonstrate that EchoTorrent achieves few-pass autoregressive generation with substantially extended temporal consistency, identity preservation, and audio-lip synchronization.
Paper Structure (18 sections, 5 equations, 4 figures, 2 tables, 2 algorithms)

This paper contains 18 sections, 5 equations, 4 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Distribution Matching Distillation
  4. Autoregressive Video Diffusion Models
  5. Method
  6. Overview
  7. Multi-Teacher Training
  8. ACC-DMD
  9. Hybrid Long Tail Forcing
  10. VAE Decoder Refiner
  11. Experiments
  12. Experimental Settings
  13. Comparison with SOTA Methods
  14. Ablation Studies
  15. Related Work
  16. ...and 3 more sections

Figures (4)

  • Figure 1: EchoTorrent, a hybrid attention architecture with 14B parameters, achieve 4-NEF, streaming, infinite duration, multiple scenarios, and high-quality multi-modal (text, image, audio) driven human video generation.
  • Figure 2: The overall training pipeline of EchoTorrent.
  • Figure 3: Hybrid Long Tail Forcing.
  • Figure 4: Qualitative Results for Long-Horizon Robustness. Visualizations for different duration ranging from 20 seconds to 1000 seconds.