StreamDiffusionV2: A Streaming System for Dynamic and Interactive Video Generation

TL;DR

<3-5 sentence high-level summary>StreamDiffusionV2 addresses the challenge of real-time, interactive video generation by adapting video diffusion models to streaming constraints through a training-free pipeline. It combines an SLO-aware batching scheduler, adaptive sink/RoPE updates, and a motion-aware denoising strategy with scalable pipeline orchestration and a DiT-block balancing mechanism to achieve near-linear FPS scaling on multi-GPU systems without sacrificing latency guarantees. The system demonstrates sub-second time-to-first-frame and high frame-rate performance across resolutions and model sizes, maintaining temporal coherence through long horizons. These results establish a practical roadmap for production-grade, live-generative video streaming across creator platforms and enterprise deployments.

Abstract

Generative models are reshaping the live-streaming industry by redefining how content is created, styled, and delivered. Previous image-based streaming diffusion models have powered efficient and creative live streaming products but have hit limits on temporal consistency due to the foundation of image-based designs. Recent advances in video diffusion have markedly improved temporal consistency and sampling efficiency for offline generation. However, offline generation systems primarily optimize throughput by batching large workloads. In contrast, live online streaming operates under strict service-level objectives (SLOs): time-to-first-frame must be minimal, and every frame must meet a per-frame deadline with low jitter. Besides, scalable multi-GPU serving for real-time streams remains largely unresolved so far. To address this, we present StreamDiffusionV2, a training-free pipeline for interactive live streaming with video diffusion models. StreamDiffusionV2 integrates an SLO-aware batching scheduler and a block scheduler, together with a sink-token--guided rolling KV cache, a motion-aware noise controller, and other system-level optimizations. Moreover, we introduce a scalable pipeline orchestration that parallelizes the diffusion process across denoising steps and network layers, achieving near-linear FPS scaling without violating latency guarantees. The system scales seamlessly across heterogeneous GPU environments and supports flexible denoising steps (e.g., 1--4), enabling both ultra-low-latency and higher-quality modes. Without TensorRT or quantization, StreamDiffusionV2 renders the first frame within 0.5s and attains 58.28 FPS with a 14B-parameter model and 64.52 FPS with a 1.3B-parameter model on four H100 GPUs, making state-of-the-art generative live streaming practical and accessible--from individual creators to enterprise-scale platforms.

Figures (18)

• Figure 1: Comparison between Batch and Streaming video generation. Different from generating a large batch of video, live-streaming video generation targets at cutting down the "time to the first frame" and producing continuous output with low latency.
• Figure 2: Generation results among various approaches. The examples above are frames picked from transferred videos among different methods, where the frame index is denoted as $T$.
• Figure 3: Generation results of CausVid and ours. The examples above are frames picked from transferred videos generated from CausVid and StreamDiffusionV2. We utilize the 1.3B model to produce the results.
• Figure 4: Roofline analysis of sequence parallelism and our pipeline orchestration. We compare the Sequence Parallelism and Pipeline Parallelism under varying batch sizes in the causal DiT, compared with the bidirectional DiT. The results demonstrate that our approach operates near the knee point of the roofline, effectively avoiding compute under-utilization as seen in the bidirectional DiT and memory bandwidth limitations in Sequence Parallelism. The model is profiled on an NVIDIA H100 SXM GPU with a peak performance of 1,979 TFLOP/s and a knee point at an arithmetic intensity (AI) of 590.75 FLOP/Byte. The token length is 1,536 (a 4-frame chunk at 480P resolution.)
• Figure 5: Communication consumption of various parallelism methods. We measure the communication latency by testing the parallel inference latency and theoretical latency (sequence or block partitioning without communication) on NVlink-connected H100 GPUs.