Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Accelerating Video Generation Inference with Sequential-Parallel 3D Positional Encoding Using a Global Time Index

Chao Yuan, Pan Li

TL;DR

This work adapts the Self-Forcing causal autoregressive framework to sequence parallel inference and implements a sequence-parallel variant of the causal rotary position embedding which it refers to as Causal-RoPE SP, which enables localized computation and reduces cross-rank communication in sequence parallel execution.

Abstract

Diffusion Transformer (DiT)-based video generation models inherently suffer from bottlenecks in long video synthesis and real-time inference, which can be attributed to the use of full spatiotemporal attention. Specifically, this mechanism leads to explosive O(N^2) memory consumption and high first-frame latency. To address these issues, we implement system-level inference optimizations for a causal autoregressive video generation pipeline. We adapt the Self-Forcing causal autoregressive framework to sequence parallel inference and implement a sequence-parallel variant of the causal rotary position embedding which we refer to as Causal-RoPE SP. This adaptation enables localized computation and reduces cross-rank communication in sequence parallel execution. In addition, computation and communication pipelines are optimized through operator fusion and RoPE precomputation. Experiments conducted on an eight GPU A800 cluster show that the optimized system achieves comparable generation quality, sub-second first-frame latency, and near real-time inference speed. For generating five second 480P videos, a 1.58x speedup is achieved, thereby providing effective support for real-time interactive applications.

Accelerating Video Generation Inference with Sequential-Parallel 3D Positional Encoding Using a Global Time Index

TL;DR

This work adapts the Self-Forcing causal autoregressive framework to sequence parallel inference and implements a sequence-parallel variant of the causal rotary position embedding which it refers to as Causal-RoPE SP, which enables localized computation and reduces cross-rank communication in sequence parallel execution.

Abstract

Diffusion Transformer (DiT)-based video generation models inherently suffer from bottlenecks in long video synthesis and real-time inference, which can be attributed to the use of full spatiotemporal attention. Specifically, this mechanism leads to explosive O(N^2) memory consumption and high first-frame latency. To address these issues, we implement system-level inference optimizations for a causal autoregressive video generation pipeline. We adapt the Self-Forcing causal autoregressive framework to sequence parallel inference and implement a sequence-parallel variant of the causal rotary position embedding which we refer to as Causal-RoPE SP. This adaptation enables localized computation and reduces cross-rank communication in sequence parallel execution. In addition, computation and communication pipelines are optimized through operator fusion and RoPE precomputation. Experiments conducted on an eight GPU A800 cluster show that the optimized system achieves comparable generation quality, sub-second first-frame latency, and near real-time inference speed. For generating five second 480P videos, a 1.58x speedup is achieved, thereby providing effective support for real-time interactive applications.
Paper Structure (22 sections, 1 equation, 6 figures, 3 algorithms)

This paper contains 22 sections, 1 equation, 6 figures, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Long Video Generation and Diffusion Model Inference
  4. Sequence Parallelism
  5. Positional Encoding
  6. Methodology
  7. Overall Optimization Framework
  8. Sequence Parallel Implementation
  9. Causal-RoPE SP: Positional Encoding for Block-Wise Autoregression
  10. Background: 3D RoPE in Video Generation
  11. Design of Causal-RoPE SP
  12. Feasibility of Local Computation
  13. Overlapped Computation and Communication
  14. Key Optimizations
  15. Performance Comparison
  16. ...and 7 more sections

Figures (6)

  • Figure 1: Overall optimization framework with three key modules.
  • Figure 2: Pipeline comparison. (a) Baseline: sequential AllGather then RoPE. (b) Optimized: local RoPE with fused communication (10.1× faster).
  • Figure 3: Causal-RoPE SP: local computation of global time index via start_frame parameter $s$.
  • Figure 4: Profiling Results
  • Figure 5: Consistent speedup across multiple resolutions and GPUs.
  • ...and 1 more figures