Video-Infinity: Distributed Long Video Generation

TL;DR

This work tackles the memory/time bottlenecks of long-form video generation with diffusion models by introducing Video-Infinity, a distributed inference pipeline. It relies on Clip parallelism to split the video latent into clips processed in parallel and Dual-scope attention to balance local and global temporal contexts without extra training. The approach achieves substantial scalability, generating up to 2300 frames in about 5 minutes on 8× 48GB GPUs and delivering up to 100× faster long-video generation than prior methods. This demonstrates a practical path toward efficient, end-to-end long-form video synthesis on multi-GPU systems with improved coherence across extended sequences.

Abstract

Diffusion models have recently achieved remarkable results for video generation. Despite the encouraging performances, the generated videos are typically constrained to a small number of frames, resulting in clips lasting merely a few seconds. The primary challenges in producing longer videos include the substantial memory requirements and the extended processing time required on a single GPU. A straightforward solution would be to split the workload across multiple GPUs, which, however, leads to two issues: (1) ensuring all GPUs communicate effectively to share timing and context information, and (2) modifying existing video diffusion models, which are usually trained on short sequences, to create longer videos without additional training. To tackle these, in this paper we introduce Video-Infinity, a distributed inference pipeline that enables parallel processing across multiple GPUs for long-form video generation. Specifically, we propose two coherent mechanisms: Clip parallelism and Dual-scope attention. Clip parallelism optimizes the gathering and sharing of context information across GPUs which minimizes communication overhead, while Dual-scope attention modulates the temporal self-attention to balance local and global contexts efficiently across the devices. Together, the two mechanisms join forces to distribute the workload and enable the fast generation of long videos. Under an 8 x Nvidia 6000 Ada GPU (48G) setup, our method generates videos up to 2,300 frames in approximately 5 minutes, enabling long video generation at a speed 100 times faster than the prior methods.

Figures (5)

• Figure 1: Multiple GPUs parallelly generate a complete video, producing 2300 frames in 5 minutes.
• Figure 2: (a) Pipeline of Video-Infinity: The latent tensor is split into clips and distributed to different devices. The diffusion model predicts noise in parallel with communication, and the noises are concatenated to produce the final output. (b) Illustration of Clip parallelism: In each layer of the video diffusion module, spatial modules operate independently, whereas temporal modules synchronize context elements $c^i_{\text{pre}}$, $c^i_{\text{post}}$, and $c^i_{\text{global}}$. Peer-to-peer and collaborative communications are employed.
• Figure 3: Tree different stages in the communication process of Clip parallelism
• Figure 4: Comparison of frame images from sample videos generated by different methods.
• Figure 5: Visualization of ablation studies on temporal module communication and context effects in video generation. Top panel: Ablation of communication between the ResLayer module and the Attention module, showcasing two adjacent frames from the video sequence generated on different GPUs. Bottom panel: Effects of ablating different contexts within the Attention module, displaying frames from videos generated post-ablation.