LoL: Longer than Longer, Scaling Video Generation to Hour

TL;DR

LoL tackles sink-collapse in autoregressive ultra-long video generation by identifying RoPE-driven phase alignment across multiple attention heads as the root cause. It introduces multi-head RoPE jitter to break inter-head homogenization and combines streaming RoPE generation with a 3D causal VAE to enable real-time, infinite-length video generation with minimal quality loss. The approach substantially mitigates sink-collapse while preserving motion dynamics, enabling long-duration demonstrations up to 12 hours and outperforming several baselines in stability and fidelity. This work highlights the critical role of position-embedding dynamics in scaling video generation and offers a practical, training-free remedy for ultra-long streaming synthesis.

Abstract

Recent research in long-form video generation has shifted from bidirectional to autoregressive models, yet these methods commonly suffer from error accumulation and a loss of long-term coherence. While attention sink frames have been introduced to mitigate this performance decay, they often induce a critical failure mode we term sink-collapse: the generated content repeatedly reverts to the sink frame, resulting in abrupt scene resets and cyclic motion patterns. Our analysis reveals that sink-collapse originates from an inherent conflict between the periodic structure of Rotary Position Embedding (RoPE) and the multi-head attention mechanisms prevalent in current generative models. To address it, we propose a lightweight, training-free approach that effectively suppresses this behavior by introducing multi-head RoPE jitter that breaks inter-head attention homogenization and mitigates long-horizon collapse. Extensive experiments show that our method successfully alleviates sink-collapse while preserving generation quality. To the best of our knowledge, this work achieves the first demonstration of real-time, streaming, and infinite-length video generation with little quality decay. As an illustration of this robustness, we generate continuous videos up to 12 hours in length, which, to our knowledge, is among the longest publicly demonstrated results in streaming video generation.

Figures (14)

• Figure 1: Streamingly generated ultra long video (12 hours) for prompt "A cinematic third-person shot of a wingsuit flyer racing through a narrow mountain valley. The flyer dives downwards, weaving smoothly between jagged cliffs as snow-capped peaks..".
• Figure 2: The plot of intra-head phase concentration which shows the normalized L2 distance to the sink frames against the mean phase of RoPE embeddings (base 10000) with respect to the sink frames. The results reveal that sink-collapse emerges almost exactly where the phase concentration attains local maxima. Besides sink-collapse events, additional drops in L2 distance also occur around local maxima.
• Figure 3: Visualization of inter-head attention homogenization. The first three latent frames serve as attention sinks, and the last three are being generated. Results are shown for the same DiT layer and diffusion step (KV cache size = 12, with 3 sink tokens) of different frames. The top row shows normal frames while the bottom two rows show two different sink-collapse frames where multiple attention heads simultaneously assign significantly higher weights to sink frames causing abrupt scene change back to these frames.
• Figure 4: Visualization of the results after applying different PE extension methods. Baseline approaches continue to exhibit sink-collapse or diminished motion, whereas our method effectively alleviates sink-collapse and best preserves motion dynamics, as shown in \ref{['tab:pe_performance_overall']}.
• Figure 5: Ablation studies on sink-collapse behavior under frequency and RoPE base modifications.