VideoTetris: Towards Compositional Text-to-Video Generation

TL;DR

VideoTetris presents a training-free Spatio-Temporal Compositional Diffusion framework that enables precise compositional text-to-video generation and robust long-video synthesis. The core approach localizes subobjects via temporal-spatial prompt decomposition and modulates cross-attention to compose scene elements, while an Enhanced Video Data Preprocessing pipeline boosts motion dynamics and prompt semantics. A Reference Frame Attention mechanism regularizes inter-frame consistency, and an auto-regressive pathway with a ControlNet-like backbone enables long videos with progressive prompts. Extensive experiments demonstrate superior compositional fidelity and long-sequence coherence compared with state-of-the-art methods, along with meaningful ablations highlighting the contributions of preprocessing and RFA. The work advances practical compositional T2V generation and provides strong foundations for scalable, high-quality long-video synthesis.

Abstract

Diffusion models have demonstrated great success in text-to-video (T2V) generation. However, existing methods may face challenges when handling complex (long) video generation scenarios that involve multiple objects or dynamic changes in object numbers. To address these limitations, we propose VideoTetris, a novel framework that enables compositional T2V generation. Specifically, we propose spatio-temporal compositional diffusion to precisely follow complex textual semantics by manipulating and composing the attention maps of denoising networks spatially and temporally. Moreover, we propose an enhanced video data preprocessing to enhance the training data regarding motion dynamics and prompt understanding, equipped with a new reference frame attention mechanism to improve the consistency of auto-regressive video generation. Extensive experiments demonstrate that our VideoTetris achieves impressive qualitative and quantitative results in compositional T2V generation. Code is available at: https://github.com/YangLing0818/VideoTetris

Figures (7)

• Figure 1: (a): Comparison in Video Generation with Compositoinal Prompts. (b): Comparision in Long Video Generation with Progressive Compositional Prompts. VideoTetris demonstrates superior performance, exhibiting precise adherence to position information and diverse attributes, consistent scene transitions, and high motion dynamics in long compositional video generation.
• Figure 2: The overall pipeline of VideoTetris. We introduce Spatio-Temporal Compositional module for compositional video generation and Reference Frame Attention for consistency regularization. For longer video generation, a ControlNet zhang2023adding-like branch can be adopted for auto-regressive generation.
• Figure 3: Illustration of Spatio-Temporal Compositional Diffusion. For a given story "A little dolphin starts exploring an old city under the sea, she first found a green turtle at the bottom, then her huge father comes along to accompany her at the right side.", we first decompose it temporally to Text Prompt #1, #2 and #3, then we decompose each of them spatially to compute each sub-region's cross attention maps. Finally, we compose them spatio-temporally to form a natural story.
• Figure 4: Qualitative Results of Video Generation with Compositional Prompts in Comparision with SOTA Text-to-Video Models
• Figure 5: More qualitative results of VideoTetris.