VSTAR: Generative Temporal Nursing for Longer Dynamic Video Synthesis

TL;DR

VSTAR tackles the challenge of producing long, dynamically evolving videos from single prompts by introducing Generative Temporal Nursing (GTN) and two inference-time techniques: Video Synopsis Prompting (VSP) and Temporal Attention Regularization (TAR). VSP uses large language models to decompose prompts into sequence-based visual states, while TAR regularizes temporal attention maps to resemble those of real dynamic videos, all without retraining or added inference cost. The approach yields longer, more visually appealing videos than existing open-source T2V models and is supported by a temporal attention analysis that offers design insights for future long-video diffusion models. Overall, VSTAR demonstrates a practical, training-free pathway to enhance temporal dynamics in T2V synthesis with strong potential for guiding future model development and training strategies.

Abstract

Despite tremendous progress in the field of text-to-video (T2V) synthesis, open-sourced T2V diffusion models struggle to generate longer videos with dynamically varying and evolving content. They tend to synthesize quasi-static videos, ignoring the necessary visual change-over-time implied in the text prompt. At the same time, scaling these models to enable longer, more dynamic video synthesis often remains computationally intractable. To address this challenge, we introduce the concept of Generative Temporal Nursing (GTN), where we aim to alter the generative process on the fly during inference to improve control over the temporal dynamics and enable generation of longer videos. We propose a method for GTN, dubbed VSTAR, which consists of two key ingredients: 1) Video Synopsis Prompting (VSP) - automatic generation of a video synopsis based on the original single prompt leveraging LLMs, which gives accurate textual guidance to different visual states of longer videos, and 2) Temporal Attention Regularization (TAR) - a regularization technique to refine the temporal attention units of the pre-trained T2V diffusion models, which enables control over the video dynamics. We experimentally showcase the superiority of the proposed approach in generating longer, visually appealing videos over existing open-sourced T2V models. We additionally analyze the temporal attention maps realized with and without VSTAR, demonstrating the importance of applying our method to mitigate neglect of the desired visual change over time.

Figures (23)

• Figure 1: Our VSTAR can generate a 64-frame video with dynamic visual evolution in a single pass. Images are subsampled from the video. Note that the first column is a GIF, best viewed in Acrobat Reader.
• Figure 2: Method overview. Our VSTAR consists of two strategies: Video Synopsis Prompting (left) and Temporal Attention Regularization (right).
• Figure 3: An illustration example of VSP. With the aid of LLMs, we can obtain more descriptive video synopsis for key stages.
• Figure 4: Temporal attention visualization of real and synthetic videos of 16 and 48 frames. Attention of real videos exhibits a band-matrix like structure, indicating high correlation with adjacent frames. Synthetic videos exhibit less-structured attention maps, especially for 48 frames, which explains the low quality of long video generation.
• Figure 5: Per-layer temporal attention analysis. We replace the temporal attention maps at different resolutions with a diagonal matrix (1st row) and an all-ones matrix (2nd row), which leads to a more dynamic or a more static video, respectively. We observe that high resolution attention has a larger impact on the video dynamics. Note that this is a GIF, best viewed in Acrobat Reader.