Frame-Level Captions for Long Video Generation with Complex Multi Scenes

TL;DR

Long-form video generation with rich, multi-scene narratives is hampered by drift and insufficient control from global prompts. The authors propose a framework built on frame-level captions, a frame-level cross-attention mechanism, Diffusion Forcing to inject temporal flexibility, and a Parallel Multi-Window Denoising inference strategy to generate long videos in parallel windows. They introduce a scalable frame-level dataset of 700k clips with dense per-frame annotations and adaptive captioning, enabling precise frame-to-text alignment. Evaluations on VBench 2.0 using WanX-2.1-T2V-1.3B show improved instruction following in complex scenes, reduced semantic confusion, and coherent long-form outputs; they also provide resources for community use.

Abstract

Generating long videos that can show complex stories, like movie scenes from scripts, has great promise and offers much more than short clips. However, current methods that use autoregression with diffusion models often struggle because their step-by-step process naturally leads to a serious error accumulation (drift). Also, many existing ways to make long videos focus on single, continuous scenes, making them less useful for stories with many events and changes. This paper introduces a new approach to solve these problems. First, we propose a novel way to annotate datasets at the frame-level, providing detailed text guidance needed for making complex, multi-scene long videos. This detailed guidance works with a Frame-Level Attention Mechanism to make sure text and video match precisely. A key feature is that each part (frame) within these windows can be guided by its own distinct text prompt. Our training uses Diffusion Forcing to provide the model with the ability to handle time flexibly. We tested our approach on difficult VBench 2.0 benchmarks ("Complex Plots" and "Complex Landscapes") based on the WanX2.1-T2V-1.3B model. The results show our method is better at following instructions in complex, changing scenes and creates high-quality long videos. We plan to share our dataset annotation methods and trained models with the research community. Project page: https://zgctroy.github.io/frame-level-captions .

Figures (9)

• Figure 1: Illustration of Semantic Confusion and Error Accumulation in Video Generation. This figure contrasts 30s videos generated by conventional video-level prompt (right) and our proposed frame-level prompt (ten video key frames below), demonstrating improved generation quality of our method in aspects of semantic confusion and error accumulation.
• Figure 2: Overview of the proposed frame-level training method. Frame-Level Cross-Attention links the visual data of each video segment (latent token) directly to its own specific text description.
• Figure 3: Three inference modes. Multiple yellow tokens represent different frame-level prompts. Less gray value represents lower noise.
• Figure 4: Complex Plot Generation. This figure illustrates the impact of different prompting strategies on visual storytelling performance in a complex narrative task based on The Ugly Duckling. The first row shows results generated using diffusion forcing with a single global prompt (video-level), while the second row presents results from our proposed method that combines diffusion forcing with multiple tailored prompts (multi-prompting).
• Figure 5: Comparative Video Generation Outputs. This figure showcases videos generated by different methods: (a) a 5s video using a model finetuned with Diffusion Forcing and video-level prompt. (b) a 30s video using model finetuned with Diffusion Forcing (DF), FIFO, and a video-level prompt. (c) a 30s video using model finetuned with DF, PMWD (Parallel Multi-Window Denoising), and a dynamic prompts (frame-level). Note: Both 30s videos were generated with a fixed 21-latent window (consistent with training conditions). This limits the historical frame context for sliding window-based methods, potentially affecting ID consistency during multi-scene transitions or object occlusions. ID consistency could be enhanced using techniques like FramePack, incorporating additional compressed historical frames, or IP injection methods.