VideoAuteur: Towards Long Narrative Video Generation

TL;DR

The paper tackles long-form narrative video generation by introducing CookGen, a large-scale cooking video dataset, and VideoAuteur, a pipeline that interleaves an auto-regressive director with rolling-context keyframe rendering and a visual-conditioned video generator. The approach jointly generates actions, captions, and visual embeddings to maintain semantic and visual coherence over long sequences, using diffusion-based rendering and robust conditioning to handle noise. Extensive experiments and ablations demonstrate state-of-the-art performance in long-narrative video generation, with clear improvements in visual fidelity, narrative consistency, and evaluation metrics. The work provides an open dataset and a scalable, multi-stage framework to advance long-form narrative synthesis in video.

Abstract

Recent video generation models have shown promising results in producing high-quality video clips lasting several seconds. However, these models face challenges in generating long sequences that convey clear and informative events, limiting their ability to support coherent narrations. In this paper, we present a large-scale cooking video dataset designed to advance long-form narrative generation in the cooking domain. We validate the quality of our proposed dataset in terms of visual fidelity and textual caption accuracy using state-of-the-art Vision-Language Models (VLMs) and video generation models, respectively. We further introduce a Long Narrative Video Director to enhance both visual and semantic coherence in generated videos and emphasize the role of aligning visual embeddings to achieve improved overall video quality. Our method demonstrates substantial improvements in generating visually detailed and semantically aligned keyframes, supported by finetuning techniques that integrate text and image embeddings within the video generation process. Project page: https://videoauteur.github.io/

Figures (23)

• Figure 1: Long Narrative Video Generation. We curate a large-scale cooking video dataset to develop an interleaved auto-regressive model -- VideoAuteur, which acts as a narrative director, sequentially generating actions, captions, and keyframes (two generated examples here). These elements condition a video generation model to create long narrative videos.
• Figure 2: CookGen contains long narrative videos annotated with actions and captions. Each source video is cut into clips and matched with the labeled "actions". We use refined pseudo labels from ASR for Howto100M videos and use manual annotations for Youcook2 videos. We use state-of-the-art VLMs (i.e. GPT-4o and an expert captioner) to provide high-quality captions for all video clips.
• Figure 3: Long Narrative Visual Condition Generation. (a) Interleaved Auto-regressive Director: an auto-regressive vison-language model, takes a user query (e.g., "How to cook a tuna sandwich?") and an initial image-text pair as input. It then generates actions, captions, and visual states (i.e., visual embeddings) step-by-step. (b) Rolling Context Conditioned Render: Apart from the semantics consistency through interleaved generation, we use a rolling of reference images as direct context conditions to further improve visual consistency with a diffusion transformer model. With them, a long narrative video can be created using these generated visual conditions (i.e., visual embeddings and/or keyframes derived from the interleaved director and the keyframe render with rolling context conditioning.)
• Figure 4: Visual-conditioned video generation. Our interleaved auto-regressive director and rolling context renderer generates both text and visual conditions, enabling the video generation process to be conditioned on keyframes (VAE embeddings) and CLIP latents. We apply Gaussian noise, random masking and random shuffling as regularization during the training process to improve robustness with the imperfect visual embeddings.
• Figure 5: Rolling Context Conditioned Render. We integrate tiled global captions, predicted visual embeddings, and a rolling context of previous keyframes to render new keyframes throughout the narrative. By combining semantic conditioning from textual captions and CLIP embeddings with detailed information from VAE embeddings, the diffusion transformer maintains consistency in visual details such as clothing, food details, and character identities. Generated frames are highlighted with red edges.