MEVG: Multi-event Video Generation with Text-to-Video Models

TL;DR

MEVG tackles multi-event video generation by repurposing a pre-trained diffusion-based text-to-video model without fine-tuning. It introduces last frame–aware latent initialization and structure-guided sampling to maintain visual and temporal coherence across sequential prompts, and leverages an LLM-based prompt generator to segment complex narratives into single-event prompts. The approach achieves superior temporal coherence and semantic alignment compared with zero-shot baselines, as evidenced by both quantitative metrics and human judgments, and supports conditioning on images and multi-text inputs. This provides a practical, training-free pathway to dynamic, multi-scene videos with coherent transitions and diverse motion patterns.

Abstract

We introduce a novel diffusion-based video generation method, generating a video showing multiple events given multiple individual sentences from the user. Our method does not require a large-scale video dataset since our method uses a pre-trained diffusion-based text-to-video generative model without a fine-tuning process. Specifically, we propose a last frame-aware diffusion process to preserve visual coherence between consecutive videos where each video consists of different events by initializing the latent and simultaneously adjusting noise in the latent to enhance the motion dynamic in a generated video. Furthermore, we find that the iterative update of latent vectors by referring to all the preceding frames maintains the global appearance across the frames in a video clip. To handle dynamic text input for video generation, we utilize a novel prompt generator that transfers course text messages from the user into the multiple optimal prompts for the text-to-video diffusion model. Extensive experiments and user studies show that our proposed method is superior to other video-generative models in terms of temporal coherency of content and semantics. Video examples are available on our project page: https://kuai-lab.github.io/eccv2024mevg.

Figures (20)

• Figure 1: An example of multi-event video generation. MEVG produces impressive output that corresponds to the given prompts and consists of chronologically continuous events.
• Figure 1: This instruction follows the five guidelines to create individual prompts based on a given scenario and the number of prompts by the user.
• Figure 2: MEVG synthesizes the consecutive video clips corresponding to distinct prompts. The overall pipeline comprises two major components: last frame-aware latent initialization and structure-guided sampling. First, in the last frame-aware latent initialization, the pre-trained text-to-video generation model adopts the repeated frame as an input to invert into the initial latent code with two novel techniques: dynamic noise and last frame-aware inversion. Second, structure-guided sampling enforces continuity within a video clip by updating the latent code.
• Figure 2: Analysis on additional cost
• Figure 3: Last Frame-aware Latent Initialization Initial latent code is crucial for maintaining global geometric structure. We apply two techniques performing different roles: (i) dynamic noise tailors flexibility differentially across each frame, and (ii) last frame-aware inversion restricts the model to minimize the divergence of the entire frames from the content of the preceding video clip.