STIV: Scalable Text and Image Conditioned Video Generation

TL;DR

STIV tackles the challenge of scalable, condition-grounded video generation by unifying text-to-video and text-image-to-video in a single diffusion-transformer framework. The method introduces frame replacement for image grounding and joint image-text classifier-free guidance, paired with a progressive training recipe and a Video Data Engine to curate high-quality training data. Extensive ablations across T2I, T2V, and TI2V demonstrate the effectiveness of stability tricks, efficient training, and frame conditioning, achieving state-of-the-art T2V/I2V performance at 512^2 with an 8.7B parameter model. The framework supports diverse downstream tasks including video prediction, frame interpolation, multi-view generation, and long video synthesis, offering a scalable, extensible recipe for future research.

Abstract

The field of video generation has made remarkable advancements, yet there remains a pressing need for a clear, systematic recipe that can guide the development of robust and scalable models. In this work, we present a comprehensive study that systematically explores the interplay of model architectures, training recipes, and data curation strategies, culminating in a simple and scalable text-image-conditioned video generation method, named STIV. Our framework integrates image condition into a Diffusion Transformer (DiT) through frame replacement, while incorporating text conditioning via a joint image-text conditional classifier-free guidance. This design enables STIV to perform both text-to-video (T2V) and text-image-to-video (TI2V) tasks simultaneously. Additionally, STIV can be easily extended to various applications, such as video prediction, frame interpolation, multi-view generation, and long video generation, etc. With comprehensive ablation studies on T2I, T2V, and TI2V, STIV demonstrate strong performance, despite its simple design. An 8.7B model with 512 resolution achieves 83.1 on VBench T2V, surpassing both leading open and closed-source models like CogVideoX-5B, Pika, Kling, and Gen-3. The same-sized model also achieves a state-of-the-art result of 90.1 on VBench I2V task at 512 resolution. By providing a transparent and extensible recipe for building cutting-edge video generation models, we aim to empower future research and accelerate progress toward more versatile and reliable video generation solutions.

Figures (26)

• Figure 1: Performance comparison of our Text-to-Video model against both open-source and closed-source state-of-the-art models on VBench huang2024vbench.
• Figure 2: Text-to-Video and Text-Image-to-Video generation samples by T2V and STIV models. The text prompts and first frame image conditions are borrowed from Sora's demos openai2024sora and MovieGenBench polyak2024movie.
• Figure 3: We replace the first frame of the noised video latents with the ground truth latent and randomly drop out the image condition. We use cross attention to incorporate the text embedding, and use QK-norm in multi-head attention, the sandwich-norm in both attention and feedforward, and stateless layernorm after singleton conditions to stabilize the training.
• Figure 4: Progressive training pipeline of the STIV model. The T2I model is first trained to initialize the T2V model, which then initializes the STIV model at both low and high resolutions. Notably, the high-res T2V model is initialized using both the high-res T2I model and the low-res T2V model.
• Figure 5: Ablation study of the STIV model, from the base T2I model to the temporally-aware T2V model, and finally to the image-conditioned TI2V model.