MicroCinema: A Divide-and-Conquer Approach for Text-to-Video Generation

TL;DR

MicroCinema tackles text-to-video generation by splitting the task into a center-frame image synthesis stage and a motion-focused image&text-to-video stage. It introduces the Appearance Injection Network and Appearance Noise Prior to preserve appearance from the center frame while enabling robust motion modeling, allowing seamless integration of strong 2D T2I models like Stable Diffusion. Empirical results on UCF-101 and MSR-VTT show state-of-the-art zero-shot FVD when trained on WebVid-10M, demonstrating superior temporal coherence and image quality without requiring large-scale video data. The approach offers a flexible, controllable pipeline that can accommodate various T2I backbones and paves the way for more efficient, high-quality text-driven video generation.

Abstract

We present MicroCinema, a straightforward yet effective framework for high-quality and coherent text-to-video generation. Unlike existing approaches that align text prompts with video directly, MicroCinema introduces a Divide-and-Conquer strategy which divides the text-to-video into a two-stage process: text-to-image generation and image\&text-to-video generation. This strategy offers two significant advantages. a) It allows us to take full advantage of the recent advances in text-to-image models, such as Stable Diffusion, Midjourney, and DALLE, to generate photorealistic and highly detailed images. b) Leveraging the generated image, the model can allocate less focus to fine-grained appearance details, prioritizing the efficient learning of motion dynamics. To implement this strategy effectively, we introduce two core designs. First, we propose the Appearance Injection Network, enhancing the preservation of the appearance of the given image. Second, we introduce the Appearance Noise Prior, a novel mechanism aimed at maintaining the capabilities of pre-trained 2D diffusion models. These design elements empower MicroCinema to generate high-quality videos with precise motion, guided by the provided text prompts. Extensive experiments demonstrate the superiority of the proposed framework. Concretely, MicroCinema achieves SOTA zero-shot FVD of 342.86 on UCF-101 and 377.40 on MSR-VTT. See https://wangyanhui666.github.io/MicroCinema.github.io/ for video samples.

Figures (16)

• Figure 1: Sample videos produced by MicroCinema, our proposed text-to-video generation system. They showcase MicroCinema's ability to create coherent and high-quality videos, with precise motion aligned with text prompts. Image reference generated by Midjourney.
• Figure 2: Overall architecture of our proposed diffusion-based image&text-to-video model in MicroCinema. The proposed AppearNet aims to provide appearance information for video generation.
• Figure 3: AppearNet injects multi-scale features into the main branch to perform a dense fusion.
• Figure 4: Comparison with Make-A-Video and Video LDM. Reference images generated by DALL-E 2 (top) and Midjourney (bottom). The generated videos from our model shows a clear and coherent motion.
• Figure 5: UCF-101 Zero shot FVD across different $\lambda$ and $\gamma$.