Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Searching Priors Makes Text-to-Video Synthesis Better

Haoran Cheng, Liang Peng, Linxuan Xia, Yuepeng Hu, Hengjia Li, Qinglin Lu, Xiaofei He, Boxi Wu

TL;DR

This paper tackles the realism gap in text-to-video synthesis by shifting from large-scale model training to a two-stage search-based approach that leverages motion priors from real videos. It introduces a video retrieval module that semantically matches prompts to existing text-video pairs, followed by motion extraction, distillation, and fine-tuning of a pre-trained T2V model using the retrieved priors. The method employs two forms of text representation (sentence-level and atomized semantics) and a two-stage matching (coarse filtering and ranking) to select the most relevant reference; keyframes are extracted and grounded to distill motion via a cosine-alignment objective, culminating in final video synthesis from the prompt. Empirical results on a single RTX 4090 show improved temporal coherence and motion realism versus state-of-the-art baselines, with user studies supporting higher perceived quality, while the authors acknowledge retrieval and keyframe extraction limitations and propose future refinements.

Abstract

Significant advancements in video diffusion models have brought substantial progress to the field of text-to-video (T2V) synthesis. However, existing T2V synthesis model struggle to accurately generate complex motion dynamics, leading to a reduction in video realism. One possible solution is to collect massive data and train the model on it, but this would be extremely expensive. To alleviate this problem, in this paper, we reformulate the typical T2V generation process as a search-based generation pipeline. Instead of scaling up the model training, we employ existing videos as the motion prior database. Specifically, we divide T2V generation process into two steps: (i) For a given prompt input, we search existing text-video datasets to find videos with text labels that closely match the prompt motions. We propose a tailored search algorithm that emphasizes object motion features. (ii) Retrieved videos are processed and distilled into motion priors to fine-tune a pre-trained base T2V model, followed by generating desired videos using input prompt. By utilizing the priors gleaned from the searched videos, we enhance the realism of the generated videos' motion. All operations can be finished on a single NVIDIA RTX 4090 GPU. We validate our method against state-of-the-art T2V models across diverse prompt inputs. The code will be public.

Searching Priors Makes Text-to-Video Synthesis Better

TL;DR

This paper tackles the realism gap in text-to-video synthesis by shifting from large-scale model training to a two-stage search-based approach that leverages motion priors from real videos. It introduces a video retrieval module that semantically matches prompts to existing text-video pairs, followed by motion extraction, distillation, and fine-tuning of a pre-trained T2V model using the retrieved priors. The method employs two forms of text representation (sentence-level and atomized semantics) and a two-stage matching (coarse filtering and ranking) to select the most relevant reference; keyframes are extracted and grounded to distill motion via a cosine-alignment objective, culminating in final video synthesis from the prompt. Empirical results on a single RTX 4090 show improved temporal coherence and motion realism versus state-of-the-art baselines, with user studies supporting higher perceived quality, while the authors acknowledge retrieval and keyframe extraction limitations and propose future refinements.

Abstract

Significant advancements in video diffusion models have brought substantial progress to the field of text-to-video (T2V) synthesis. However, existing T2V synthesis model struggle to accurately generate complex motion dynamics, leading to a reduction in video realism. One possible solution is to collect massive data and train the model on it, but this would be extremely expensive. To alleviate this problem, in this paper, we reformulate the typical T2V generation process as a search-based generation pipeline. Instead of scaling up the model training, we employ existing videos as the motion prior database. Specifically, we divide T2V generation process into two steps: (i) For a given prompt input, we search existing text-video datasets to find videos with text labels that closely match the prompt motions. We propose a tailored search algorithm that emphasizes object motion features. (ii) Retrieved videos are processed and distilled into motion priors to fine-tune a pre-trained base T2V model, followed by generating desired videos using input prompt. By utilizing the priors gleaned from the searched videos, we enhance the realism of the generated videos' motion. All operations can be finished on a single NVIDIA RTX 4090 GPU. We validate our method against state-of-the-art T2V models across diverse prompt inputs. The code will be public.
Paper Structure (43 sections, 10 equations, 7 figures, 2 tables)

This paper contains 43 sections, 10 equations, 7 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Text-to-Video Synthesis
  4. Video Editing
  5. Video Motion Customization
  6. Preliminaries
  7. Denoising Diffusion Probabilistic Models (DDPM)
  8. Video Diffusion Model
  9. Methodology
  10. Text Vectorization
  11. Sentence-level Semantic Vector.
  12. Atomized Semantic Vectors.
  13. Semantic Matching
  14. Coarse Filtering.
  15. Ranking.
  16. ...and 28 more sections

Figures (7)

  • Figure 1: Samples produced by the proposed text-to-video method. The highlighted words of prompts are core semantic units extracted by the proposed method, guiding the search for prior knowledge during video generation. Green highlights indicate the motion semantics of the motion, while blue ones indicate the objects directly related to the motion. Please refer to https://hrcheng98.github.io/Search_T2V/ for the best view.
  • Figure 2: Pipeline overview. This pipeline searches for videos with similar motion semantics, and extracts relevant information to fine-tune a pre-trained T2V model for video generation.
  • Figure 3: Qualitative comparison with state-of-the-art models. Compared to Video LDM videoldm and Make-a-Video makeavideo, our method generates more temporally coherent motion. Compared to PyoCo pyoco, ours generates more detailed and realistic frames. Best view in https://hrcheng98.github.io/Search_T2V/.
  • Figure 4: Qualitative comparison with existing T2V models. All the results are generated by the officially released models. Compared to CV (abbr. for CogVideo cogvideo), ZS (abbr. for ZeroScope_v2), and S1 (abbr. for Show-1 show1), our method generates more realistic appearances and more vivid motions. For ADL (abbr. for AnimateDiff-Lightning lin2024animatedifflightning), although it produces more detailed and realistic images, it fails to accurately represent the motion information required by the text (e.g., the dog is barely running, and the little girl's hand is almost static). In contrast, our method generates videos with greater motion range and more dynamic realism. Best view in https://hrcheng98.github.io/Search_T2V/.
  • Figure 5: Comparison of different search and key frame extraction strategies. We display a close-up of the tiger's mouth at the bottom of the image to better compare the "eating" action. The full method obtains the best results. Best view in https://hrcheng98.github.io/Search_T2V/.
  • ...and 2 more figures