The Devil is in the Prompts: Retrieval-Augmented Prompt Optimization for Text-to-Video Generation

TL;DR

Prompt sensitivity poses a major challenge for text-to-video generation. RAPO introduces a Retrieval-Augmented Prompt Optimization framework that combines a relation-graph–driven word augmentation, sentence refactoring via a fine-tuned LLM, an instruction-based rewrite path, and a discriminator to select the best prompt, all trained to resemble training-prompt distributions. Evaluations on LaVie and Latte across VBench, EvalCrafter, and T2V-CompBench show RAPO delivers consistent gains in static visual quality, temporal coherence, and multi-object scene fidelity, outperforming plain prompts and other optimization baselines. The work demonstrates that aligning user-provided prompts with the training-distribution through retrieval and targeted instruction tuning yields robust improvements across diverse T2V benchmarks and models.

Abstract

The evolution of Text-to-video (T2V) generative models, trained on large-scale datasets, has been marked by significant progress. However, the sensitivity of T2V generative models to input prompts highlights the critical role of prompt design in influencing generative outcomes. Prior research has predominantly relied on Large Language Models (LLMs) to align user-provided prompts with the distribution of training prompts, albeit without tailored guidance encompassing prompt vocabulary and sentence structure nuances. To this end, we introduce RAPO, a novel Retrieval-Augmented Prompt Optimization framework. In order to address potential inaccuracies and ambiguous details generated by LLM-generated prompts. RAPO refines the naive prompts through dual optimization branches, selecting the superior prompt for T2V generation. The first branch augments user prompts with diverse modifiers extracted from a learned relational graph, refining them to align with the format of training prompts via a fine-tuned LLM. Conversely, the second branch rewrites the naive prompt using a pre-trained LLM following a well-defined instruction set. Extensive experiments demonstrate that RAPO can effectively enhance both the static and dynamic dimensions of generated videos, demonstrating the significance of prompt optimization for user-provided prompts.

Figures (6)

• Figure 1: Videos generated using LaVie wang2023lavie conditioned on user-provided prompts and optimized prompts from RAPO. The optimized prompts can significantly enhance both the static and dynamic qualities of the generated videos, making them more visually appealing.
• Figure 2: Overview of RAPO. The naive prompt is optimized by two branches respectively. For the first branch, it is enriched by the word augmentation module based on a constructed relation graph and a frozen Large Language Model (LLM). Subsequently, augmented prompt is refactored by a finetuned LLM into a specific format in the sentence refactoring module. For the second branch, the naive prompt is directly rewritten by a frozen LLM. Finally, the prompt selection module selects the better one from two branches' results as input for T2V model.
• Figure 3: The construction of relation graph. Relation graph consists of multiple nodes (scenes acting as core nodes with modifiers connected as sub-nodes). For each prompt in database, LLM extracts scene and related modifiers. Based on whether the extracted scene is already in the graph or not, different methods are used to incorporate the new information into the graph.
• Figure 4: Qualitative comparisons across dynamic and static dimensions. This figure showcases videos generated using LaVie with short prompts, GPT-4 and Open-sora prompt optimizations, and our RAPO method. Videos produced with RAPO exhibit significantly sharper spatial details, smoother temporal transitions, and a closer semantic alignment with the input text.
• Figure 5: Visualization on attention map on multiple objects from different prompts. Adding description of the relative spatial position between objects can improve multi-object generation.