RAPO++: Cross-Stage Prompt Optimization for Text-to-Video Generation via Data Alignment and Test-Time Scaling

TL;DR

RAPO++ tackles prompt misalignment in text-to-video generation by integrating training-data-aligned prompt refinement (RAPO), test-time iterative scaling (SSPO), and LLM fine-tuning. The three-stage RAPO pipeline—Word Augmentation, Sentence Refactoring, and Prompt Selection—aligns prompts with training distributions, while SSPO enables reflection-driven, non-backbone modifications during inference. Fine-tuning the rewrite LLM with Stage-2 prompts further boosts generalization across models. Across five T2V models and five benchmarks, RAPO++ achieves significant improvements in semantic alignment, compositional reasoning, temporal stability, and physical plausibility, positioning prompt optimization as a scalable, model-agnostic approach for high-quality T2V generation.

Abstract

Prompt design plays a crucial role in text-to-video (T2V) generation, yet user-provided prompts are often short, unstructured, and misaligned with training data, limiting the generative potential of diffusion-based T2V models. We present \textbf{RAPO++}, a cross-stage prompt optimization framework that unifies training-data--aligned refinement, test-time iterative scaling, and large language model (LLM) fine-tuning to substantially improve T2V generation without modifying the underlying generative backbone. In \textbf{Stage 1}, Retrieval-Augmented Prompt Optimization (RAPO) enriches user prompts with semantically relevant modifiers retrieved from a relation graph and refactors them to match training distributions, enhancing compositionality and multi-object fidelity. \textbf{Stage 2} introduces Sample-Specific Prompt Optimization (SSPO), a closed-loop mechanism that iteratively refines prompts using multi-source feedback -- including semantic alignment, spatial fidelity, temporal coherence, and task-specific signals such as optical flow -- yielding progressively improved video generation quality. \textbf{Stage 3} leverages optimized prompt pairs from SSPO to fine-tune the rewriter LLM, internalizing task-specific optimization patterns and enabling efficient, high-quality prompt generation even before inference. Extensive experiments across five state-of-the-art T2V models and five benchmarks demonstrate that RAPO++ achieves significant gains in semantic alignment, compositional reasoning, temporal stability, and physical plausibility, outperforming existing methods by large margins. Our results highlight RAPO++ as a model-agnostic, cost-efficient, and scalable solution that sets a new standard for prompt optimization in T2V generation. The code is available at https://github.com/Vchitect/RAPO.

Figures (10)

• Figure 1: Overview of RAPO++. The framework couples training-data–aligned prompt refinement with test-time scaling to enhance Text-to-Video (T2V) generation without altering the generative backbone. Stage 1 RAPO: Retrieval-Augmented Prompt Optimization (Sec. \ref{['sec:RAPO']}). User prompts are augmented via a retrieval-based relation graph and refactored by a fine-tuned LLM, while a frozen LLM provides alternative rewrites. A discriminator then selects the best candidate, ensuring prompts align with training distributions while preserving intent. Stage 2 SSPO: Sample-Specific Prompt Optimization at Test-time (Sec. \ref{['sec:RAPO_plus']}). Multiple candidates are evaluated by VLM verifiers and task-specific metrics, with misalignments guiding iterative refinement. This process enhances temporal coherence, fidelity, and semantic alignment during inference, and also yields prompt pairs for LLM fine-tuning. Stage 3: LLM Fine-Tuning & Evaluation (Sec. \ref{['sec:LLM_Fine-Tuning']}& \ref{['sec:experiments']}). The prompt pairs collected from Stage 2 are used to fine-tune the LLM, further enhancing its generalization and robustness across models. The fine-tuned LLM is then validated across different benchmarks, demonstrating consistent and transferable improvements in T2V generation.
• Figure 2: Generation results under different iterations of prompt refinement at inference utilizing SSPO. The initial prompt is "valkyrie riding flying horses through the clouds". As the number of iterations increases (from left to right), the generated video becomes more detailed and vivid, and more consistent with the user's intent.
• Figure 3: The construction of relation graph. Relation graph consists of multiple nodes (scenes acting as core nodes with modifiers connected as sub-nodes). For each prompt in database, LLM extracts scene and related modifiers. Based on whether the extracted scene is already in the graph or not, different methods are used to incorporate the new information into the graph.
• Figure 4: Qualitative comparisons across dynamic and static dimensions. This figure showcases videos generated using LaVie with short prompts, GPT-4 and Open-sora prompt optimizations, and our RAPO method. Videos produced with RAPO exhibit significantly sharper spatial details, smoother temporal transitions, and a closer semantic alignment with the input text.
• Figure 5: Qualitative comparisons using LaVie with initial prompts (left) and optimized prompts from RAPO++ (right). We present qualitative comparisons from the dynamic and static dimension. The videos generated by RAPO++ exhibit sharper details, smoother temporal transitions, and better alignment with the input text.