RISE-T2V: Rephrasing and Injecting Semantics with LLM for Expansive Text-to-Video Generation

TL;DR

RISE-T2V tackles the semantic gap in text-to-video diffusion by unifying prompt rephrasing and semantic feature extraction into a single step. The Rephrasing Adapter leverages next-token hidden states from an LLM to condition video diffusion models, enabling richer, user-aligned prompts without extra encoders. The framework supports dense, multi-scene, and multilingual text encodings, and is validated through extensive quantitative and qualitative experiments across multiple LLMs and diffusion architectures. The results demonstrate improved aesthetics, motion, and text alignment, highlighting the practical impact of integrating LLM-driven rephrasing with diffusion-based video generation.

Abstract

Most text-to-video(T2V) diffusion models depend on pre-trained text encoders for semantic alignment, yet they often fail to maintain video quality when provided with concise prompts rather than well-designed ones. The primary issue lies in their limited textual semantics understanding. Moreover, these text encoders cannot rephrase prompts online to better align with user intentions, which limits both the scalability and usability of the models, To address these challenges, we introduce RISE-T2V, which uniquely integrates the processes of prompt rephrasing and semantic feature extraction into a single and seamless step instead of two separate steps. RISE-T2V is universal and can be applied to various pre-trained LLMs and video diffusion models(VDMs), significantly enhancing their capabilities for T2V tasks. We propose an innovative module called the Rephrasing Adapter, enabling diffusion models to utilize text hidden states during the next token prediction of the LLM as a condition for video generation. By employing a Rephrasing Adapter, the video generation model can implicitly rephrase basic prompts into more comprehensive representations that better match the user's intent. Furthermore, we leverage the powerful capabilities of LLMs to enable video generation models to accomplish a broader range of T2V tasks. Extensive experiments demonstrate that RISE-T2V is a versatile framework applicable to different video diffusion model architectures, significantly enhancing the ability of T2V models to generate high-quality videos that align with user intent. Visual results are available on the webpage at https://rise-t2v.github.io.

Figures (16)

• Figure 1: High-level pipeline of our method. (a). Directly using CLIP/T5 as a feature extractor. (b). Using LLM as prompt rewriter and feeding text into CLIP/T5 for feature extraction. (c). Our method combines prompt rephrasing with semantic feature extraction in a seamless process. The proposed Rephrasing Adapter bridges the gap between LLM-rephrasing feature and pre-trained diffusion model.
• Figure 2: Overview. (a). The inference pipeline of RISE-T2V. The Rephrasing Adapter can integrate with various LLMs and diffusion models. It enables diffusion models to utilize the text hidden states from the LLM's next token prediction, serving as a condition for video generation. (b).The training scheme for RISE-T2V. In stage 1, we train the RA to adapt the rephrased text encodings to the diffusion model. In stage 2, we train the model on videos to achieve motion adaptation.
• Figure 3: Illustration of RA. (a).The training data for the RA is constructed by combining the user instruction $y_{inst}$, original text prompt $y_{ori}$, and enriched text prompt $y_{rephrase}$ are combined into a unified input text, encoded by the LLM, and the hidden states $h_{rephrase}^{\prime}$ are extracted for the training. (b).Visual Comparison: Rephrasing Feature vs. Encoded Feature. The lower row image is clearer and more aesthetically pleasing.
• Figure 4:
• Figure 7: Qualitative comparison. Our method can generate videos with high aesthetics and strong semantics alignment.