Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Free$^2$Guide: Training-Free Text-to-Video Alignment using Image LVLM

Jaemin Kim, Bryan Sangwoo Kim, Jong Chul Ye

TL;DR

Text-to-video diffusion models struggle to maintain accurate alignment with prompts due to temporal dependencies. The paper introduces Free$^2$Guide, a gradient-free, training-free framework that guides diffusion sampling using non-differentiable rewards via path integral control, enabling LVLM-based feedback without backpropagation. It adopts frame stitching to adapt image-based LVLMs for temporal understanding and proposes ensemble strategies to combine multiple rewards, achieving improved text-video alignment and video quality. Empirical results on LaVie and VideoCrafter2 demonstrate notable gains in alignment and quality, with efficiency and robustness benefits that support scalable use of black-box LVLMs for T2V generation.

Abstract

Diffusion models have achieved impressive results in generative tasks for text-to-video (T2V) synthesis. However, achieving accurate text alignment in T2V generation remains challenging due to the complex temporal dependencies across frames. Existing reinforcement learning (RL)-based approaches to enhance text alignment often require differentiable reward functions trained for videos, hindering their scalability and applicability. In this paper, we propose \textbf{Free$^2$Guide}, a novel gradient-free and training-free framework for aligning generated videos with text prompts. Specifically, leveraging principles from path integral control, Free$^2$Guide approximates guidance for diffusion models using non-differentiable reward functions, thereby enabling the integration of powerful black-box Large Vision-Language Models (LVLMs) as reward models. To enable image-trained LVLMs to assess text-to-video alignment, we leverage \textit{stitching} between video frames and use system prompts to capture sequential attributions. Our framework supports the flexible ensembling of multiple reward models to synergistically enhance alignment without significant computational overhead. Experimental results confirm that Free$^2$Guide using image-trained LVLMs significantly improves text-to-video alignment, thereby enhancing the overall video quality. Our results and code are available at https://kjm981995.github.io/free2guide/

Free$^2$Guide: Training-Free Text-to-Video Alignment using Image LVLM

TL;DR

Text-to-video diffusion models struggle to maintain accurate alignment with prompts due to temporal dependencies. The paper introduces FreeGuide, a gradient-free, training-free framework that guides diffusion sampling using non-differentiable rewards via path integral control, enabling LVLM-based feedback without backpropagation. It adopts frame stitching to adapt image-based LVLMs for temporal understanding and proposes ensemble strategies to combine multiple rewards, achieving improved text-video alignment and video quality. Empirical results on LaVie and VideoCrafter2 demonstrate notable gains in alignment and quality, with efficiency and robustness benefits that support scalable use of black-box LVLMs for T2V generation.

Abstract

Diffusion models have achieved impressive results in generative tasks for text-to-video (T2V) synthesis. However, achieving accurate text alignment in T2V generation remains challenging due to the complex temporal dependencies across frames. Existing reinforcement learning (RL)-based approaches to enhance text alignment often require differentiable reward functions trained for videos, hindering their scalability and applicability. In this paper, we propose \textbf{FreeGuide}, a novel gradient-free and training-free framework for aligning generated videos with text prompts. Specifically, leveraging principles from path integral control, FreeGuide approximates guidance for diffusion models using non-differentiable reward functions, thereby enabling the integration of powerful black-box Large Vision-Language Models (LVLMs) as reward models. To enable image-trained LVLMs to assess text-to-video alignment, we leverage \textit{stitching} between video frames and use system prompts to capture sequential attributions. Our framework supports the flexible ensembling of multiple reward models to synergistically enhance alignment without significant computational overhead. Experimental results confirm that FreeGuide using image-trained LVLMs significantly improves text-to-video alignment, thereby enhancing the overall video quality. Our results and code are available at https://kjm981995.github.io/free2guide/

Paper Structure

This paper contains 34 sections, 11 equations, 11 figures, 13 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Text-to-Video diffusion model
  4. Preliminaries
  5. Video Latent Diffusion Model
  6. Guidance in Diffusion Model
  7. Path Integral Control
  8. Free$^2$Guide
  9. Video Guidance leveraging Image LVLMs
  10. Image-based LVLMs as a Video Reward Model
  11. Ensembling Reward Functions
  12. Guidance using Path Integral Control
  13. Experiments
  14. Results
  15. Analysis
  16. ...and 19 more sections

Figures (11)

  • Figure 1: Representative video results using Free$^2$Guide, a novel framework that enables training-Free, gradient-Free video Guidance leveraging a Large Vision-Language Model. Each image shows the first frame of a video.
  • Figure 2: Overall pipeline of training-free gradient-free Free$^2$Guide. Free$^2$Guide leverages LVLMs' ability to comprehend stitched images, utilizing this capability to enhance frame-to-frame dynamic understanding and applying it within the video domain to improve text-video alignment. It also enables an effective ensemble approach that integrates large-scale image-based models to improve video generation guidance.
  • Figure 3: Qualitative results of our method. Comparison with LaVie on the left and VideoCrafter2 on the right.
  • Figure 4: Results for T2V-CompBench.
  • Figure 5: Fixed NFE comparison on VBench.
  • ...and 6 more figures