Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Search

TL;DR

The paper tackles misalignment in text-to-video diffusion by introducing DLBS, an inference-time search over diffusion latents, paired with a lookahead estimator to stabilize reward evaluation. It argues that perceptual video quality requires reward calibration, combining multiple metrics to better align with AI and human preferences. Empirically, DLBS with calibrated rewards improves perceptual quality across multiple base models and larger SoTA models, while remaining computationally efficient and compatible with finetuning methods. The work offers practical guidance: allocate more compute to lookahead and search budget rather than simply increasing denoising steps, boosting both quality and diversity in generated videos.

Abstract

The remarkable progress in text-to-video diffusion models enables the generation of photorealistic videos, although the content of these generated videos often includes unnatural movement or deformation, reverse playback, and motionless scenes. Recently, an alignment problem has attracted huge attention, where we steer the output of diffusion models based on some measure of the content's goodness. Because there is a large room for improvement of perceptual quality along the frame direction, we should address which metrics we should optimize and how we can optimize them in the video generation. In this paper, we propose diffusion latent beam search with lookahead estimator, which can select a better diffusion latent to maximize a given alignment reward at inference time. We then point out that improving perceptual video quality with respect to alignment to prompts requires reward calibration by weighting existing metrics. This is because when humans or vision language models evaluate outputs, many previous metrics to quantify the naturalness of video do not always correlate with the evaluation. We demonstrate that our method improves the perceptual quality evaluated on the calibrated reward, VLMs, and human assessment, without model parameter update, and outputs the best generation compared to greedy search and best-of-N sampling under much more efficient computational cost. The experiments highlight that our method is beneficial to many capable generative models, and provide a practical guideline: we should prioritize the inference-time compute allocation into enabling the lookahead estimator and increasing the search budget, rather than expanding the denoising steps.

Figures (40)

• Figure 1: (Left) Diffusion latent beam search (DLBS) seeks a better diffusion path over the reverse process; sampling $K$ latents per beam and possessing $B$ beams for the next step, which mitigates the effect from inaccurate $\operatorname{argmax}$. Lookahead (LA) estimator notably reduces the noise at latent reward evaluation by interpolating the rest of the time steps from the current latent with deterministic DDIM. (Right) DLBS achieves much better computational-efficiency than best-of-N (BoN), as achieving higher performance gains under the same execution time. LA estimator (DLBS-LA) could remarkably boost efficiency only with marginal overhead on top of DLBS.
• Figure 1: Performance of DLBS with DPO finetuned VideoCrafter2 on DEVIL-high and MSRVTT-test datasets. While DPO alone yields marginal improvements, combining it with DLBS leads to notable gains, demonstrating the compatibility of inference-time search with fine-tuning approaches.
• Figure 2: Comparison of text-to-video results between DLBS-LA, base models, and other sampling methods on SoTA models (Latte ma2024latte, CogVideoX yang2024cogvideox, and Wan 2.1 wan2025). DLBS-LA produces more dynamic, natural, and prompt-aligned videos than all baselines.
• Figure 3: 2D-histogram and correlation between reward functions for perceptual video quality huang2023vbench and AI feedback from Gemini geminiteam2023gemini. A single reward (e.g., subject consistency; blue) is often not aligned well with a preference from Gemini, which happens for all the prompt sets with different dynamics grades (see \ref{['fig:hist_gemini_full']}). The calibrated reward, a linear combination of perceptual metrics via brute-force search (green), achieves the best Pearson correlation coefficient in all settings (statistically significant with $p<0.01$).
• Figure 4: The coefficient of calibrated reward $w_i$ with feedback from Gemini. Each set of prompts, which has a different dynamics grade, requires a distinct mixture of perceptual video qualities.