VISTA: A Test-Time Self-Improving Video Generation Agent

TL;DR

VISTA tackles the problem of prompt sensitivity and misalignment in text-to-video generation by introducing a two-phase, autonomous test-time framework that jointly optimizes visual, audio, and contextual quality. It combines a structured Initialization Phase (prompt planning and binary tournament selection) with a Self-Improvement Phase (MMAC critiques and Deep Thinking Prompting Agent) to iteratively refine the video and its prompting. The approach is evaluated on single- and multi-scene benchmarks, showing up to 60% pairwise win rates against state-of-the-art baselines and 66.4% human preference for VISTA outputs, with comprehensive ablations highlighting the importance of each component. The results suggest VISTA can scale with compute and generalize across models, offering a pathway toward more reliable, user-aligned video synthesis without fine-tuning, albeit with limitations related to evaluator biases and model capabilities.

Abstract

Despite rapid advances in text-to-video synthesis, generated video quality remains critically dependent on precise user prompts. Existing test-time optimization methods, successful in other domains, struggle with the multi-faceted nature of video. In this work, we introduce VISTA (Video Iterative Self-improvemenT Agent), a novel multi-agent system that autonomously improves video generation through refining prompts in an iterative loop. VISTA first decomposes a user idea into a structured temporal plan. After generation, the best video is identified through a robust pairwise tournament. This winning video is then critiqued by a trio of specialized agents focusing on visual, audio, and contextual fidelity. Finally, a reasoning agent synthesizes this feedback to introspectively rewrite and enhance the prompt for the next generation cycle. Experiments on single- and multi-scene video generation scenarios show that while prior methods yield inconsistent gains, VISTA consistently improves video quality and alignment with user intent, achieving up to 60% pairwise win rate against state-of-the-art baselines. Human evaluators concur, preferring VISTA outputs in 66.4% of comparisons.

Figures (15)

• Figure 1: The workflow of our proposed multi-agent framework, VISTA. : MLLM Agent; : Adversarial MLLM Agent; : Video Generation Agent.
• Figure 2: Average win/tie/lose between VISTA and Direct Prompting (DP) on two benchmarks. The individual benchmark results are in \ref{['appdx:additonal-results-gemini']}
• Figure 3: Evaluation results using conventional metrics on single-scene (a) and multi-scene (b) benchmarks. Numerical results are provided in \ref{['tab:meta-results-on-traditional-metrics', 'tab:ourdata-results-on-traditional-metrics']}.
• Figure 4: Summary of human results. The individual annotators' results are in Appx.-\ref{['tab:human-eval-results']}
• Figure 5: Cost analysis. Left: total token consumption, including both input and output tokens per iteration. Right: number of newly sampled videos per iteration. Results are averaged over two datasets. Tokens for video generation are unavailable and thus excluded.