Spotlight: Identifying and Localizing Video Generation Errors Using VLMs

TL;DR

Spotlight introduces a benchmark and dataset for identifying and localizing fine-grained errors in AI-generated videos using vision-language models. It provides 1,604 error annotations across 600 videos from 200 prompts, spanning six error types, with temporal localization and reasoning. The study finds that current VLMs underperform humans in precise localization and reasoning, but inference-time strategies, especially Multi-Agent, can substantially close the gap, approaching large models in performance. This work highlights the need for fine-grained evaluation tools and offers pathways for training reward models and safety detectors for text-to-video systems.

Abstract

Current text-to-video models (T2V) can generate high-quality, temporally coherent, and visually realistic videos. Nonetheless, errors still often occur, and are more nuanced and local compared to the previous generation of T2V models. While current evaluation paradigms assess video models across diverse dimensions, they typically evaluate videos holistically without identifying when specific errors occur or describing their nature. We address this gap by introducing Spotlight, a novel task aimed at localizing and explaining video-generation errors. We generate 600 videos using 200 diverse textual prompts and three state-of-the-art video generators (Veo 3, Seedance, and LTX-2), and annotate over 1600 fine-grained errors across six types, including motion, physics, and prompt adherence. We observe that adherence and physics errors are predominant and persist across longer segments, whereas appearance-disappearance and body pose errors manifest in shorter segments. We then evaluate current VLMs on Spotlight and find that VLMs lag significantly behind humans in error identification and localization in videos. We propose inference-time strategies to probe the limits of current VLMs on our task, improving performance by nearly 2x. Our task paves a way forward to building fine-grained evaluation tools and more sophisticated reward models for video generators.

Figures (13)

• Figure 1: We introduce Spotlight, a benchmark for testing vision-language models on identifying and localizing errors in AI-generated videos. To the best of our knowledge, it is the first dataset with fine-grained, temporally localized and global error annotations. The dataset includes 1,604 localized error instances across 600 videos and six error categories, providing a detailed view of subtle inconsistencies in otherwise realistic generations.
• Figure 2: Analysis of T2V models on Spotlight data.
• Figure 3: Example of Pairwise Matching. After filling in matrix $F$, we compute the best match pairs. $M$ and $M^\%$ are computed subsequently.
• Figure 4: Our Baseline Methods. We experiment with three approaches to perform our error reasoning and localization task.
• Figure 5: Comparing # of errors detected by humans and models. Comparing against ground truth data in Spotlight, we observe that both models produce different error distributions for the three T2V generators.