VIPER: Process-aware Evaluation for Generative Video Reasoning

TL;DR

VIPER introduces a process-aware evaluation framework for Generative Video Reasoning by formalizing Process-outcome Consistency (POC@r) and deploying a hierarchical VLM-based rubric across 16 tasks in 6 domains. The benchmark reveals substantial outcome-hacking in state-of-the-art video models, with POC@1.0 scores often below 30% and 20–40% hacking rates, underscoring a gap between final answers and reasoning processes. Test-time scaling and process constraint incorporation offer some performance gains, but the gains are limited, highlighting the need for stronger process-level reasoning and evaluation. Public release of VIPER and the rubric aims to catalyze progress toward truly generalizable visual reasoning in video models.

Abstract

Recent breakthroughs in video generation have demonstrated an emerging capability termed Chain-of-Frames (CoF) reasoning, where models resolve complex tasks through the generation of continuous frames. While these models show promise for Generative Video Reasoning (GVR), existing evaluation frameworks often rely on single-frame assessments, which can lead to outcome-hacking, where a model reaches a correct conclusion through an erroneous process. To address this, we propose a process-aware evaluation paradigm. We introduce VIPER, a comprehensive benchmark spanning 16 tasks across temporal, structural, symbolic, spatial, physics, and planning reasoning. Furthermore, we propose Process-outcome Consistency (POC@r), a new metric that utilizes VLM-as-Judge with a hierarchical rubric to evaluate both the validity of the intermediate steps and the final result. Our experiments reveal that state-of-the-art video models achieve only about 20% POC@1.0 and exhibit a significant outcome-hacking. We further explore the impact of test-time scaling and sampling robustness, highlighting a substantial gap between current video generation and true generalized visual reasoning. Our benchmark will be publicly released.

Figures (21)

• Figure 1: POC@1.0 performance overview of representative video model on VIPER across 6 domains.
• Figure 2: Illustration of reasoning via Chain-of-Frames. As shown, while models are capable of reaching the correct answer, they often generate erroneous intermediate frames (e.g., "Extra book" or "Wrong Path").
• Figure 3: Overview of VIPER. VIPER consists of 16 tasks from 6 domains focusing on different reasoning abilites.
• Figure 4: Human-model alignment under of combinations of raw input and hierarchical rubric.
• Figure 5: Representative failure patterns selected from VIPER.