VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perception

TL;DR

This paper introduces Visual Test-Time Scaling (VTTS), a framework that enables multimodal large language models to reason more effectively by iteratively refining visual perception during inference. Central to VTTS is Iterative Perception (ITP), which progressively narrows focus to high-confidence spatio-temporal regions guided by evolving textual predictions, aided by reinforcement learning via Generalized Reward Policy Optimization (GRPO). To train and evaluate this paradigm, the authors present VTTS-80K, a richly annotated dataset with QA, spatio-temporal cues, and thought processes, enabling targeted supervision and iterative reasoning. Across more than 15 benchmarks spanning video understanding, grounded QA, and spatio-temporal tasks, VideoChat-R1.5 demonstrates consistent improvements over strong baselines, validating the effectiveness and generalization of iterative perception in MLLMs.

Abstract

Inducing reasoning in multimodal large language models (MLLMs) is critical for achieving human-level perception and understanding. Existing methods mainly leverage LLM reasoning to analyze parsed visuals, often limited by static perception stages. This paper introduces Visual Test-Time Scaling (VTTS), a novel approach to enhance MLLMs' reasoning via iterative perception during inference. VTTS mimics humans' hierarchical attention by progressively refining focus on high-confidence spatio-temporal regions, guided by updated textual predictions. Specifically, VTTS employs an Iterative Perception (ITP) mechanism, incorporating reinforcement learning with spatio-temporal supervision to optimize reasoning. To support this paradigm, we also present VTTS-80K, a dataset tailored for iterative perception. These designs allows a MLLM to enhance its performance by increasing its perceptual compute. Extensive experiments validate VTTS's effectiveness and generalization across diverse tasks and benchmarks. Our newly introduced Videochat-R1.5 model has achieved remarkable improvements, with an average increase of over 5\%, compared to robust baselines such as Qwen2.5VL-3B and -7B, across more than 15 benchmarks that encompass video conversation, video reasoning, and spatio-temporal perception.

Figures (14)

• Figure 1: Performance comparison with other models on multiple video benchmarks
• Figure 2: Schemes of Test time scaling methods. BoN refers to generating N candidate items and selecting the best one, Vision-aided CoT involves incorporating visual information once into the reasoning process, and ITP entails iteratively generating spatiotemporal clues and selectively adding visual information to the reasoning based on these clues.
• Figure 3: Inference of iterative perception.
• Figure 4: VTTS-80K dataset generation pipeline and data distribution.
• Figure 5: Ablation on perception times.