RVTBench: A Benchmark for Visual Reasoning Tasks

TL;DR

RVTs unify vision-language reasoning across outputs such as segmentation, grounding, summaries, and VQA by formalizing a reasoning process $\mathcal{R}$ that maps input video $\mathcal{X}$ and implicit query $\mathcal{Q}$ to task-specific outputs. The authors introduce RVTBench, an automated benchmark built on digital twin representations that decouples perception from reasoning, enabling controlled, multi-level reasoning across semantic, spatial, and temporal dimensions with 3,896 queries derived from 200 videos. A zero-shot baseline, RVTagent, demonstrates strong generalization by planning a reasoning graph via an LLM and executing it on a DT, without fine-tuning. The benchmark and baseline provide a scalable, multi-output platform for evaluating visual reasoning in realistic video scenarios, with potential impact on embodied AI and interactive systems; future work includes extending beyond physical attributes to abstract or causal reasoning and integrating DT representations more directly into inference.

Abstract

Visual reasoning, the capability to interpret visual input in response to implicit text query through multi-step reasoning, remains a challenge for deep learning models due to the lack of relevant benchmarks. Previous work in visual reasoning has primarily focused on reasoning segmentation, where models aim to segment objects based on implicit text queries. This paper introduces reasoning visual tasks (RVTs), a unified formulation that extends beyond traditional video reasoning segmentation to a diverse family of visual language reasoning problems, which can therefore accommodate multiple output formats including bounding boxes, natural language descriptions, and question-answer pairs. Correspondingly, we identify the limitations in current benchmark construction methods that rely solely on large language models (LLMs), which inadequately capture complex spatial-temporal relationships and multi-step reasoning chains in video due to their reliance on token representation, resulting in benchmarks with artificially limited reasoning complexity. To address this limitation, we propose a novel automated RVT benchmark construction pipeline that leverages digital twin (DT) representations as structured intermediaries between perception and the generation of implicit text queries. Based on this method, we construct RVTBench, a RVT benchmark containing 3,896 queries of over 1.2 million tokens across four types of RVT (segmentation, grounding, VQA and summary), three reasoning categories (semantic, spatial, and temporal), and four increasing difficulty levels, derived from 200 video sequences. Finally, we propose RVTagent, an agent framework for RVT that allows for zero-shot generalization across various types of RVT without task-specific fine-tuning.

Figures (4)

• Figure 1: Comparison of benchmark dataset construction approaches for reasoning visual tasks. (a) Traditional VLM-based approach using extensive prompts and human supervision, which struggles with multi-step reasoning, temporal understanding, spatial relations preservation, and scalability. (b) Our proposed digital twin representation approach that leverages specialized vision foundation models to create structured intermediate representations enabling more complex implicit text queries.
• Figure 2: Overview of our automated benchmark dataset construction pipeline for reasoning visual tasks. The pipeline includes three components to generate complex data samples without human intervention: (1) Digital twin representation construction, where specialized vision foundation models extract multi-dimensional information from input video frames—including global descriptions, scene-level semantic context, spatial relationships, and instance-specific attributes with depth statistics. This creates a structured JSON that preserves continuous visual-spatial-temporal relationships. (2) Object selection and reasoning tree construction, which first identifies objects of interest from down-sampled DT representations, assigns appropriate task types, and then builds a hierarchical reasoning graph with increasing complexity levels (L1-L4). Each level progressively incorporates more complex relationships between target objects and their attributes. (3) Benchmark dataset construction, which leverages the reasoning tree to generate task-specific implicit queries with corresponding ground-truth annotations at varying difficulty levels, incorporating semantic, spatial, and temporal reasoning categories.
• Figure 3: Visualization of the RVTBench composition and examples across different dimensions. (a-d) Task-specific sunburst charts illustrating the distribution of queries across reasoning categories (semantic, spatial, temporal) and difficulty levels (L1-L4) for segmentation, grounding, summary, and VQA tasks. Each chart includes representative examples that demonstrate the progression in reasoning complexity, from simple attribute identification at L1 (e.g., "Segment the bear with a thick, shaggy coat") to complex multi-step reasoning chains at L4 (e.g., "Identify the left arm of the blonde-haired woman standing behind the glasses-wearing woman at the entrance of the game store"). Reasoning categories are color-coded (semantic: blue, spatial: red, temporal: green) with annotations highlighting specific reasoning types. (e) Token distribution analysis by task type and reasoning category, revealing. (f) Hierarchical breakdown of token distribution across difficulty levels and reasoning categories for each task type.
• Figure 4: Examples of RVTBench across all four task types and difficulty levels. (a) Reasoning segmentation. (b) Reasoning grounding. (c) Reasoning VQA. (d) Reasoning summary. For each example, we also demonstrate the reasoning tree accordingly (lower right of each panel), where nodes represent visual elements and edges indicate relationships between them. The complexity increases from level 1 to level 4 through the incorporation of additional semantic attributes, spatial relationships, and temporal dynamics.