VisualTrans: A Benchmark for Real-World Visual Transformation Reasoning

TL;DR

VisualTrans introduces a real-world Visual Transformation Reasoning benchmark built from egocentric manipulation videos, spanning 12 tasks and evaluating spatial, procedural, and quantitative reasoning with 472 QA pairs. Its automated data construction pipeline integrates GroundingDINO and Gemini 2.5 Pro for object detection and metadata generation, followed by human verification to ensure quality. Experimental results show current vision-language models lag in dynamic, multi-step and causal reasoning, underscoring the need for improved temporal modeling and reasoning capabilities. The benchmark addresses sim-to-real gaps and provides a scalable, interpretable framework for diagnosing and improving VTR in real-world settings.

Abstract

Visual transformation reasoning (VTR) is a vital cognitive capability that empowers intelligent agents to understand dynamic scenes, model causal relationships, and predict future states, and thereby guiding actions and laying the foundation for advanced intelligent systems. However, existing benchmarks suffer from a sim-to-real gap, limited task complexity, and incomplete reasoning coverage, limiting their practical use in real-world scenarios. To address these limitations, we introduce VisualTrans, the first comprehensive benchmark specifically designed for VTR in real-world human-object interaction scenarios. VisualTrans encompasses 12 semantically diverse manipulation tasks and systematically evaluates three essential reasoning dimensions - spatial, procedural, and quantitative - through 6 well-defined subtask types. The benchmark features 472 high-quality question-answer pairs in various formats, including multiple-choice, open-ended counting, and target enumeration. We introduce a scalable data construction pipeline built upon first-person manipulation videos, which integrates task selection, image pair extraction, automated metadata annotation with large multimodal models, and structured question generation. Human verification ensures the final benchmark is both high-quality and interpretable. Evaluations of various state-of-the-art vision-language models show strong performance in static spatial tasks. However, they reveal notable shortcomings in dynamic, multi-step reasoning scenarios, particularly in areas like intermediate state recognition and transformation sequence planning. These findings highlight fundamental weaknesses in temporal modeling and causal reasoning, providing clear directions for future research aimed at developing more capable and generalizable VTR systems. The dataset and code are available at https://github.com/WangYipu2002/VisualTrans.

Figures (48)

• Figure 1: Overview of VisualTrans. We introduce VisualTrans, a real-world benchmark for Visual Transformation Reasoning (VTR), featuring 12 egocentric manipulation tasks and evaluating three core reasoning dimensions—spatial, procedural, and quantitative—through systematically constructed question-answer pairs, enabling diagnosis of model capabilities in dynamic scene understanding and causal inference.
• Figure 2: Overview of Construction Pipeline.
• Figure 3: Overview of VisualTrans. (a) Task distribution across three reasoning types. (b) Scene type distribution from real-world manipulation.
• Figure 4: Error Analyses. This figure illustrates the five error types identified in VisualTrans. For each error type, a representative example is provided, with the specific nature of the error clearly indicated.
• Figure 5: Egodex is a video collection featuring diverse manipulation scenarios. Each image pair in our benchmark corresponds to the first and last frame of a video, capturing the transformation process of a manipulation action.