VisChainBench: A Benchmark for Multi-Turn, Multi-Image Visual Reasoning Beyond Language Priors

TL;DR

VisChainBench introduces a large-scale, multi-turn, multi-image visual reasoning benchmark designed to operate with minimal language priors. It employs a multi-agent data-generation pipeline to create 1,457 tasks over 20,431 images across three domains, enabling evaluation of image-to-image reasoning across extended sequences. The study shows a clear performance gap between proprietary and open LVLMs, with model size and training on structured data driving improvements, and finds limited benefits from current long-thinking or chain-of-thought approaches in image-only settings. The work provides open-source benchmark construction tools and prompts a shift toward image-centric reasoning benchmarks to push LVLM capabilities beyond language-conditioned tasks.

Abstract

Understanding multi-image, multi-turn scenarios is a critical yet underexplored capability for Large Vision-Language Models (LVLMs). Existing benchmarks predominantly focus on static or horizontal comparisons -- e.g., spotting visual differences or assessing appropriateness -- while relying heavily on language cues. Such settings overlook progressive, context-dependent reasoning and the challenge of visual-to-visual inference. To bridge this gap, we present VisChainBench, a large-scale benchmark designed to rigorously evaluate LVLMs' ability to perform multi-step visual reasoning across sequential, interdependent tasks with minimal language guidance. VisChainBench contains 1,457 tasks spanning over 20,000 images across three diverse domains (e.g., daily scenarios, engineering troubleshooting), structured to mimic real-world decision-making processes. Uniquely, the benchmark is constructed using a multi-agent generation pipeline, ensuring high visual diversity and controlled language bias. All the benchmark data and code for benchmark construction are available for viewing and download via following Link: https://huggingface.co/datasets/eyehole/VisChainBench

Figures (13)

• Figure 1: Traditional VQA tasks rely on language prompts and responses (top). VisChainBench introduces a purely visual paradigm, where the context, prompt, and answer are all images-challenging models to perform multi-step visual reasoning without textual cues (bottom).
• Figure 2: Comparison of task formats in prior benchmarks. Previous benchmarks are often text-heavy and encourage shallow image comparisons, whereas our benchmark emphasizes progressive image-grounded reasoning.
• Figure 3:
• Figure 4: Task distribution across domains. Our benchmark features tasks in three primary formats. Data was collected with emphasis on problem decomposition and solution strategies across diverse domains.
• Figure 5: Benchmark Construction Process.The workflow begins with automated task generation using Llama3.3-70B to produce structured JSON task descriptions. Then we use a Qwen2-VL-72B model to get corresponding images based on the Task description files and perform consistency verification. Validated tasks will be processed through human quality checks and modifications. This pipeline ensures systematic task creation while maintaining quality control between each processing stage.