BLINK-Twice: You see, but do you observe? A Reasoning Benchmark on Visual Perception

TL;DR

BLINK-Twice targets a gap in multimodal reasoning benchmarks by introducing a vision-centered dataset that requires image-content-based reasoning without external knowledge. It combines seven visual challenges, GPT-4o-generated natural adversarial samples, and annotated reasoning chains to evaluate not only final answers but the quality of the accompanying reasoning. The study evaluates 20 MLLMs, revealing that many models 'see' rather than truly observe, with chain-of-thought methods offering limited and sometimes inefficient gains; however, multi-turn observation and active visual reasoning (as seen in the o3 model) yield meaningful improvements. The dataset and evaluation protocol, including No-Acc, Yes-Acc, Q-Acc, I-Acc, G-Acc, and CoT-score, are publicly available to spur development of genuinely image-grounded multimodal reasoning systems.

Abstract

Recently, Multimodal Large Language Models (MLLMs) have made rapid progress, particularly in enhancing their reasoning capabilities. However, existing reasoning benchmarks still primarily assess language-based reasoning, often treating visual input as replaceable context. To address this gap, we introduce BLINK-Twice, a vision-centric reasoning benchmark grounded in challenging perceptual tasks. Instead of relying on external knowledge, our tasks require models to reason from visual content alone, shifting the focus from language-based to image-grounded reasoning. Compared to prior perception benchmarks, it moves beyond shallow perception ("see") and requires fine-grained observation and analytical reasoning ("observe"). BLINK-Twice integrates three core components: seven types of visual challenges for testing visual reasoning, natural adversarial image pairs that enforce reliance on visual content, and annotated reasoning chains for fine-grained evaluation of the reasoning process rather than final answers alone. We evaluate 20 leading MLLMs, including 12 foundation models and 8 reasoning-enhanced models. BLINK-Twice poses a significant challenge to current models. While existing reasoning strategies in the language space-such as chain-of-thought or self-criticism can improve performance, they often result in unstable and redundant reasoning. We observe that repeated image observation improves performance across models, and active visual interaction, as demonstrated by models like o3, highlights the need for a new paradigm for vision reasoning. The dataset is publicly available at https://github.com/PicoTrex/BLINK-Twice

Figures (6)

• Figure 1: BLINK-Twice Task Overview: (a) Visual Reasoning task requiring detailed observation and careful reasoning; (b) Natural adversarial samples with similar appearance but opposite semantics, forcing models to rely on visual input; (c) Reasoning step annotation including detailed visual clues and true reality to evaluate thought chain output.
• Figure 2: Overview of BLINK-Twice Dataset. (a) Distribution and examples of different visual challenges; (b) Pipeline for automatic adversarial sample generation and (c) reasoning chain annotation.
• Figure 3: BLINK-Twice qualitative evaluation visualization results. The upper part shows the results of a single difficult image. The lower part displays the results for a group of images.
• Figure 4: Comparison and Visualization of Reasoning Models and MLLMs.(a) Comparison of Q-Acc between reasoning models and based MLLMs. (b) The visualization illustrates the model’s response process, highlighting step-by-step reasoning and self-critical chains of thought.
• Figure 5: The distribution of generation length of Reasoning model.