Reasoning under Vision: Understanding Visual-Spatial Cognition in Vision-Language Models for CAPTCHA

TL;DR

This work reframes CAPTCHA challenges as probes of visual–spatial reasoning in vision–language models and introduces CAPTCHA-X, a real-world benchmark with reasoning annotations across seven categories. It demonstrates that step-by-step reasoning significantly enhances solving performance and spatial grounding, with a reported average gain of $38.75\%$ over non-reasoning baselines and a new state-of-the-art average accuracy of $83.9\%$ achieved by a reasoning-centered agentic framework. The study also reveals empirical reasoning scaling laws that link reasoning effort to accuracy and task difficulty, and shows strong transfer to unseen CAPTCHA types. Collectively, these results establish reasoning as a core driver of visual–spatial intelligence and provide a rigorous benchmark and pipeline for future multimodal AI research.

Abstract

CAPTCHA, originally designed to distinguish humans from robots, has evolved into a real-world benchmark for assessing the spatial reasoning capabilities of vision-language models. In this work, we first show that step-by-step reasoning is crucial for vision-language models (VLMs) to solve CAPTCHAs, which represent high-difficulty spatial reasoning tasks, and that current commercial vision-language models still struggle with such reasoning. In particular, we observe that most commercial VLMs (e.g., Gemini, Claude, GPT, etc.) fail to effectively solve CAPTCHAs and thus achieve low accuracy (around 21.9 percent). However, our findings indicate that requiring the model to perform step-by-step reasoning before generating the final coordinates can significantly enhance its solving accuracy, underscoring the severity of the gap. To systematically study this issue, we introduce CAPTCHA-X, the first real-world CAPTCHA benchmark with reasoning, covering seven categories of CAPTCHAs (such as Gobang, hCaptcha, etc.) with step-by-step action solutions and grounding annotations. We further define five reasoning-oriented metrics that enable a comprehensive evaluation of models reasoning capabilities. To validate the effectiveness of reasoning, we also propose a general agentic VLM-based framework that incorporates the models inherent reasoning abilities. Our method achieves state-of-the-art performance across five high-difficulty CAPTCHA types, with an average solving accuracy of 83.9 percent, substantially surpassing existing baselines. These results reveal the limitations of current models and highlight the importance of reasoning in advancing visual-spatial challenges in the future.

Figures (6)

• Figure 1: Grounding annotation (red) versus threshold-based annotation (black) in a GeeTest Gobang puzzle, along with recorded mouse actions and reasoning steps. These mouse actions and reasoning steps are generated by using carefully designed prompts.
• Figure 2: Distribution of our benchmark.
• Figure 3: Our Agentic Vision-Language Model Pipeline.
• Figure 4: Model Accuracy and L2 Distance with and without reasoning. We averaged over multiple evaluation runs to reduce randomness. Orange bars indicate Human performance, while purple bars represent our Agent-2.5-Pro's performance. Blue and light-blue bars correspond to model results with reasoning (WR) and without reasoning (WOR), respectively. For accuracy, higher values are better, whereas for L2 distance, lower values are better instead. A clear trend emerges: in nearly all models, WR significantly outperforms WOR.
• Figure 5: Reasoning Evaluation with Multi-Dimensions: The radar chart shows overall reasoning metrics averaged across CAPTCHA categories.