COGNITION: From Evaluation to Defense against Multimodal LLM CAPTCHA Solvers

TL;DR

This study analyzes how multimodal large language models threaten visual CAPTCHA security by evaluating seven MLLMs on 18 real-world CAPTCHA types under a black-box threat model that includes latency, retries, and cost. It reveals a pronounced hardness gap: recognition-oriented tasks are easily solved with low cost and latency, while fine-grained localization and counting tasks remain robust against current solvers. Prompt engineering and few-shot demonstrations offer limited gains, with reasoning traces pointing to localization and counting errors as key failure modes. The authors distill defense-oriented guidelines to design more robust CAPTCHA tasks and discuss implications for abuse-mitigation pipelines in real-world platforms.

Abstract

This paper studies how multimodal large language models (MLLMs) undermine the security guarantees of visual CAPTCHA. We identify the attack surface where an adversary can cheaply automate CAPTCHA solving using off-the-shelf models. We evaluate 7 leading commercial and open-source MLLMs across 18 real-world CAPTCHA task types, measuring single-shot accuracy, success under limited retries, end-to-end latency, and per-solve cost. We further analyze the impact of task-specific prompt engineering and few-shot demonstrations on solver effectiveness. We reveal that MLLMs can reliably solve recognition-oriented and low-interaction CAPTCHA tasks at human-like cost and latency, whereas tasks requiring fine-grained localization, multi-step spatial reasoning, or cross-frame consistency remain significantly harder for current models. By examining the reasoning traces of such MLLMs, we investigate the underlying mechanisms of why models succeed/fail on specific CAPTCHA puzzles and use these insights to derive defense-oriented guidelines for selecting and strengthening CAPTCHA tasks. We conclude by discussing implications for platform operators deploying CAPTCHA as part of their abuse-mitigation pipeline.Code Availability (https://anonymous.4open.science/r/Captcha-465E/).

Figures (9)

• Figure 1: CAPTCHA robustness evaluation framework against MLLMs.
• Figure 2: Heatmap of CAPTCHA task difficulty in Exp1 (original prompts). Task types (rows) are sorted by cross-model average Pass@1. Columns correspond to MLLMs, and each cell reports Pass@1 (%).
• Figure 3: Cross-model Pass@1 distributions per task type in Exp1 (original prompts). Each box shows the spread across models. The dashed line marks a $40\%$ threshold.
• Figure 4: Heatmap of CAPTCHA task difficulty in Exp2 (optimized prompts). Task types (rows) are sorted by cross-model average Pass@1, and columns correspond to MLLMs.
• Figure 5: Cross-model Pass@1 distributions per task type in Exp2 (optimized prompts). Compared to Exp1, most recognition-style tasks rise well above the 40% threshold, while the six hard task types remain low across models.