IllusionCAPTCHA: A CAPTCHA based on Visual Illusion

TL;DR

This work addresses the vulnerability of traditional CAPTCHAs to modern AI, especially multimodal LLMs, by proposing IllusionCAPTCHA, an illusion-based, human friendly yet AI-hard CAPTCHA. The authors conduct an empirical study of LLM performance across CAPTCHA types, followed by a human user study, establishing that reasoning-based CAPTCHAs pose challenges for both humans and AI. IllusionCAPTCHA combines visually deceptive images generated via an illusion diffusion pipeline with a structured multi-choice format and inducement prompts to bias AI decisions, achieving AI-hardness while preserving human usability. The results show strong first-attempt human success (≈87%) and near-complete AI failure, suggesting practical security improvements against LLM-based CAPTCHA attacks.

Abstract

CAPTCHAs have long been essential tools for protecting applications from automated bots. Initially designed as simple questions to distinguish humans from bots, they have become increasingly complex to keep pace with the proliferation of CAPTCHA-cracking techniques employed by malicious actors. However, with the advent of advanced large language models (LLMs), the effectiveness of existing CAPTCHAs is now being undermined. To address this issue, we have conducted an empirical study to evaluate the performance of multimodal LLMs in solving CAPTCHAs and to assess how many attempts human users typically need to pass them. Our findings reveal that while LLMs can solve most CAPTCHAs, they struggle with those requiring complex reasoning type of CAPTCHA that also presents significant challenges for human users. Interestingly, our user study shows that the majority of human participants require a second attempt to pass these reasoning CAPTCHAs, a finding not reported in previous research. Based on empirical findings, we present IllusionCAPTCHA, a novel security mechanism employing the "Human-Easy but AI-Hard" paradigm. This new CAPTCHA employs visual illusions to create tasks that are intuitive for humans but highly confusing for AI models. Furthermore, we developed a structured, step-by-step method that generates misleading options, which particularly guide LLMs towards making incorrect choices and reduce their chances of successfully solving CAPTCHAs. Our evaluation shows that IllusionCAPTCHA can effectively deceive LLMs 100% of the time. Moreover, our structured design significantly increases the likelihood of AI errors when attempting to solve these challenges. Results from our user study indicate that 86.95% of participants successfully passed the CAPTCHA on their first attempt, outperforming other CAPTCHA systems.

Figures (5)

• Figure 1: Text-based CAPTCHA
• Figure 2: Image-based CAPTCHA
• Figure 3: Reasoning-based CAPTCHA
• Figure 4: Overview of IllusionCAPTCHA
• Figure 5: An example of the original and illusionary image