Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EnSolver: Uncertainty-Aware Ensemble CAPTCHA Solvers with Theoretical Guarantees

Duc C. Hoang, Behzad Ousat, Amin Kharraz, Cuong V. Nguyen

TL;DR

EnSolver introduces uncertainty-aware CAPTCHA solvers that use deep ensembles to detect and skip out-of-distribution CAPTCHAs, mitigating failure-driven lockouts. LEnSolver extends this by imposing a maximum number of skips, ensuring practical progress in solving attempts. The authors derive novel theoretical guarantees via an out-of-distribution error bound (OEB) and provide lower bounds on right-decision and success rates for EnSolver and LEnSolver, respectively. Empirical results on in- and out-of-distribution CAPTCHA data show robust performance improvements over strong baselines and confirm the relevance of the theoretical bounds for real-world settings.

Abstract

The popularity of text-based CAPTCHA as a security mechanism to protect websites from automated bots has prompted researches in CAPTCHA solvers, with the aim of understanding its failure cases and subsequently making CAPTCHAs more secure. Recently proposed solvers, built on advances in deep learning, are able to crack even the very challenging CAPTCHAs with high accuracy. However, these solvers often perform poorly on out-of-distribution samples that contain visual features different from those in the training set. Furthermore, they lack the ability to detect and avoid such samples, making them susceptible to being locked out by defense systems after a certain number of failed attempts. In this paper, we propose EnSolver, a family of CAPTCHA solvers that use deep ensemble uncertainty to detect and skip out-of-distribution CAPTCHAs, making it harder to be detected. We prove novel theoretical bounds on the effectiveness of our solvers and demonstrate their use with state-of-the-art CAPTCHA solvers. Our experiments show that the proposed approaches perform well when cracking CAPTCHA datasets that contain both in-distribution and out-of-distribution samples.

EnSolver: Uncertainty-Aware Ensemble CAPTCHA Solvers with Theoretical Guarantees

TL;DR

EnSolver introduces uncertainty-aware CAPTCHA solvers that use deep ensembles to detect and skip out-of-distribution CAPTCHAs, mitigating failure-driven lockouts. LEnSolver extends this by imposing a maximum number of skips, ensuring practical progress in solving attempts. The authors derive novel theoretical guarantees via an out-of-distribution error bound (OEB) and provide lower bounds on right-decision and success rates for EnSolver and LEnSolver, respectively. Empirical results on in- and out-of-distribution CAPTCHA data show robust performance improvements over strong baselines and confirm the relevance of the theoretical bounds for real-world settings.

Abstract

The popularity of text-based CAPTCHA as a security mechanism to protect websites from automated bots has prompted researches in CAPTCHA solvers, with the aim of understanding its failure cases and subsequently making CAPTCHAs more secure. Recently proposed solvers, built on advances in deep learning, are able to crack even the very challenging CAPTCHAs with high accuracy. However, these solvers often perform poorly on out-of-distribution samples that contain visual features different from those in the training set. Furthermore, they lack the ability to detect and avoid such samples, making them susceptible to being locked out by defense systems after a certain number of failed attempts. In this paper, we propose EnSolver, a family of CAPTCHA solvers that use deep ensemble uncertainty to detect and skip out-of-distribution CAPTCHAs, making it harder to be detected. We prove novel theoretical bounds on the effectiveness of our solvers and demonstrate their use with state-of-the-art CAPTCHA solvers. Our experiments show that the proposed approaches perform well when cracking CAPTCHA datasets that contain both in-distribution and out-of-distribution samples.
Paper Structure (23 sections, 7 theorems, 43 equations, 4 figures, 3 tables, 4 algorithms)

This paper contains 23 sections, 7 theorems, 43 equations, 4 figures, 3 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Uncertainty-Aware CAPTCHA Solver Using Deep Ensembles
  4. Uncertainty-Aware CAPTCHA Solver
  5. EnSolver: The Deep Ensemble Solver
  6. LEnSolver: The Limited-Skip EnSolver
  7. Theoretical Guarantees for EnSolver and LEnSolver
  8. Mathematical Settings
  9. Theoretical Guarantee for EnSolver
  10. Theoretical Guarantee for LEnSolver
  11. Experiments
  12. Experiment Settings
  13. Right Decision Rates of EnSolvers
  14. Success Rates of LEnSolvers
  15. Effects of Maximum Number of Skips
  16. ...and 8 more sections

Key Result

Lemma 1

For any EnSolver $m$ with ensemble size $M$, output domain size $N_{\mathcal{S}}$, and uncertainty threshold $\tau$, its OEB satisfies:

Figures (4)

  • Figure 1: The main component of EnSolver that predicts an output string together with an associated uncertainty level given an input CAPTCHA image. The input image is first fed into each base model, each of which produces a string as output. The output strings form a distribution, which is used to compute the final prediction and the uncertainty level.
  • Figure 2: A sample CAPTCHA image in our new dataset. The ground truth label consists of the bounding boxes of each character, the correct letter for each character, and the correct output string NP5tZ.
  • Figure 3: Our generated dataset (a) and public datasets (b)-(i) used in our experiments. Each dataset has unique visual features.
  • Figure 4: Actual and theoretical success rates of LEnSolver with respect to the maximum number of skips (T) for different values of $\alpha$ and base model types.

Theorems & Definitions (10)

  • Definition 1
  • Definition 2
  • Lemma 1
  • Lemma 2
  • Theorem 1
  • Definition 3
  • Lemma 3
  • Lemma 4
  • Theorem 2
  • Lemma 5