Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evidence of Cognitive Biases in Capture-the-Flag Cybersecurity Competitions

Carolina Carreira, Anu Aggarwal, Alejandro Cuevas, Maria José Ferreira, Hanan Hibshi, Cleotilde Gonzalez

TL;DR

This paper investigates how cognitive biases shape attacker decision-making in cybersecurity by analyzing over 525,000 submissions from picoCTF. Using a mixed-methods approach, it identifies signatures of availability bias and the sunk cost fallacy in naturalistic, large-scale data, combining qualitative coding, descriptive statistics, and GLMs. Key findings include a notable share of misformatted-but-correct submissions linked to availability bias and persistence in problem-solving despite declining success probabilities tied to the sunk cost fallacy, with demographic patterns suggesting variation in bias expression. The authors propose a bias-informed defense framework—comprising bias triggers, sensors, and adaptive defenses—to anticipate adversarial actions and guide future experimental integration within CTFs and real systems.

Abstract

Understanding how cognitive biases influence adversarial decision-making is essential for developing effective cyber defenses. Capture-the-Flag (CTF) competitions provide an ecologically valid testbed to study attacker behavior at scale, simulating real-world intrusion scenarios under pressure. We analyze over 500,000 submission logs from picoCTF, a large educational CTF platform, to identify behavioral signatures of cognitive biases with defensive implications. Focusing on availability bias and the sunk cost fallacy, we employ a mixed-methods approach combining qualitative coding, descriptive statistics, and generalized linear modeling. Our findings show that participants often submitted flags with correct content but incorrect formatting (availability bias), and persisted in attempting challenges despite repeated failures and declining success probabilities (sunk cost fallacy). These patterns reveal that biases naturally shape attacker behavior in adversarial contexts. Building on these insights, we outline a framework for bias-informed adaptive defenses that anticipate, rather than simply react to, adversarial actions.

Evidence of Cognitive Biases in Capture-the-Flag Cybersecurity Competitions

TL;DR

This paper investigates how cognitive biases shape attacker decision-making in cybersecurity by analyzing over 525,000 submissions from picoCTF. Using a mixed-methods approach, it identifies signatures of availability bias and the sunk cost fallacy in naturalistic, large-scale data, combining qualitative coding, descriptive statistics, and GLMs. Key findings include a notable share of misformatted-but-correct submissions linked to availability bias and persistence in problem-solving despite declining success probabilities tied to the sunk cost fallacy, with demographic patterns suggesting variation in bias expression. The authors propose a bias-informed defense framework—comprising bias triggers, sensors, and adaptive defenses—to anticipate adversarial actions and guide future experimental integration within CTFs and real systems.

Abstract

Understanding how cognitive biases influence adversarial decision-making is essential for developing effective cyber defenses. Capture-the-Flag (CTF) competitions provide an ecologically valid testbed to study attacker behavior at scale, simulating real-world intrusion scenarios under pressure. We analyze over 500,000 submission logs from picoCTF, a large educational CTF platform, to identify behavioral signatures of cognitive biases with defensive implications. Focusing on availability bias and the sunk cost fallacy, we employ a mixed-methods approach combining qualitative coding, descriptive statistics, and generalized linear modeling. Our findings show that participants often submitted flags with correct content but incorrect formatting (availability bias), and persisted in attempting challenges despite repeated failures and declining success probabilities (sunk cost fallacy). These patterns reveal that biases naturally shape attacker behavior in adversarial contexts. Building on these insights, we outline a framework for bias-informed adaptive defenses that anticipate, rather than simply react to, adversarial actions.

Paper Structure

This paper contains 24 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Cognitive Bias in CTF Competitions
  4. Methods
  5. The picoCTF Dataset
  6. Data Preprocessing.
  7. Ethical Considerations.
  8. Demographics.
  9. Key Variables in the Analysis.
  10. Operationalization of Cognitive Biases Signatures
  11. Availability Bias.
  12. Sunk Cost Fallacy.
  13. Results: Availability Bias
  14. Qualitative Coding Results
  15. Descriptive Results
  16. ...and 9 more sections

Figures (2)

  • Figure 1: 2Warm challenge from picoGym with hint showing the additional instructions for flag
  • Figure 2: Empirical probability of a correct submission as a function of the number of prior incorrect attempts on the same challenge. Each point represents the average success rate observed among all submissions that occurred after a given number of failed attempts, aggregated across all users and challenges.