Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CipherBank: Exploring the Boundary of LLM Reasoning Capabilities through Cryptography Challenges

Yu Li, Qizhi Pei, Mengyuan Sun, Honglin Lin, Chenlin Ming, Xin Gao, Jiang Wu, Conghui He, Lijun Wu

TL;DR

CipherBank addresses the gap in evaluating LLM cryptographic reasoning by introducing a large-scale, real-world cryptographic decryption benchmark. It assembles 2,358 problems from 262 plaintexts across 5 domains and 14 subdomains, spanning 3 encryption categories and 9 algorithms with 5 difficulty levels. The study evaluates 18 SOTA LLMs including GPT-4o and o1, revealing pronounced gaps between general chat and reasoning-optimized models, and between handling classical cryptography versus real-world decryption tasks. Through comprehensive analyses of length, noise, and hints, CipherBank exposes limitations and suggests directions to improve structured cryptographic reasoning in LLMs. The benchmark has implications for AI safety and cybersecurity applications where robust decryption reasoning matters.

Abstract

Large language models (LLMs) have demonstrated remarkable capabilities, especially the recent advancements in reasoning, such as o1 and o3, pushing the boundaries of AI. Despite these impressive achievements in mathematics and coding, the reasoning abilities of LLMs in domains requiring cryptographic expertise remain underexplored. In this paper, we introduce CipherBank, a comprehensive benchmark designed to evaluate the reasoning capabilities of LLMs in cryptographic decryption tasks. CipherBank comprises 2,358 meticulously crafted problems, covering 262 unique plaintexts across 5 domains and 14 subdomains, with a focus on privacy-sensitive and real-world scenarios that necessitate encryption. From a cryptographic perspective, CipherBank incorporates 3 major categories of encryption methods, spanning 9 distinct algorithms, ranging from classical ciphers to custom cryptographic techniques. We evaluate state-of-the-art LLMs on CipherBank, e.g., GPT-4o, DeepSeek-V3, and cutting-edge reasoning-focused models such as o1 and DeepSeek-R1. Our results reveal significant gaps in reasoning abilities not only between general-purpose chat LLMs and reasoning-focused LLMs but also in the performance of current reasoning-focused models when applied to classical cryptographic decryption tasks, highlighting the challenges these models face in understanding and manipulating encrypted data. Through detailed analysis and error investigations, we provide several key observations that shed light on the limitations and potential improvement areas for LLMs in cryptographic reasoning. These findings underscore the need for continuous advancements in LLM reasoning capabilities.

CipherBank: Exploring the Boundary of LLM Reasoning Capabilities through Cryptography Challenges

TL;DR

CipherBank addresses the gap in evaluating LLM cryptographic reasoning by introducing a large-scale, real-world cryptographic decryption benchmark. It assembles 2,358 problems from 262 plaintexts across 5 domains and 14 subdomains, spanning 3 encryption categories and 9 algorithms with 5 difficulty levels. The study evaluates 18 SOTA LLMs including GPT-4o and o1, revealing pronounced gaps between general chat and reasoning-optimized models, and between handling classical cryptography versus real-world decryption tasks. Through comprehensive analyses of length, noise, and hints, CipherBank exposes limitations and suggests directions to improve structured cryptographic reasoning in LLMs. The benchmark has implications for AI safety and cybersecurity applications where robust decryption reasoning matters.

Abstract

Large language models (LLMs) have demonstrated remarkable capabilities, especially the recent advancements in reasoning, such as o1 and o3, pushing the boundaries of AI. Despite these impressive achievements in mathematics and coding, the reasoning abilities of LLMs in domains requiring cryptographic expertise remain underexplored. In this paper, we introduce CipherBank, a comprehensive benchmark designed to evaluate the reasoning capabilities of LLMs in cryptographic decryption tasks. CipherBank comprises 2,358 meticulously crafted problems, covering 262 unique plaintexts across 5 domains and 14 subdomains, with a focus on privacy-sensitive and real-world scenarios that necessitate encryption. From a cryptographic perspective, CipherBank incorporates 3 major categories of encryption methods, spanning 9 distinct algorithms, ranging from classical ciphers to custom cryptographic techniques. We evaluate state-of-the-art LLMs on CipherBank, e.g., GPT-4o, DeepSeek-V3, and cutting-edge reasoning-focused models such as o1 and DeepSeek-R1. Our results reveal significant gaps in reasoning abilities not only between general-purpose chat LLMs and reasoning-focused LLMs but also in the performance of current reasoning-focused models when applied to classical cryptographic decryption tasks, highlighting the challenges these models face in understanding and manipulating encrypted data. Through detailed analysis and error investigations, we provide several key observations that shed light on the limitations and potential improvement areas for LLMs in cryptographic reasoning. These findings underscore the need for continuous advancements in LLM reasoning capabilities.
Paper Structure (32 sections, 1 equation, 19 figures, 20 tables)

This paper contains 32 sections, 1 equation, 19 figures, 20 tables.

Table of Contents

  1. Introduction
  2. CipherBank Construction
  3. Plaintext Data: Design, Sources, and Real-World Alignment
  4. Encryption Algorithms
  5. CipherBank Statistics
  6. Evaluations
  7. Evaluation Setup
  8. Benchmark Results
  9. Detailed Analysis
  10. Impact of Plaintext Length
  11. Effect of Noise on Model Robustness
  12. Effect of Encryption Scope
  13. Effect of Explicit Algorithm Hints on Decryption Performance
  14. Error Analysis
  15. Related Work
  16. ...and 17 more sections

Figures (19)

  • Figure 1: Comprehensive Performance of SOTA Chat and Reasoning Models on CipherBank.
  • Figure 2: Overview of CipherBank. CipherBank consists of simulated privacy data encrypted using various algorithms. The left side of the figure shows five domains, 14 subdomains, and selected tags. The right side displays three encryption categories, nine specific algorithms, and their corresponding difficulty levels.
  • Figure 3: Evaluation of LLM Performance Under Different Encryption and Prompting Conditions.
  • Figure 4: Decryption Error Distribution. The left represents chat models, while the right corresponds to reasoning models.
  • Figure 5: System Prompt
  • ...and 14 more figures