Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CIPHER: Cryptographic Insecurity Profiling via Hybrid Evaluation of Responses

Max Manolov, Tony Gao, Siddharth Shukla, Cheng-Ting Chou, Ryan Lagasse

TL;DR

The paper tackles the risk of cryptographic insecurity in LLM-generated Python code by introducing CIPHER, a standardized benchmark that uses insecure/neutral/secure prompt triplets, a cryptography-focused vulnerability taxonomy, and line-level evidence to profile vulnerability incidence. It provides a reproducible scoring pipeline and applies it to seven widely used LLMs, showing consistently high vulnerability rates even under explicit secure prompting, thus revealing limits to prompt-based security steering. The work situates CIPHER within related efforts on LLM evaluation and crypto API misuse, emphasizing controlled prompting, taxonomy-driven judgments, and scalable, argument-based scoring. By releasing the dataset and pipeline, the authors offer a practical foundation for ongoing measurement and improvement of cryptographic safety in code-generation systems.

Abstract

Large language models (LLMs) are increasingly used to assist developers with code, yet their implementations of cryptographic functionality often contain exploitable flaws. Minor design choices (e.g., static initialization vectors or missing authentication) can silently invalidate security guarantees. We introduce CIPHER(\textbf{C}ryptographic \textbf{I}nsecurity \textbf{P}rofiling via \textbf{H}ybrid \textbf{E}valuation of \textbf{R}esponses), a benchmark for measuring cryptographic vulnerability incidence in LLM-generated Python code under controlled security-guidance conditions. CIPHER uses insecure/neutral/secure prompt variants per task, a cryptography-specific vulnerability taxonomy, and line-level attribution via an automated scoring pipeline. Across a diverse set of widely used LLMs, we find that explicit ``secure'' prompting reduces some targeted issues but does not reliably eliminate cryptographic vulnerabilities overall. The benchmark and reproducible scoring pipeline will be publicly released upon publication.

CIPHER: Cryptographic Insecurity Profiling via Hybrid Evaluation of Responses

TL;DR

The paper tackles the risk of cryptographic insecurity in LLM-generated Python code by introducing CIPHER, a standardized benchmark that uses insecure/neutral/secure prompt triplets, a cryptography-focused vulnerability taxonomy, and line-level evidence to profile vulnerability incidence. It provides a reproducible scoring pipeline and applies it to seven widely used LLMs, showing consistently high vulnerability rates even under explicit secure prompting, thus revealing limits to prompt-based security steering. The work situates CIPHER within related efforts on LLM evaluation and crypto API misuse, emphasizing controlled prompting, taxonomy-driven judgments, and scalable, argument-based scoring. By releasing the dataset and pipeline, the authors offer a practical foundation for ongoing measurement and improvement of cryptographic safety in code-generation systems.

Abstract

Large language models (LLMs) are increasingly used to assist developers with code, yet their implementations of cryptographic functionality often contain exploitable flaws. Minor design choices (e.g., static initialization vectors or missing authentication) can silently invalidate security guarantees. We introduce CIPHER(\textbf{C}ryptographic \textbf{I}nsecurity \textbf{P}rofiling via \textbf{H}ybrid \textbf{E}valuation of \textbf{R}esponses), a benchmark for measuring cryptographic vulnerability incidence in LLM-generated Python code under controlled security-guidance conditions. CIPHER uses insecure/neutral/secure prompt variants per task, a cryptography-specific vulnerability taxonomy, and line-level attribution via an automated scoring pipeline. Across a diverse set of widely used LLMs, we find that explicit ``secure'' prompting reduces some targeted issues but does not reliably eliminate cryptographic vulnerabilities overall. The benchmark and reproducible scoring pipeline will be publicly released upon publication.
Paper Structure (55 sections, 4 figures, 9 tables)

This paper contains 55 sections, 4 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM-as-a-Judge Evaluation
  4. Model-Based Vulnerability Detection and Repair
  5. Security Benchmarks for LLM Code Generation
  6. Cryptographic Misuse Benchmarks and Program Analysis
  7. Benchmark Construction
  8. Data model and core unit of evaluation
  9. Prompt family.
  10. Prompt instance.
  11. Reference implementations and labels.
  12. Constructing prompt families
  13. Sourcing tasks and threat models.
  14. Inclusion criteria.
  15. Template-driven authoring for control.
  16. ...and 40 more sections

Figures (4)

  • Figure 1: Example structure of triplet prompts, model outputs, and judge-detected vulnerabilities.
  • Figure 2: Vulnerability rate (fraction of generations with $\geq 1$ detected vulnerability) for each model (error bars: 95% CI).
  • Figure 3: Log(1 + vulnerability count) per benchmark category for each model.
  • Figure 4: Targeted compliance under security guidance. For each prompt family, we define a target vulnerability (the one intentionally introduced in the vulnerable reference). The plot reports, for secure vs. insecure prompt variants, the fraction of generations in which the target vulnerability is present (lower is better for secure prompts; higher indicates greater compliance with harmful instructions for insecure prompts).