CryptoQA: A Large-scale Question-answering Dataset for AI-assisted Cryptography

TL;DR

CryptoQA introduces the first large-scale, domain-specific QA dataset for cryptography, enabling quantitative and qualitative evaluation of 15 LLMs on factuality and mathematical reasoning tasks. The authors show substantial gaps in cryptographic reasoning, with the best open models reaching around 0.70 on general questions and lower on math-oriented prompts. They demonstrate the value of CryptoQA by achieving 7–13% performance gains through fine-tuning a top model, and they validate the dataset via expert reviews and diverse evaluation metrics. The work highlights the need for domain-adapted AI assistants in cryptography and provides a standardized benchmark and training resource to drive progress.

Abstract

Large language models (LLMs) excel at many general-purpose natural language processing tasks. However, their ability to perform deep reasoning and mathematical analysis, particularly for complex tasks as required in cryptography, remains poorly understood, largely due to the lack of suitable data for evaluation and training. To address this gap, we present CryptoQA, the first large-scale question-answering (QA) dataset specifically designed for cryptography. CryptoQA contains over two million QA pairs drawn from curated academic sources, along with contextual metadata that can be used to test the cryptographic capabilities of LLMs and to train new LLMs on cryptographic tasks. We benchmark 15 state-of-the-art LLMs on CryptoQA, evaluating their factual accuracy, mathematical reasoning, consistency, referencing, backward reasoning, and robustness to adversarial samples. In addition to quantitative metrics, we provide expert reviews that qualitatively assess model outputs and establish a gold-standard baseline. Our results reveal significant performance deficits of LLMs, particularly on tasks that require formal reasoning and precise mathematical knowledge. This shows the urgent need for LLM assistants tailored to cryptography research and development. We demonstrate that, by using CryptoQA, LLMs can be fine-tuned to exhibit better performance on cryptographic tasks.

Figures (13)

• Figure 1: Our new dataset CryptoQA is the first large-scale cryptography dataset with over 2.5M QA pairs with rich metadata, i.e., different topics, categories, sources, and types of QA pairs (cf. \ref{['sec:dataset']}). We use CryptoQA to quantitatively and qualitatively evaluate existing state-of-the-art LLMs for their cryptographic capabilities, such as factuality and mathematical reasoning, w.r.t. different metrics (cf. \ref{['sec:evaluation', 'sec:results']}). (Icons generated by deepai.org.)
• Figure 2: The main evaluation metrics when prompting our candidate LLMs with the original dataset versus the source (as DOI) along the original dataset (\ref{['fig:original']}), and the paraphrased dataset (\ref{['fig:paraphrased']}). More comprehensive figures showing other evaluation metrics can be found in \ref{['apx:comprehensive-results']}.
• Figure 3: The averaged scores over all LLMs for the different metadata attributes. Further metrics can be found in \ref{['apx:comprehensive-results']}.
• Figure 4: The LLMs' performance on our backward and adversarial subsets.
• Figure 5: Gold Standard: The human's and the LLM's performance on our qualitative subset by only evaluating the answers. The first items represent each expert level, the 'PhD (w/out IDK)' represents the PhD holders responses when they were certain (excluding the ' I do not know' responses), and the 'Avg. Human' represent the average performance across all expertise levels. More comprehensive results can be found in \ref{['apx:comprehensive-results']}