When RSA Fails: Exploiting Prime Selection Vulnerabilities in Public Key Cryptography

TL;DR

RSA security rests on factoring the modulus $n=pq$, but improper prime selection—such as very close primes or shared primes across keys—undermines the math. The paper analyzes Fermat's factorization and GCD-based attacks, surveys landmark evidence (e.g., $6\times10^6$ certificates with $>6\times10^6$ keys examined and $0.2\%$ vulnerable, from Heninger et al.) and Böck's 2023 findings, and attributes real-world weaknesses to weak entropy in embedded devices. It documents ROCA and related incidents, and proposes mitigations including entropy validation, prime-distance checks like $|p-q| > 2^{n/2-100}$, post-generation Fermat testing, and hardware RNGs, while addressing post-quantum considerations. The work emphasizes that cryptographic security is as much about secure implementation as it is about algorithms, with practical guidance for practitioners and policymakers to reduce risk during the transition to post-quantum cryptography.

Abstract

This paper explores vulnerabilities in RSA cryptosystems that arise from improper prime number selection during key generation. We examine two primary attack vectors: Fermat's factorization method, which exploits RSA keys generated with primes that are too close together, and the Greatest Common Divisor (GCD) attack, which exploits keys that share a common prime factor. Drawing from landmark research including Heninger et al.'s ``Mining Your Ps and Qs'' study, which discovered over 64,000 vulnerable TLS hosts, and B{ö}ck's 2023 analysis of Fermat factorization in deployed systems, we demonstrate that these vulnerabilities remain prevalent in real-world cryptographic implementations. Our analysis reveals that weak random number generation in embedded devices is the primary cause of these failures, and we discuss mitigation strategies including proper entropy collection and prime validation checks.