LLMs unlock new paths to monetizing exploits
Nicholas Carlini, Milad Nasr, Edoardo Debenedetti, Barry Wang, Christopher A. Choquette-Choo, Daphne Ippolito, Florian Tramèr, Matthew Jagielski
TL;DR
The paper analyzes how state-of-the-art LLMs can shift the economics of cyberattacks by enabling both long-tail exploitation and targeted, per-device attacks. It provides an economic framework and a suite of empirical case studies showing how LLMs improve data mining for sensitive information, automate exploitation in low-user-count apps, mimic trusted devices, and execute actions as an authenticated user, among other vectors. While current demonstrations are costly and not yet widely profitable, the authors argue that rapid declines in LLM inference costs will expand viable attack surfaces, necessitating defense-in-depth and proactive mitigations such as PII-mining controls and on-device threat profiling. The work also discusses ethical considerations and outlines future directions for defense and research into LLM-based cyber defenses. Overall, the study highlights a near-future cybersecurity landscape where adversaries with LLMs can scale personalized exploits, prompting urgent attention to robust, layered defenses and responsible model-use monitoring. $value = (profit\ per\ exploit) \times (\#\ impacted) - (cost\ to\ identify\ vulnerability\ and\ develop\ the\ exploit)$ should be read as a guiding economic lens for evaluating attack viability across contexts.
Abstract
We argue that Large language models (LLMs) will soon alter the economics of cyberattacks. Instead of attacking the most commonly used software and monetizing exploits by targeting the lowest common denominator among victims, LLMs enable adversaries to launch tailored attacks on a user-by-user basis. On the exploitation front, instead of human attackers manually searching for one difficult-to-identify bug in a product with millions of users, LLMs can find thousands of easy-to-identify bugs in products with thousands of users. And on the monetization front, instead of generic ransomware that always performs the same attack (encrypt all your data and request payment to decrypt), an LLM-driven ransomware attack could tailor the ransom demand based on the particular content of each exploited device. We show that these two attacks (and several others) are imminently practical using state-of-the-art LLMs. For example, we show that without any human intervention, an LLM finds highly sensitive personal information in the Enron email dataset (e.g., an executive having an affair with another employee) that could be used for blackmail. While some of our attacks are still too expensive to scale widely today, the incentives to implement these attacks will only increase as LLMs get cheaper. Thus, we argue that LLMs create a need for new defense-in-depth approaches.