Evaluating Large Language Models' Capability to Launch Fully Automated Spear Phishing Campaigns: Validated on Human Subjects

TL;DR

This study benchmarks frontier language models on fully automated spear phishing campaigns and compares AI-generated emails to human-crafted and hybrid approaches in a real-world 101-participant study. It develops an end-to-end AI phishing tool that integrates OSINT reconnaissance, hyper-personalized email generation, and campaign execution, and it analyzes detection methods for phishing intent using multiple LLMs. The paper finds AI-generated and AI-assisted phishing to be on par with human experts and markedly more cost-efficient, while also providing an AI-based framework for detecting suspicious emails with high accuracy. It highlights significant security implications, including the need for advanced defense mechanisms, personalized mitigation techniques, and policy-level actions to address AI-enhanced cyber threats and their economics.

Abstract

In this paper, we evaluate the capability of large language models to conduct personalized phishing attacks and compare their performance with human experts and AI models from last year. We include four email groups with a combined total of 101 participants: A control group of arbitrary phishing emails, which received a click-through rate (recipient pressed a link in the email) of 12%, emails generated by human experts (54% click-through), fully AI-automated emails 54% (click-through), and AI emails utilizing a human-in-the-loop (56% click-through). Thus, the AI-automated attacks performed on par with human experts and 350% better than the control group. The results are a significant improvement from similar studies conducted last year, highlighting the increased deceptive capabilities of AI models. Our AI-automated emails were sent using a custom-built tool that automates the entire spear phishing process, including information gathering and creating personalized vulnerability profiles for each target. The AI-gathered information was accurate and useful in 88% of cases and only produced inaccurate profiles for 4% of the participants. We also use language models to detect the intention of emails. Claude 3.5 Sonnet scored well above 90% with low false-positive rates and detected several seemingly benign emails that passed human detection. Lastly, we analyze the economics of phishing, highlighting how AI enables attackers to target more individuals at lower cost and increase profitability by up to 50 times for larger audiences.

Figures (13)

• Figure 1: Overview of AI-automated phishing campaigns. The process includes target identification, synthetic attacker profile creation, personalized email generation, and campaign execution with self-learning capabilities.
• Figure 2: Example of an abbreviated profile written about one of the authors by our AI reconnaissance tool.
• Figure 3: Example of phishing email created by a human expert.
• Figure 4: Email message generated by Claude 3.5 Sonnet based on an AI-written profile of one of the authors.
• Figure 5: Success rate of the phishing emails for each group. The success rate is the percentage of group members that pressed a link in the phishing email they received. AI Hybrid refers to AI with a human-in-the-loop; for detailed explanations on each group, see section \ref{['sec:groups']}.