The Impact of Emerging Phishing Threats: Assessing Quishing and LLM-generated Phishing Emails against Organizations

TL;DR

The paper addresses the growing threat of phishing by evaluating two emerging vectors—quishing via QR codes and OSINT-assisted LLM-generated phishing—through large-scale simulations across three organizations. It employs three phishing email variants ($E_B$, $E_Q$, $E_L$) and measures engagement, landing-page visits, and credential submissions, while also assessing perceived phishing awareness (PPA) among employees. Core findings include that $E_B$ and $E_Q$ are statistically equivalent in driving landing-page visits, that OSINT+LLM emails can be highly effective (especially in smaller firms), and that higher PPA strongly predicts lower susceptibility ($R^2=1.0$, $p<0.001$, $ ho=-1.0$). The work highlights urgent defense gaps against quishing and AI-generated social engineering and suggests practical countermeasures, such as enhanced quishing training and detectors for machine-generated content, to bolster organizational resilience. Overall, the results provide a data-driven basis for prioritizing countermeasures and benchmarking across organizations as phishing threats continue to evolve.

Abstract

Modern organizations are persistently targeted by phishing emails. Despite advances in detection systems and widespread employee training, attackers continue to innovate, posing ongoing threats. Two emerging vectors stand out in the current landscape: QR-code baits and LLM-enabled pretexting. Yet, little is known about the effectiveness of current defenses against these attacks, particularly when it comes to real-world impact on employees. This gap leaves uncertainty around to what extent related countermeasures are justified or needed. Our work addresses this issue. We conduct three phishing simulations across organizations of varying sizes -- from small-medium businesses to a multinational enterprise. In total, we send over 71k emails targeting employees, including: a "traditional" phishing email with a click-through button; a nearly-identical "quishing" email with a QR code instead; and a phishing email written with the assistance of an LLM and open-source intelligence. Our results show that quishing emails have the same effectiveness as traditional phishing emails at luring users to the landing webpage -- which is worrying, given that quishing emails are much harder to identify even by operational detectors. We also find that LLMs can be very good "social engineers": in one company, over 30% of the emails opened led to visiting the landing webpage -- a rate exceeding some prior benchmarks. Finally, we complement our study by conducting a survey across the organizations' employees, measuring their "perceived" phishing awareness. Our findings suggest a correlation between higher self-reported awareness and organizational resilience to phishing attempts.

Figures (11)

• Figure 1: Emails used in our experiments. Our emails shared a similar design, but each email presented some company-specific traits to increase authenticity (e.g., we put the company logo at the bottom right). All emails bring the user to the same landing webpage (which was also specific to each company).
• Figure 2: Extraction and exploitation of OSINT for $\mathbb{E}_L$. Operations denoted with a "brain-cog" image have been carried out with an LLM.
• Figure 3: Landing page. All of our emails would point to a webpage with a similar design as this one, showing the typical "Microsoft login".
• Figure 4: Interface of Microsoft Defender Attack Simulation module. This is just an example, no confidential information is shown.
• Figure 5: Interface of GoPhish. This is just an example, no confidential information is shown.