Quantifying Psychological Sophistication of Malicious Emails
Theodore Longtchi, Rosana Montañez Rodriguez, Kora Gwartney, Ekzhin Ear, David P. Azari, Christopher P. Kelley, Shouhuai Xu
TL;DR
This work tackles the challenge of defending against cyber social engineering by quantifying the psychological sophistication of malicious emails. It introduces a two-dimensional framework built on Psychological Techniques (PTechs) and Psychological Tactics (PTacs) to decompose email content into low-level cues and high-level framing, respectively. Using a case study of 1,036 malicious emails and a structured calibration/grading pipeline, the authors demonstrate that phishing emails are generally more sophisticated than other types and reveal patterns in PTech/PTac usage and their correlations. The framework’s findings—such as the prominence of Attention Grabbing and Impersonation, and the role of social-event contextualization—inform targeted defenses and motivate an automated tool for real-time sophistication assessment. The study also provides a Krippendorff’s Alpha–based method for measuring inter-rater agreement and highlights avenues for extending the framework to other cyber-social engineering domains.
Abstract
Malicious emails including Phishing, Spam, and Scam are one significant class of cyber social engineering attacks. Despite numerous defenses to counter them, the problem remains largely open. The ineffectiveness of current defenses can be attributed to our superficial understanding of the psychological properties that make these attacks successful. This problem motivates us to investigate the psychological sophistication, or sophistication for short, of malicious emails. We propose an innovative framework that accommodates two important and complementary aspects of sophistication, dubbed Psychological Techniques, PTechs, and Psychological Tactics, PTacs. We propose metrics and grading rules for human experts to assess the sophistication of malicious emails via the lens of these PTechs and PTacs. To demonstrate the usefulness of the framework, we conduct a case study based on 1,036 malicious emails assessed by four independent graders. Our results show that malicious emails are psychologically sophisticated, while exhibiting both commonalities and different patterns in terms of their PTechs and PTacs. Results also show that previous studies might have focused on dealing with the less proliferated PTechs such as Persuasion and PTacs such as Reward, rather than the most proliferated PTechs such as Attention Grabbing and Impersonation, and PTacs such as Fit and Form and Familiarity that are identified in this study. We also found among others that social events are widely exploited by attackers in contextualizing their malicious emails. These findings could be leveraged to guide the design of effective defenses against malicious emails.