Training Users Against Human and GPT-4 Generated Social Engineering Attacks

TL;DR

Phishing threats increasingly leverage LLMs to craft realistic emails and styling, motivating an evaluation of training methods under AI-generated content. The authors compare human-written, GPT-4-written, and GPT-4-styled emails in a structured 2×2 experiment and propose an Instance-Based Learning (IBL) cognitive framework to predict learning and optimize training via embedding-based email selection. Key findings include significant style effects, an AI-writing bias that affects phishing judgments, and simulations showing that adaptive IBL-driven email selection yields larger learning gains than random training. The work demonstrates how cognitive modeling and AI-generated training data can inform safer cybersecurity education and guide future phishing-defense training designs.

Abstract

In real-world decision making, outcomes are often delayed, meaning individuals must make multiple decisions before receiving any feedback. Moreover, feedback can be presented in different ways: it may summarize the overall results of multiple decisions (aggregated feedback) or report the outcome of individual decisions after some delay (clustered feedback). Despite its importance, the timing and presentation of delayed feedback has received little attention in cognitive modeling of decision-making, which typically focuses on immediate feedback. To address this, we conducted an experiment to compare the effect of delayed vs. immediate feedback and aggregated vs. clustered feedback. We also propose a Hierarchical Instance-Based Learning (HIBL) model that captures how people make decisions in delayed feedback settings. HIBL uses a super-model that chooses between sub-models to perform the decision-making task until an outcome is observed. Simulations show that HIBL best predicts human behavior and specific patterns, demonstrating the flexibility of IBL models.

Figures (5)

• Figure 1: An example of the email identification task shown to participants
• Figure 2: Top-Left: The original plain-text email written by human experts Bottom-Left: The GPT-4 stylized version of this original email. Bottom-Right: The fully GPT-4 rewritten and stylized version of the email. Top-Right: The stripped plain-text version of the fully GPT-4 rewritten email.
• Figure 3: Pre and post-training categorization accuracy for ham and phishing emails by experimental condition.
• Figure 4: Linear regression comparing the percentage of emails categorized as being phishing emails and the proportion of emails identified as being AI written. Regressions are split between each of the four experimental conditions. Shaded regions represent 95% confidence intervals of linear regression with $R^2$ and slope labeled.
• Figure 5: All improvement measures refer to the percentage point difference between pre-training and post-training accuracy. Left: Humans participant data. Middle: Simulated IBL agents improvement under random email selection. Right: Simulated IBL agent improvement under IBL email selection method.