Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Shadow of Fraud: The Emerging Danger of AI-powered Social Engineering and its Possible Cure

Jingru Yu, Yi Yu, Xuhong Wang, Yilun Lin, Manzhi Yang, Yu Qiao, Fei-Yue Wang

TL;DR

The paper surveys AI-powered social engineering threats and introduces the 3E framework (Enlarging, Enriching, Emerging) to describe evolving attack modalities. It proposes a quantitative risk assessment pathway based on a Markov decision process to model attack dynamics and derive metrics for spreading and impact. A defense taxonomy is presented, spanning targets and the AI lifecycle, with methods such as adversarial training, differential privacy, watermarking, and architectural isolation, alongside ethical and legal considerations. The work highlights research gaps and provides guidance for researchers and practitioners to monitor, quantify, and mitigate AI-enabled SE threats in real-world settings.

Abstract

Social engineering (SE) attacks remain a significant threat to both individuals and organizations. The advancement of Artificial Intelligence (AI), including diffusion models and large language models (LLMs), has potentially intensified these threats by enabling more personalized and convincing attacks. This survey paper categorizes SE attack mechanisms, analyzes their evolution, and explores methods for measuring these threats. It highlights the challenges in raising awareness about the risks of AI-enhanced SE attacks and offers insights into developing proactive and adaptable defense strategies. Additionally, we introduce a categorization of the evolving nature of AI-powered social engineering attacks into "3E phases": Enlarging, wherein the magnitude of attacks expands through the leverage of digital media; Enriching, introducing novel attack vectors and techniques; and Emerging, signifying the advent of novel threats and methods. Moreover, we emphasize the necessity for a robust framework to assess the risk of AI-powered SE attacks. By identifying and addressing gaps in existing research, we aim to guide future studies and encourage the development of more effective defenses against the growing threat of AI-powered social engineering.

The Shadow of Fraud: The Emerging Danger of AI-powered Social Engineering and its Possible Cure

TL;DR

The paper surveys AI-powered social engineering threats and introduces the 3E framework (Enlarging, Enriching, Emerging) to describe evolving attack modalities. It proposes a quantitative risk assessment pathway based on a Markov decision process to model attack dynamics and derive metrics for spreading and impact. A defense taxonomy is presented, spanning targets and the AI lifecycle, with methods such as adversarial training, differential privacy, watermarking, and architectural isolation, alongside ethical and legal considerations. The work highlights research gaps and provides guidance for researchers and practitioners to monitor, quantify, and mitigate AI-enabled SE threats in real-world settings.

Abstract

Social engineering (SE) attacks remain a significant threat to both individuals and organizations. The advancement of Artificial Intelligence (AI), including diffusion models and large language models (LLMs), has potentially intensified these threats by enabling more personalized and convincing attacks. This survey paper categorizes SE attack mechanisms, analyzes their evolution, and explores methods for measuring these threats. It highlights the challenges in raising awareness about the risks of AI-enhanced SE attacks and offers insights into developing proactive and adaptable defense strategies. Additionally, we introduce a categorization of the evolving nature of AI-powered social engineering attacks into "3E phases": Enlarging, wherein the magnitude of attacks expands through the leverage of digital media; Enriching, introducing novel attack vectors and techniques; and Emerging, signifying the advent of novel threats and methods. Moreover, we emphasize the necessity for a robust framework to assess the risk of AI-powered SE attacks. By identifying and addressing gaps in existing research, we aim to guide future studies and encourage the development of more effective defenses against the growing threat of AI-powered social engineering.
Paper Structure (27 sections, 4 equations, 5 figures, 2 tables)

This paper contains 27 sections, 4 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Systematic literature review methodology
  3. Survey Scope
  4. Paper collection strategy
  5. Findings
  6. Research progress
  7. Evolving landscapes of social engineering: from enlargement to emergence
  8. Human-based attacks: Traditional social engineering techniques
  9. Transition to computer-based attacks: Automation and scalability
  10. Enlarging: scaling attack targets through online medium
  11. Enriching: expanding attack vectors through deep learning techniques
  12. Emerging: empowering novel attack techniques through large language models
  13. Social engineering attack scenarios
  14. Financial scenarios
  15. Medical scenarios
  16. ...and 12 more sections

Figures (5)

  • Figure 1: Advances in AI-powered social engineering attack in the context of evolving social systems.
  • Figure 2: Survey scope: establishing understanding of SE attack evolution, analyzing connections to emerging AI risks and examining real-world case studies.
  • Figure 3: Keyword co-occurrence analysis in social engineering research.
  • Figure 4: The 3E evolving landscapes of social engineering parallel to AI techniques across three phases of social system.
  • Figure 5: Research trends related to evolving social engineering attack in different scenarios.