Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Defending Against Social Engineering Attacks in the Age of LLMs

Lin Ai, Tharindu Kumarage, Amrita Bhattacharjee, Zizhou Liu, Zheng Hui, Michael Davinroy, James Cook, Laura Cassani, Kirill Trapeznikov, Matthias Kirchner, Arslan Basharat, Anthony Hoogs, Joshua Garland, Huan Liu, Julia Hirschberg

TL;DR

ConvoSentinel is proposed, a modular defense pipeline that improves detection at both the message and the conversation levels, offering enhanced adaptability and cost-effectiveness and highlights the need for advanced strategies to leverage LLMs in cybersecurity.

Abstract

The proliferation of Large Language Models (LLMs) poses challenges in detecting and mitigating digital deception, as these models can emulate human conversational patterns and facilitate chat-based social engineering (CSE) attacks. This study investigates the dual capabilities of LLMs as both facilitators and defenders against CSE threats. We develop a novel dataset, SEConvo, simulating CSE scenarios in academic and recruitment contexts, and designed to examine how LLMs can be exploited in these situations. Our findings reveal that, while off-the-shelf LLMs generate high-quality CSE content, their detection capabilities are suboptimal, leading to increased operational costs for defense. In response, we propose ConvoSentinel, a modular defense pipeline that improves detection at both the message and the conversation levels, offering enhanced adaptability and cost-effectiveness. The retrieval-augmented module in ConvoSentinel identifies malicious intent by comparing messages to a database of similar conversations, enhancing CSE detection at all stages. Our study highlights the need for advanced strategies to leverage LLMs in cybersecurity.

Defending Against Social Engineering Attacks in the Age of LLMs

TL;DR

ConvoSentinel is proposed, a modular defense pipeline that improves detection at both the message and the conversation levels, offering enhanced adaptability and cost-effectiveness and highlights the need for advanced strategies to leverage LLMs in cybersecurity.

Abstract

The proliferation of Large Language Models (LLMs) poses challenges in detecting and mitigating digital deception, as these models can emulate human conversational patterns and facilitate chat-based social engineering (CSE) attacks. This study investigates the dual capabilities of LLMs as both facilitators and defenders against CSE threats. We develop a novel dataset, SEConvo, simulating CSE scenarios in academic and recruitment contexts, and designed to examine how LLMs can be exploited in these situations. Our findings reveal that, while off-the-shelf LLMs generate high-quality CSE content, their detection capabilities are suboptimal, leading to increased operational costs for defense. In response, we propose ConvoSentinel, a modular defense pipeline that improves detection at both the message and the conversation levels, offering enhanced adaptability and cost-effectiveness. The retrieval-augmented module in ConvoSentinel identifies malicious intent by comparing messages to a database of similar conversations, enhancing CSE detection at all stages. Our study highlights the need for advanced strategies to leverage LLMs in cybersecurity.
Paper Structure (48 sections, 2 equations, 10 figures, 14 tables)

This paper contains 48 sections, 2 equations, 10 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Can LLMs Be Manipulated to Conduct CSE Attempts?
  3. SEConvo
  4. Data Generation
  5. Single-LLM Simulation
  6. Dual-Agent Interaction
  7. Data Statistics
  8. Data Annotation and Quality
  9. Are LLMs Effective Detectors of CSE?
  10. Target Agent Defense Rate
  11. LLM CSE Detection
  12. Does Message-Level Analysis Enhance CSE Detection?
  13. ConvoSentinel
  14. Conversational Context of Message-Level SI Requests
  15. RAG Integrated Snippet-Level Intent
  16. ...and 33 more sections

Figures (10)

  • Figure 1: Examples of benign vs malicious conversations with SI requests.
  • Figure 2: Data generation modes: single-LLM simulation (top) and dual-agent interaction (bottom).
  • Figure 3: Inter-annotator agreement compared to sample-level ambiguity standard deviation and sample-level maximum ambiguity values.
  • Figure 4: Distribution of samples (%) across varying values of sample-level ambiguity standard deviation and sample-level maximum ambiguity.
  • Figure 5: Distribution of deceived conversations (%) across varying degrees of ambiguity.
  • ...and 5 more figures