Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scam Shield: Multi-Model Voting and Fine-Tuned LLMs Against Adversarial Attacks

Chen-Wei Chang, Shailik Sarkar, Hossein Salemi, Hyungmin Kim, Shutonu Mitra, Hemant Purohit, Fengxiu Zhang, Michin Hong, Jin-Hee Cho, Chang-Tien Lu

TL;DR

This work presents a Hierarchical Scam Detection System (HSDS) that fuses a lightweight four-model voting front end with a domain-adapted LoRA-finetuned LLaMA 8B Instruct backend to robustly detect adversarial scams while reducing inference time. A curated adversarial dataset and an efficient LoRA-based fine-tuning strategy enable strong zero-shot performance, with a majority-vote ensemble that outperforms both traditional classifiers and non-finetuned LLM baselines. Experimental results show the HSDS achieves around 0.90 accuracy with high precision and recall across diverse scam types, while significantly lowering computation compared to running the LLM directly. The findings demonstrate the practicality of a hybrid, open-source framework for resilient, scalable scam detection in real-world deployments and highlight avenues for further optimization and benchmarking against state-of-the-art LLMs.

Abstract

Scam detection remains a critical challenge in cybersecurity as adversaries craft messages that evade automated filters. We propose a Hierarchical Scam Detection System (HSDS) that combines a lightweight multi-model voting front end with a fine-tuned LLaMA 3.1 8B Instruct back end to improve accuracy and robustness against adversarial attacks. An ensemble of four classifiers provides preliminary predictions through majority vote, and ambiguous cases are escalated to the fine-tuned model, which is optimized with adversarial training to reduce misclassification. Experiments show that this hierarchical design both improves adversarial scam detection and shortens inference time by routing most cases away from the LLM, outperforming traditional machine-learning baselines and proprietary LLM baselines. The findings highlight the effectiveness of a hybrid voting mechanism and adversarial fine-tuning in fortifying LLMs against evolving scam tactics, enhancing the resilience of automated scam detection systems.

Scam Shield: Multi-Model Voting and Fine-Tuned LLMs Against Adversarial Attacks

TL;DR

This work presents a Hierarchical Scam Detection System (HSDS) that fuses a lightweight four-model voting front end with a domain-adapted LoRA-finetuned LLaMA 8B Instruct backend to robustly detect adversarial scams while reducing inference time. A curated adversarial dataset and an efficient LoRA-based fine-tuning strategy enable strong zero-shot performance, with a majority-vote ensemble that outperforms both traditional classifiers and non-finetuned LLM baselines. Experimental results show the HSDS achieves around 0.90 accuracy with high precision and recall across diverse scam types, while significantly lowering computation compared to running the LLM directly. The findings demonstrate the practicality of a hybrid, open-source framework for resilient, scalable scam detection in real-world deployments and highlight avenues for further optimization and benchmarking against state-of-the-art LLMs.

Abstract

Scam detection remains a critical challenge in cybersecurity as adversaries craft messages that evade automated filters. We propose a Hierarchical Scam Detection System (HSDS) that combines a lightweight multi-model voting front end with a fine-tuned LLaMA 3.1 8B Instruct back end to improve accuracy and robustness against adversarial attacks. An ensemble of four classifiers provides preliminary predictions through majority vote, and ambiguous cases are escalated to the fine-tuned model, which is optimized with adversarial training to reduce misclassification. Experiments show that this hierarchical design both improves adversarial scam detection and shortens inference time by routing most cases away from the LLM, outperforming traditional machine-learning baselines and proprietary LLM baselines. The findings highlight the effectiveness of a hybrid voting mechanism and adversarial fine-tuning in fortifying LLMs against evolving scam tactics, enhancing the resilience of automated scam detection systems.

Paper Structure

This paper contains 24 sections, 1 equation, 2 figures, 7 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Overview of Existing Approaches.
  4. Limitations of the State-of-the-Art.
  5. System Design and Methodology
  6. Dataset for Fine-Tuning and Augmentation
  7. Generation of Adversarial Scam Messages
  8. Experimental Methodologies
  9. Fine-Tuning Method
  10. Multi-Model Voting System
  11. LLaMA-Based Scam Classification
  12. EXPERIMENTAL RESULTS & ANALYSES
  13. Evaluation settings.
  14. Headline finding.
  15. Fine-Tuning Performance Evaluation on Adversarial Scam Detection
  16. ...and 9 more sections

Figures (2)

  • Figure 1: Comparison of Original and Adversarially Modified Scam Messages with ChatGPT’s Scam Detection Response.
  • Figure 2: Overview of our hierarchical scam detection system integrating multi-model voting and fine-tuned LLaMA 8B Instruct for enhanced detection accuracy.