Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CLASP: Cost-Optimized LLM-based Agentic System for Phishing Detection

Fouad Trad, Ali Chehab

TL;DR

CLASP targets phishing detection by combining three specialized LLM-based agents (URL, screenshot, and HTML) within a progressive analysis framework to balance detection performance and cost. The approach avoids model retraining by leveraging prompt-driven LLMs and selectively escalates data processing, achieving an accuracy of 0.834, recall of 0.847, and F1 of 0.830 on a 2,126-site dataset while costing roughly $3.18 per 1,000 websites and processing each site in 2.78 seconds. It outperforms leading commercial tools in recall and F1, and the authors publicly release their dataset to spur further research. The work demonstrates the practical viability of multi-agent LLM systems for real-time phishing detection with scalable, explainable outputs from individual agents.

Abstract

Phishing websites remain a significant cybersecurity threat, necessitating accurate and cost-effective detection mechanisms. In this paper, we present CLASP, a novel system that effectively identifies phishing websites by leveraging multiple intelligent agents, built using large language models (LLMs), to analyze different aspects of a web resource. The system processes URLs or QR codes, employing specialized LLM-based agents that evaluate the URL structure, webpage screenshot, and HTML content to predict potential phishing threats. To optimize performance while minimizing operational costs, we experimented with multiple combination strategies for agent-based analysis, ultimately designing a strategic combination that ensures the per-website evaluation expense remains minimal without compromising detection accuracy. We tested various LLMs, including Gemini 1.5 Flash and GPT-4o mini, to build these agents and found that Gemini 1.5 Flash achieved the best performance with an F1 score of 83.01% on a newly curated dataset. Also, the system maintained an average processing time of 2.78 seconds per website and an API cost of around $3.18 per 1,000 websites. Moreover, CLASP surpasses leading previous solutions, achieving over 40% higher recall and a 20% improvement in F1 score for phishing detection on the collected dataset. To support further research, we have made our dataset publicly available, supporting the development of more advanced phishing detection systems.

CLASP: Cost-Optimized LLM-based Agentic System for Phishing Detection

TL;DR

CLASP targets phishing detection by combining three specialized LLM-based agents (URL, screenshot, and HTML) within a progressive analysis framework to balance detection performance and cost. The approach avoids model retraining by leveraging prompt-driven LLMs and selectively escalates data processing, achieving an accuracy of 0.834, recall of 0.847, and F1 of 0.830 on a 2,126-site dataset while costing roughly $3.18 per 1,000 websites and processing each site in 2.78 seconds. It outperforms leading commercial tools in recall and F1, and the authors publicly release their dataset to spur further research. The work demonstrates the practical viability of multi-agent LLM systems for real-time phishing detection with scalable, explainable outputs from individual agents.

Abstract

Phishing websites remain a significant cybersecurity threat, necessitating accurate and cost-effective detection mechanisms. In this paper, we present CLASP, a novel system that effectively identifies phishing websites by leveraging multiple intelligent agents, built using large language models (LLMs), to analyze different aspects of a web resource. The system processes URLs or QR codes, employing specialized LLM-based agents that evaluate the URL structure, webpage screenshot, and HTML content to predict potential phishing threats. To optimize performance while minimizing operational costs, we experimented with multiple combination strategies for agent-based analysis, ultimately designing a strategic combination that ensures the per-website evaluation expense remains minimal without compromising detection accuracy. We tested various LLMs, including Gemini 1.5 Flash and GPT-4o mini, to build these agents and found that Gemini 1.5 Flash achieved the best performance with an F1 score of 83.01% on a newly curated dataset. Also, the system maintained an average processing time of 2.78 seconds per website and an API cost of around $3.18 per 1,000 websites. Moreover, CLASP surpasses leading previous solutions, achieving over 40% higher recall and a 20% improvement in F1 score for phishing detection on the collected dataset. To support further research, we have made our dataset publicly available, supporting the development of more advanced phishing detection systems.
Paper Structure (18 sections, 3 figures, 3 tables)

This paper contains 18 sections, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background and Preliminaries
  3. Large Language Models (LLMs)
  4. Agentic AI Systems
  5. Related Work
  6. Methodology
  7. Experiments
  8. Experimental Setup
  9. Data
  10. Models
  11. Prompts
  12. Assessing Baseline Agents
  13. Majority Voting
  14. Progressive Analysis
  15. Testing with GPT-4o mini
  16. ...and 3 more sections

Figures (3)

  • Figure 1: Comparison of (a) Majority Voting and (b) Progressive Analysis.
  • Figure 2: Screenshot of a detected phishing website.
  • Figure 3: Screenshot of a legitimate website obtained from a scanned QR code.