Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Large Multimodal Agents for Accurate Phishing Detection with Enhanced Token Optimization and Cost Reduction

Fouad Trad, Ali Chehab

TL;DR

This work evaluates large multimodal models for phishing detection by analyzing both URLs and webpage screenshots via APIs, showing that multimodal signals yield higher accuracy than either modality alone. To address API costs, it introduces a two-tier agentic approach where an initial URL-only assessment is followed by a full URL+image analysis only if needed, achieving substantial cost reductions with minimal performance loss. Across two models, Gemini 1.5 Flash and GPT-4o mini, the multimodal strategy achieves up to 94% accuracy, while the agentic approach retains near-equivalent performance (92–93% accuracy) at a fraction of the token cost. The study provides detailed cost analysis, demonstrating significant scalability advantages for real-world deployment, and discusses implications for future research including model ensembles and real-time, cost-aware phishing detection systems.

Abstract

With the rise of sophisticated phishing attacks, there is a growing need for effective and economical detection solutions. This paper explores the use of large multimodal agents, specifically Gemini 1.5 Flash and GPT-4o mini, to analyze both URLs and webpage screenshots via APIs, thus avoiding the complexities of training and maintaining AI systems. Our findings indicate that integrating these two data types substantially enhances detection performance over using either type alone. However, API usage incurs costs per query that depend on the number of input and output tokens. To address this, we propose a two-tiered agentic approach: initially, one agent assesses the URL, and if inconclusive, a second agent evaluates both the URL and the screenshot. This method not only maintains robust detection performance but also significantly reduces API costs by minimizing unnecessary multi-input queries. Cost analysis shows that with the agentic approach, GPT-4o mini can process about 4.2 times as many websites per $100 compared to the multimodal approach (107,440 vs. 25,626), and Gemini 1.5 Flash can process about 2.6 times more websites (2,232,142 vs. 862,068). These findings underscore the significant economic benefits of the agentic approach over the multimodal method, providing a viable solution for organizations aiming to leverage advanced AI for phishing detection while controlling expenses.

Large Multimodal Agents for Accurate Phishing Detection with Enhanced Token Optimization and Cost Reduction

TL;DR

This work evaluates large multimodal models for phishing detection by analyzing both URLs and webpage screenshots via APIs, showing that multimodal signals yield higher accuracy than either modality alone. To address API costs, it introduces a two-tier agentic approach where an initial URL-only assessment is followed by a full URL+image analysis only if needed, achieving substantial cost reductions with minimal performance loss. Across two models, Gemini 1.5 Flash and GPT-4o mini, the multimodal strategy achieves up to 94% accuracy, while the agentic approach retains near-equivalent performance (92–93% accuracy) at a fraction of the token cost. The study provides detailed cost analysis, demonstrating significant scalability advantages for real-world deployment, and discusses implications for future research including model ensembles and real-time, cost-aware phishing detection systems.

Abstract

With the rise of sophisticated phishing attacks, there is a growing need for effective and economical detection solutions. This paper explores the use of large multimodal agents, specifically Gemini 1.5 Flash and GPT-4o mini, to analyze both URLs and webpage screenshots via APIs, thus avoiding the complexities of training and maintaining AI systems. Our findings indicate that integrating these two data types substantially enhances detection performance over using either type alone. However, API usage incurs costs per query that depend on the number of input and output tokens. To address this, we propose a two-tiered agentic approach: initially, one agent assesses the URL, and if inconclusive, a second agent evaluates both the URL and the screenshot. This method not only maintains robust detection performance but also significantly reduces API costs by minimizing unnecessary multi-input queries. Cost analysis shows that with the agentic approach, GPT-4o mini can process about 4.2 times as many websites per $100 compared to the multimodal approach (107,440 vs. 25,626), and Gemini 1.5 Flash can process about 2.6 times more websites (2,232,142 vs. 862,068). These findings underscore the significant economic benefits of the agentic approach over the multimodal method, providing a viable solution for organizations aiming to leverage advanced AI for phishing detection while controlling expenses.

Paper Structure

This paper contains 27 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background and Preliminaries
  3. Large Language Models (LLMs)
  4. Large Multimodal Models (LMMs)
  5. Agentic AI systems
  6. Phishing Detection
  7. Related Work
  8. Methodology
  9. URL-Based Detection
  10. Image-Based Detection
  11. Multimodal Analysis (URL + Image)
  12. Agentic Approach
  13. Experiments
  14. Experimental Setup
  15. Data
  16. ...and 12 more sections

Figures (3)

  • Figure 1: URL-based, Image-based, and Multimodal (URL+image) approaches for Phishing detection
  • Figure 2: Agentic Approach: two Agents Operating Sequentially for Phishing Detection
  • Figure 3: Comparison of performance metrics across the four approaches for both models