Benchmarking Large Language Models for Zero-shot and Few-shot Phishing URL Detection

TL;DR

This work benchmarks zero-shot and few-shot prompting of three proprietary large language models for phishing URL detection, addressing the need for adaptable, data-efficient defenses in an AI-enabled threat landscape. Using a unified evaluation on a balanced 10,000-url dataset and additional imbalanced tests, the study demonstrates that few-shot prompting generally improves classification performance across models, with Grok-3-Beta achieving the best overall metrics (e.g., AUROC and AUPRC around 0.94–0.96). The paper provides detailed per-class analyses and ROC/PR curves, highlighting trade-offs between precision and recall and showing how prompt design influences results. These findings underscore the practical potential of prompt-based phishing URL detection in large-scale defense systems and motivate further exploration of prompt optimization and broader benchmark coverage.

Abstract

The Uniform Resource Locator (URL), introduced in a connectivity-first era to define access and locate resources, remains historically limited, lacking future-proof mechanisms for security, trust, or resilience against fraud and abuse, despite the introduction of reactive protections like HTTPS during the cybersecurity era. In the current AI-first threatscape, deceptive URLs have reached unprecedented sophistication due to the widespread use of generative AI by cybercriminals and the AI-vs-AI arms race to produce context-aware phishing websites and URLs that are virtually indistinguishable to both users and traditional detection tools. Although AI-generated phishing accounted for a small fraction of filter-bypassing attacks in 2024, phishing volume has escalated over 4,000% since 2022, with nearly 50% more attacks evading detection. At the rate the threatscape is escalating, and phishing tactics are emerging faster than labeled data can be produced, zero-shot and few-shot learning with large language models (LLMs) offers a timely and adaptable solution, enabling generalization with minimal supervision. Given the critical importance of phishing URL detection in large-scale cybersecurity defense systems, we present a comprehensive benchmark of LLMs under a unified zero-shot and few-shot prompting framework and reveal operational trade-offs. Our evaluation uses a balanced dataset with consistent prompts, offering detailed analysis of performance, generalization, and model efficacy, quantified by accuracy, precision, recall, F1 score, AUROC, and AUPRC, to reflect both classification quality and practical utility in threat detection settings. We conclude few-shot prompting improves performance across multiple LLMs.

Figures (5)

• Figure 1: Phishing URL classification methodology. Each URL $u$ is processed via zero-shot ($\mathcal{P}_\text{ZS}$) or few-shot ($\mathcal{P}_\text{FS}$) prompting. Zero-shot prompts contain task instruction $\mathcal{I}$ and query $\mathcal{Q}(u)$; few-shot prompts additionally include examples $\mathcal{E}$. The LLM $\mathcal{M}$ generates responses parsed to binary predictions $\hat{y} \in \{0,1\}$ (0=phishing, 1=legitimate). Predictions are evaluated against ground truth $y_i$ using macro-averaged accuracy, precision, recall, F1-score, AUROC, and AUPRC across all test URLs.
• Figure 2: Confusion matrices for all models under zero-shot and few-shot prompting.
• Figure 3: GPT-4o ROC (left) and PR (right) curves under zero-shot and few-shot prompting.
• Figure 4: Claude-3.7 ROC (left) and PR (right) curves under zero-shot and few-shot prompting.
• Figure 5: Grok-3-Beta ROC (left) and PR (right) curves under zero-shot and few-shot prompting.