ScamFerret: Detecting Scam Websites Autonomously with Large Language Models
Hiroki Nakano, Takashi Koide, Daiki Chiba
TL;DR
ScamFerret addresses the rising difficulty of detecting sophisticated scam websites across multiple languages without relying on large labeled datasets. It employs an autonomous, tool-augmented LLM agent that autonomously collects data from a target URL, integrates external information (DNS, WHOIS, reviews, etc.), and iteratively reasons with a ReAct-inspired framework to classify scams and provide explanations. The system achieves high accuracy (0.972 for four English scam types and 0.993 across English, German, and Japanese online shopping sites with GPT-4) and outperforms traditional detectors, while offering transparent rationale and a ground-truth dataset of 2,400 URLs. This work demonstrates the practical potential of LLM-powered, explainable web security and outlines future directions for cost reduction and multimodal extensions.
Abstract
With the rise of sophisticated scam websites that exploit human psychological vulnerabilities, distinguishing between legitimate and scam websites has become increasingly challenging. This paper presents ScamFerret, an innovative agent system employing a large language model (LLM) to autonomously collect and analyze data from a given URL to determine whether it is a scam. Unlike traditional machine learning models that require large datasets and feature engineering, ScamFerret leverages LLMs' natural language understanding to accurately identify scam websites of various types and languages without requiring additional training or fine-tuning. Our evaluation demonstrated that ScamFerret achieves 0.972 accuracy in classifying four scam types in English and 0.993 accuracy in classifying online shopping websites across three different languages, particularly when using GPT-4. Furthermore, we confirmed that ScamFerret collects and analyzes external information such as web content, DNS records, and user reviews as necessary, providing a basis for identifying scam websites from multiple perspectives. These results suggest that LLMs have significant potential in enhancing cybersecurity measures against sophisticated scam websites.