Evaluating the Effectiveness and Robustness of Visual Similarity-based Phishing Detection Models

TL;DR

This work conducts the first comprehensive, large-scale evaluation of seven visual similarity–based phishing detectors using a real-world dataset of over 451k phishing sites, revealing substantial performance gaps relative to curated benchmarks and exposing vulnerabilities to adversarial logo manipulations. By standardizing brand knowledge across models with base and extended reference lists and testing under both visible and perturbation-based logo manipulations, the study identifies critical weaknesses in logo-dependent approaches, such as reliance on static reference lists and limited generalization to unlearned brands. The findings argue for robust, multi-modal defenses that integrate OCR, adversarial training, and preprocessing, and demonstrate the value of open science through public datasets, code, and retrained models to accelerate improvements in phishing detection. The work emphasizes practical implications for deploying visual similarity–based defenses in real-world settings and provides concrete guidance for enhancing robustness against evasion tactics.

Abstract

Phishing attacks pose a significant threat to Internet users, with cybercriminals elaborately replicating the visual appearance of legitimate websites to deceive victims. Visual similarity-based detection systems have emerged as an effective countermeasure, but their effectiveness and robustness in real-world scenarios have been underexplored. In this paper, we comprehensively scrutinize and evaluate the effectiveness and robustness of popular visual similarity-based anti-phishing models using a large-scale dataset of 451k real-world phishing websites. Our analyses of the effectiveness reveal that while certain visual similarity-based models achieve high accuracy on curated datasets in the experimental settings, they exhibit notably low performance on real-world datasets, highlighting the importance of real-world evaluation. Furthermore, we find that the attackers evade the detectors mainly in three ways: (1) directly attacking the model pipelines, (2) mimicking benign logos, and (3) employing relatively simple strategies such as eliminating logos from screenshots. To statistically assess the resilience and robustness of existing models against adversarial attacks, we categorize the strategies attackers employ into visible and perturbation-based manipulations and apply them to website logos. We then evaluate the models' robustness using these adversarial samples. Our findings reveal potential vulnerabilities in several models, emphasizing the need for more robust visual similarity techniques capable of withstanding sophisticated evasion attempts. We provide actionable insights for enhancing the security of phishing defense systems, encouraging proactive actions.

Figures (6)

• Figure 1: Examples of Original Login Form and Adversarial Manipulation. An attacker changes the textual logo ('facebook') to its upper case ('FACEBOOK') and its font. The (b) example is found in our real-world phishing dataset.
• Figure 2: Overview of Our Experiment. We collect real-world phishing and benign websites (). We prepare two reference datasets (and ). We then carefully select seven popular visual similarity-based anti-phishing models and re-train them using the prepared datasets (). The effectiveness and robustness of these models are systematically evaluated (, , and ).
• Figure 3: Analysis of a Facebook phishing logo using LIME. Black means negative or no contributions.
• Figure 4: Perturbated Logos Cropped by Faster-RCNN.
• Figure 5: Text Logo Case (Original logo and benign URL).