Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Benchmark Datasets for Machine Learning Based Website Phishing Detection: An experimental study

Abdelhakim Hannousse, Salima Yahiouche

TL;DR

This work tackles the lack of standardized benchmarks for ML-based website phishing detection by proposing a general scheme to construct reproducible and extensible datasets and by building a representative dataset with 87 features spanning URL-based, content-based, and external-based classes. Through a series of experiments on this dataset, Random Forest consistently emerges as the most predictive classifier, with external-service features providing the strongest discriminative power and hybrid feature sets achieving top accuracies around 96–97%. Feature selection via filter methods (notably chi-square with incremental removal) improves performance beyond wrappers, while combining models trained on different feature classes does not beat a single, well-tuned hybrid model. The study also demonstrates practical applicability by releasing dataset/code and illustrating a Chrome plugin, highlighting the approach’s potential to standardize comparisons and track phishing tactics over time.

Abstract

In this paper, we present a general scheme for building reproducible and extensible datasets for website phishing detection. The aim is to (1) enable comparison of systems using different features, (2) overtake the short-lived nature of phishing websites, and (3) keep track of the evolution of phishing tactics. For experimenting the proposed scheme, we start by adopting a refined classification of website phishing features and we systematically select a total of 87 commonly recognized ones, we classify them, and we made them subjects for relevance and runtime analysis. We use the collected set of features to build a dataset in light of the proposed scheme. Thereafter, we use a conceptual replication approach to check the genericity of former findings for the built dataset. Specifically, we evaluate the performance of classifiers on individual classes and on combinations of classes, we investigate different combinations of models, and we explore the effects of filter and wrapper methods on the selection of discriminative features. The results show that Random Forest is the most predictive classifier. Features gathered from external services are found the most discriminative where features extracted from web page contents are found less distinguishing. Besides external service based features, some web page content features are found time consuming and not suitable for runtime detection. The use of hybrid features provided the best accuracy score of 96.61%. By investigating different feature selection methods, filter-based ranking together with incremental removal of less important features improved the performance up to 96.83% better than wrapper methods.

Towards Benchmark Datasets for Machine Learning Based Website Phishing Detection: An experimental study

TL;DR

This work tackles the lack of standardized benchmarks for ML-based website phishing detection by proposing a general scheme to construct reproducible and extensible datasets and by building a representative dataset with 87 features spanning URL-based, content-based, and external-based classes. Through a series of experiments on this dataset, Random Forest consistently emerges as the most predictive classifier, with external-service features providing the strongest discriminative power and hybrid feature sets achieving top accuracies around 96–97%. Feature selection via filter methods (notably chi-square with incremental removal) improves performance beyond wrappers, while combining models trained on different feature classes does not beat a single, well-tuned hybrid model. The study also demonstrates practical applicability by releasing dataset/code and illustrating a Chrome plugin, highlighting the approach’s potential to standardize comparisons and track phishing tactics over time.

Abstract

In this paper, we present a general scheme for building reproducible and extensible datasets for website phishing detection. The aim is to (1) enable comparison of systems using different features, (2) overtake the short-lived nature of phishing websites, and (3) keep track of the evolution of phishing tactics. For experimenting the proposed scheme, we start by adopting a refined classification of website phishing features and we systematically select a total of 87 commonly recognized ones, we classify them, and we made them subjects for relevance and runtime analysis. We use the collected set of features to build a dataset in light of the proposed scheme. Thereafter, we use a conceptual replication approach to check the genericity of former findings for the built dataset. Specifically, we evaluate the performance of classifiers on individual classes and on combinations of classes, we investigate different combinations of models, and we explore the effects of filter and wrapper methods on the selection of discriminative features. The results show that Random Forest is the most predictive classifier. Features gathered from external services are found the most discriminative where features extracted from web page contents are found less distinguishing. Besides external service based features, some web page content features are found time consuming and not suitable for runtime detection. The use of hybrid features provided the best accuracy score of 96.61%. By investigating different feature selection methods, filter-based ranking together with incremental removal of less important features improved the performance up to 96.83% better than wrapper methods.

Paper Structure

This paper contains 19 sections, 13 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Scheme for constructing reproducible and extensible datasets
  4. Dataset preparation
  5. URL collection
  6. Feature extraction
  7. Dataset generation
  8. Experiments
  9. Classifier evaluation
  10. Feature selection methods evaluation
  11. Experiment V: Feature extraction runtime analysis
  12. Experimental results
  13. Experiment I. Performance of classifiers trained on different classes of features
  14. Experiment II. Performance of combined models each trained on a different class of features
  15. Experiment III. Performance by training on features selected using filter ranking methods
  16. ...and 4 more sections

Figures (13)

  • Figure 1: Number of publications per year focusing on the use of machine learning for the detection of phishing websites. The results are taken the 16th of September 2020 from https://app.dimensions.ai using the following query search: "website" AND "phishing detection" AND "machine learning".
  • Figure 2: Dataset construction process.
  • Figure 3: Classification of antiphishing features.
  • Figure 4: Identification process for best classifier(s) and best combination of feature classes.
  • Figure 5: Combination of models trained on different classes of features.
  • ...and 8 more figures