Toward More Generalized Malicious URL Detection Models

TL;DR

The paper shows that state-of-the-art malicious URL detectors suffer strong generalization gaps when faced with open-domain daily queries due to token-level data bias. It introduces Adversarial Debiasing Feature Embedding (ADE), which uses a Bias Predictor and gradient reversal to minimize the mutual information between biased token signals and the embedding, yielding invariant representations. ADE substantially improves out-of-domain and time-drift generalization for CNN and RNN backbones (e.g., URLNet, MalConv, LSTM) across URL/HTML/EXE tasks, with large AUC gains and strong statistical significance. This approach offers more robust, bias-resistant security ML models and highlights the need for bias-aware evaluation and interpretability in real-world malware detection systems.

Abstract

This paper reveals a data bias issue that can severely affect the performance while conducting a machine learning model for malicious URL detection. We describe how such bias can be identified using interpretable machine learning techniques, and further argue that such biases naturally exist in the real world security data for training a classification model. We then propose a debiased training strategy that can be applied to most deep-learning based models to alleviate the negative effects from the biased features. The solution is based on the technique of self-supervised adversarial training to train deep neural networks learning invariant embedding from biased data. We conduct a wide range of experiments to demonstrate that the proposed strategy can lead to significantly better generalization capability for both CNN-based and RNN-based detection models.

Figures (6)

• Figure 1: The blue bars represents the distribution of token class distribution entropy in IMDB movie review dataset for binary sentiment classification. The orange bars represents token class distribution entropy in ISCX-URL-2016 URL dataset. The low class-level entropy in URL dataset shows that most token in URL dataset only appears in one specific class.
• Figure 2: We demonstrate three url samples with the visualized interpretation of URLNet. Red refers to the attribution of the malicious class, and Blue refers to the benign class. The first example shows unreasonable highlights on URL-encoded tokens. The second and third example highlights domain names. The highlights are strongly correlated with the tokens' class-level distribution, which is shown in table \ref{['tab:interpret_ratio']}. These three samples can be found on VirusTotal.
• Figure 3: We calculate the entropy of class distribution for each token that appears in the malicious class and the benign class. $1.0$ means the token only appears uniformly in different class and $0.0$ means the token only appears in certain class. The histogram shows severe data bias in all four categories.
• Figure 4: The overall architecture of the adversarial debiasing training procedure.
• Figure 5: The performance of URLNet trained on VirusTotal over time according to a sliding window on the timeline. URLNet with ADE surpasses the other by a large margin at nearly all times.