Counteracting Concept Drift by Learning with Future Malware Predictions

TL;DR

This work tackles concept drift in malware detection, including adversarial concept drift, by evaluating two predictive strategies: adversarial training and GAN-based predictions. It formalizes time-based train/test splits, introduces an UpperBound baseline, and uses Ember2018 and Avast Internal2019 data to compare methods. Adversarial training offers robustness to adversarial samples but provides limited improvements for future data, whereas GAN-based Conditional CycleGANs (CCyGANs) can predict future distribution changes and yield notable gains for drift-prone malware families, especially when integrated as additional training data. The findings suggest practical value in CCyGAN-based drift prediction to enhance malware detectors, with potential for broader application to other domains facing distribution shifts.

Abstract

The accuracy of deployed malware-detection classifiers degrades over time due to changes in data distributions and increasing discrepancies between training and testing data. This phenomenon is known as the concept drift. While the concept drift can be caused by various reasons in general, new malicious files are created by malware authors with a clear intention of avoiding detection. The existence of the intention opens a possibility for predicting such future samples. Including predicted samples in training data should consequently increase the accuracy of the classifiers on new testing data. We compare two methods for predicting future samples: (1) adversarial training and (2) generative adversarial networks (GANs). The first method explicitly seeks for adversarial examples against the classifier that are then used as a part of training data. Similarly, GANs also generate synthetic training data. We use GANs to learn changes in data distributions within different time periods of training data and then apply these changes to generate samples that could be in testing data. We compare these prediction methods on two different datasets: (1) Ember public dataset and (2) the internal dataset of files incoming to Avast. We show that while adversarial training yields more robust classifiers, this method is not a good predictor of future malware in general. This is in contrast with previously reported positive results in different domains (including natural language processing and spam detection). On the other hand, we show that GANs can be successfully used as predictors of future malware. We specifically examine malware families that exhibit significant changes in their data distributions over time and the experimental results confirm that GAN-based predictions can significantly improve the accuracy of the classifier on new, previously unseen data.

Figures (9)

• Figure 1: Schema of our approach: Conditional Cycle GANs (CCyGANs) $G_1,\ldots,G_{k-1}$ are trained to predict malware samples from subsequent time periods and then used to generated predictions from training data. The predictions are then combined with original training data. Testing data are from the time periods following the training window.
• Figure 2: Architecture of neural networks used for experiments with full feature set. All layers are fully connected (denoted fc), with batch normalization, selu activation function, and dropout regularization (with parameter set to $0.2$).
• Figure 3: FPR per testing period using natural samples (normal model) and adversarial samples (FGSM and PGD models) for Ember2018 (left subfigure) and Internal2019 dataset. The shaded regions visualize the standard error of the mean.
• Figure 4: Model degradation over multiple time periods for Ember2018 and the Internal2019 dataset. Each model is trained for a time period ending at the designated month/week and the TPRs for all subsequent time periods are visualized. The shaded regions correspond to the standard error of the mean.
• Figure 5: Comparison of TPRs on generated adversarial samples on Internal2019 data between normal training and adversarial training using FGSM (left subfigure) and PGD (right subfigure). The results confirm that adversarial training improves robustness of models w.r.t. generated adversarial samples. The shaded regions correspond to the standard error of the mean.