MalCL: Leveraging GAN-Based Generative Replay to Combat Catastrophic Forgetting in Malware Classification

TL;DR

Malware classification must adapt to rapidly evolving threats without forgetting historical knowledge. The paper introduces MalCL, a GAN-based continual learning framework that uses generative replay with feature matching and sophisticated replay-sampleSelection to retain past malware family knowledge while learning new families. Across Windows and Android datasets, MalCL significantly outperforms prior GR-based methods, with best configurations yielding substantial gains and approaching joint-training upper bounds. This approach offers a practical path to robust, privacy-preserving continual learning for large-scale malware defenses and highlights the importance of domain-specific replay strategies and task construction.

Abstract

Continual Learning (CL) for malware classification tackles the rapidly evolving nature of malware threats and the frequent emergence of new types. Generative Replay (GR)-based CL systems utilize a generative model to produce synthetic versions of past data, which are then combined with new data to retrain the primary model. Traditional machine learning techniques in this domain often struggle with catastrophic forgetting, where a model's performance on old data degrades over time. In this paper, we introduce a GR-based CL system that employs Generative Adversarial Networks (GANs) with feature matching loss to generate high-quality malware samples. Additionally, we implement innovative selection schemes for replay samples based on the model's hidden representations. Our comprehensive evaluation across Windows and Android malware datasets in a class-incremental learning scenario -- where new classes are introduced continuously over multiple tasks -- demonstrates substantial performance improvements over previous methods. For example, our system achieves an average accuracy of 55% on Windows malware samples, significantly outperforming other GR-based models by 28%. This study provides practical insights for advancing GR-based malware classification systems. The implementation is available at \url {https://github.com/MalwareReplayGAN/MalCL}\footnote{The code will be made public upon the presentation of the paper}.

Figures (6)

• Figure 1: An attacker reuses legacy malware to evade systems updated with only new malwares.
• Figure 2: MalCL continual learning pipeline for malware classification using N tasks.
• Figure 3: GAN architecture: (a) Generator and (b) Discriminator, adapted for generating replay malware samples, and (c) Classifier, optimized for malware family classification in continual learning tasks.
• Figure 4: Impact of three replay sample selections on MalCL performance when using the EMBER Dataset. We further report the mean accuracies for each task when comparing BCE with FML.
• Figure 5: The number of replay sample classes per task using L1 to BMean Logits. We omit the first task, as no replay sample is utilized to train the classifier.