Mitigating the Impact of Malware Evolution on API Sequence-based Windows Malware Detector
Xingyuan Wei, Ce Li, Qiujian Lv, Ning Li, Degang Sun, Yan Wang
TL;DR
This work tackles the degradation of API sequence-based Windows malware detectors caused by continual malware evolution. It introduces MME, a framework that (i) builds an API knowledge graph from Win32 API documents and applies TransE embeddings to capture API semantics, (ii) encodes API arguments to reflect system resource access via a hierarchical feature-hash representation, and (iii) employs a contrastive encoder to preserve similarity among evolved malware within a family. Empirically, MME significantly reduces false negatives and improves F1 scores across LSTM, Text-CNN, and Transformer detectors, while substantially cutting the human labeling effort required for periodic retraining. The framework demonstrates strong sustainability, robustness to evolution, and compatibility with existing detectors, offering practical gains for long-term malware defense with reduced maintenance costs.
Abstract
In dynamic Windows malware detection, deep learning models are extensively deployed to analyze API sequences. Methods based on API sequences play a crucial role in malware prevention. However, due to the continuous updates of APIs and the changes in API sequence calls leading to the constant evolution of malware variants, the detection capability of API sequence-based malware detection models significantly diminishes over time. We observe that the API sequences of malware samples before and after evolution usually have similar malicious semantics. Specifically, compared to the original samples, evolved malware samples often use the API sequences of the pre-evolution samples to achieve similar malicious behaviors. For instance, they access similar sensitive system resources and extend new malicious functions based on the original functionalities. In this paper, we propose a framework MME(Mitigating the impact of Malware Evolution), a framework that can enhance existing API sequence-based malware detectors and mitigate the adverse effects of malware evolution. To help detection models capture the similar semantics of these post-evolution API sequences, our framework represents API sequences using API knowledge graphs and system resource encodings and applies contrastive learning to enhance the model's encoder. Results indicate that, compared to regular Text-CNN, our framework can significantly reduce the false positive rate by 13.10% and improve the F1-Score by 8.47% on five years of data, achieving the best experimental results. Additionally, evaluations show that our framework can save on the human costs required for model maintenance. We only need 1% of the budget per month to reduce the false positive rate by 11.16% and improve the F1-Score by 6.44%.