Zero Day Malware Detection with Alpha: Fast DBI with Transformer Models for Real World Application
Matthew Gaber, Mohiuddin Ahmed, Helge Janicke
TL;DR
Alpha tackles zero-day malware detection by fusing Peekaboo DBI ASM-level behavior with Transformer-based modeling in a layered pipeline that filters out familiar functions and focuses on novel context. The three-layer architecture (Function Loss SVM, DistilBERT function classification, final SVM) plus per-type fine-tuning and custom tokenizers enables robust detection of truly unseen samples, achieving near-perfect accuracy for key families on short data slices. The work demonstrates high practical potential by showing that 1-minute slices can suffice in real-world settings, and it offers open access to trained models and code. Overall, Alpha represents a significant advance in proactive malware defense by leveraging authentic behavioral data and ASM semantics to outperform prior methods in zero-day detection scenarios.
Abstract
The effectiveness of an AI model in accurately classifying novel malware hinges on the quality of the features it is trained on, which in turn depends on the effectiveness of the analysis tool used. Peekaboo, a Dynamic Binary Instrumentation (DBI) tool, defeats malware evasion techniques to capture authentic behavior at the Assembly (ASM) instruction level. This behavior exhibits patterns consistent with Zipf's law, a distribution commonly seen in natural languages, making Transformer models particularly effective for binary classification tasks. We introduce Alpha, a framework for zero day malware detection that leverages Transformer models and ASM language. Alpha is trained on malware and benign software data collected through Peekaboo, enabling it to identify entirely new samples with exceptional accuracy. Alpha eliminates any common functions from the test samples that are in the training dataset. This forces the model to rely on contextual patterns and novel ASM instruction combinations to detect malicious behavior, rather than memorizing familiar features. By combining the strengths of DBI, ASM analysis, and Transformer architectures, Alpha offers a powerful approach to proactively addressing the evolving threat of malware. Alpha demonstrates perfect accuracy for Ransomware, Worms and APTs with flawless classification for both malicious and benign samples. The results highlight the model's exceptional performance in detecting truly new malware samples.