MAD-OOD: A Deep Learning Cluster-Driven Framework for an Out-of-Distribution Malware Detection and Classification

TL;DR

The paper tackles out-of-distribution malware detection by addressing intra-family variation from polymorphic and metamorphic malware. It introduces MAD-OOD, a two-stage framework that first uses Gaussian Discriminant Analysis to establish spherical class boundaries in embedding space and detect OOD, then refines predictions with a second-stage multi-input network that fuses cluster-derived signals with refined embeddings. A Z-score based, multi-boundary analysis enhances in-distribution confidence across classes. Empirical results on datasets with 25 known families and novel OOD variants show superior OOD detection and a peak AUC of 0.911 on unseen malware, underscoring its practical scalability and interpretability.

Abstract

Out of distribution (OOD) detection remains a critical challenge in malware classification due to the substantial intra family variability introduced by polymorphic and metamorphic malware variants. Most existing deep learning based malware detectors rely on closed world assumptions and fail to adequately model this intra class variation, resulting in degraded performance when confronted with previously unseen malware families. This paper presents MADOOD, a novel two stage, cluster driven deep learning framework for robust OOD malware detection and classification. In the first stage, malware family embeddings are modeled using class conditional spherical decision boundaries derived from Gaussian Discriminant Analysis (GDA), enabling statistically grounded separation of indistribution and OOD samples without requiring OOD data during training. Z score based distance analysis across multiple class centroids is employed to reliably identify anomalous samples in the latent space. In the second stage, a deep neural network integrates cluster based predictions, refined embeddings, and supervised classifier outputs to enhance final classification accuracy. Extensive evaluations on benchmark malware datasets comprising 25 known families and multiple novel OOD variants demonstrate that MADOOD significantly outperforms state of the art OOD detection methods, achieving an AUC of up to 0.911 on unseen malware families. The proposed framework provides a scalable, interpretable, and statistically principled solution for real world malware detection and anomaly identification in evolving cybersecurity environments.

Figures (3)

• Figure 1: Class conditional probability through Gaussian Discriminant Analysislee2020multi
• Figure 2: Confusion Matrix on sample test set containing previously unseen out-of-Distribution variants
• Figure 3: ROC Performance Evaluation of the Proposed Framework on the prediction of Previously Unseen Out-of-Distribution Classification