Malware families discovery via Open-Set Recognition on Android manifest permissions

TL;DR

This work tackles the problem of detecting novel Android malware families using manifest permissions by extending open-set recognition to tree-based models. It couples a closed-set Gradient Boosting classifier with a MaxLogit-based novelty detector, applying a threshold $\tau$ on the maximum logit $\max(\mathbf{z}_{\mathbf{p}})$ to flag novel samples while maintaining seamless integration with existing pipelines. The approach is evaluated on the public Drebin dataset and a proprietary dataset, showing strong closed-set classification and effective novelty detection, with advantages over a nearest-neighbor OSR baseline in both accuracy and speed. In real-world deployment, the system achieves around 83% accuracy on telemetry data and demonstrates practical novelty detection capabilities, including discovering new malware variants, while highlighting the importance of threshold tuning and careful handling of false alarms for human analysts.

Abstract

Malware are malicious programs that are grouped into families based on their penetration technique, source code, and other characteristics. Classifying malware programs into their respective families is essential for building effective defenses against cyber threats. Machine learning models have a huge potential in malware detection on mobile devices, as malware families can be recognized by classifying permission data extracted from Android manifest files. Still, the malware classification task is challenging due to the high-dimensional nature of permission data and the limited availability of training samples. In particular, the steady emergence of new malware families makes it impossible to acquire a comprehensive training set covering all the malware classes. In this work, we present a malware classification system that, on top of classifying known malware, detects new ones. In particular, we combine an open-set recognition technique developed within the computer vision community, namely MaxLogit, with a tree-based Gradient Boosting classifier, which is particularly effective in classifying high-dimensional data. Our solution turns out to be very practical, as it can be seamlessly employed in a standard classification workflow, and efficient, as it adds minimal computational overhead. Experiments on public and proprietary datasets demonstrate the potential of our solution, which has been deployed in a business environment.

Figures (6)

• Figure 1: Depiction of our proposed open-set classifier $\mathcal{K}$.
• Figure 2: Analogy between the Logits vector in a neural network-based classifier (a) and the decision values in a tree-based Gradient Boosting classifier (b). In both cases, the vector $\vb*{z} _{ \vb*{p} }$ represents the raw output values before applying the softmax function $\sigma( \vb*{z} _{ \vb*{p} })$.
• Figure 3: Imbalance in the cardinality $\#$ of each malware family $\ell$ in our proprietary dataset, where malware families with less than $10$ samples are represented by pale-colored bars.
• Figure 4: Recall confusion matrix of the open-set classifier $\mathcal{K}$, on the public (first row) and proprietary (last row) datasets, when (a),(d) instances of the less populous classes are grouped in the dummy class $others$ and treated as $novel$, and (b),(e) each class is sequentially treated as $novel$ in the leave-one-class-out process, along with the associated novelty detection ROC curves (c),(f).
• Figure 5: Novelty detection ROC curves for OSNN on the public dataset, with each class treated as $novel$ in the leave-one-class-out process.