Applying Graph Analysis for Unsupervised Fast Malware Fingerprinting
ElMouatez Billah Karbab, Mourad Debbabi
TL;DR
TrapNet introduces a scalable, unsupervised malware clustering framework that relies on static analysis and a learned similarity network. Central to the approach is FloatHash, a compact embedding derived from opcode and function-invocation sequences, produced via two variants and dimensionality reduction. The framework builds a similarity graph and applies fast graph community detection to identify malware families, achieving impressive speed and competitive purity on large real-world datasets (e.g., 250k samples in ~12 minutes with 82% purity). The results demonstrate strong scalability and practical utility for triage workflows, outperforming prior static-analysis baselines in throughput while maintaining meaningful clustering quality.
Abstract
Malware proliferation is increasing at a tremendous rate, with hundreds of thousands of new samples identified daily. Manual investigation of such a vast amount of malware is an unrealistic, time-consuming, and overwhelming task. To cope with this volume, there is a clear need to develop specialized techniques and efficient tools for preliminary filtering that can group malware based on semantic similarity. In this paper, we propose TrapNet, a novel, scalable, and unsupervised framework for malware fingerprinting and grouping. TrapNet employs graph community detection techniques for malware fingerprinting and family attribution based on static analysis, as follows: (1) TrapNet detects packed binaries and unpacks them using known generic packer tools. (2) From each malware sample, it generates a digest that captures the underlying semantics. Since the digest must be dense, efficient, and suitable for similarity checking, we designed FloatHash (FH), a novel numerical fuzzy hashing technique that produces a short real-valued vector summarizing the underlying assembly items and their order. FH is based on applying Principal Component Analysis (PCA) to ordered assembly items (e.g., opcodes, function calls) extracted from the malware's assembly code. (3) Representing malware with short numerical vectors enables high-performance, large-scale similarity computation, which allows TrapNet to build a malware similarity network. (4) Finally, TrapNet employs state-of-the-art community detection algorithms to identify dense communities, which represent groups of malware with similar semantics. Our extensive evaluation of TrapNet demonstrates its effectiveness in terms of the coverage and purity of the detected communities, while also highlighting its runtime efficiency, which outperforms other state-of-the-art solutions.