R+R: Revisiting Static Feature-Based Android Malware Detection using Machine Learning

TL;DR

This work targets reproducibility gaps in static-feature Android malware detection using ML. It rigorously evaluates six models on two datasets (Drebin, APIGraph) under offline and continuous active learning, with careful deduplication, hyperparameter tuning, and multi-seed reporting. The key finding is that well-tuned tree-based models like XGBoost often outperform neural nets once duplicates are removed, challenging assumptions about neural superiority in this domain. The authors provide an open-source benchmarking framework to promote transparent, reproducible security research and robust baselines for future malware-detection studies.

Abstract

Static feature-based Android malware detection using machine learning (ML) remains critical due to its scalability and efficiency. However, existing approaches often overlook security-critical reproducibility concerns, such as dataset duplication, inadequate hyperparameter tuning, and variance from random initialization. This can significantly compromise the practical effectiveness of these systems. In this paper, we systematically investigate these challenges by proposing a more rigorous methodology for model selection and evaluation. Using two widely used datasets, Drebin and APIGraph, we evaluate six ML models of varying complexity under both offline and continuous active learning settings. Our analysis demonstrates that, contrary to popular belief, well-tuned, simpler models, particularly tree-based methods like XGBoost, consistently outperform more complex neural networks, especially when duplicates are removed. To promote transparency and reproducibility, we open-source our codebase, which is extensible for integrating new models and datasets, facilitating reproducible security research.

Figures (16)

• Figure 1: Performance comparison of the best and worst performing neural network models for Android malware detection using the state-of-the-art continuous active learning method from chen2023continuous. Top: F1-score. Bottom: False Negative Rate (FNR). Models were initialized with 5 different random seeds. The average F1-score over months differs by 10.6%, and the False Negative Rate (FNR) differs by 13.9% between the two models, despite using the same hyperparameters. This highlights reproducibility challenges in machine learning research for Android malware detection.
• Figure 2: Fraction of unique samples retained after the deduplication process
• Figure 3: F1-score over the test months on the APIGraph for (left) deduplicated (right) duplicated datasets
• Figure 4: Performance in test months on the Drebin datasets for Merged Training setting
• Figure 5: Frequency of unique and duplicate malware samples. Left: APIGraph in November 2015 with 190 of 488 samples sharing identical features. Right: Drebin in January 2021, with 348 of 385 samples duplicates of the same input.