Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Malware Detection based on API Calls: A Reproducibility Study

Juhani Merilehto

TL;DR

This paper performs an independent reproduction of a frequency-based, order-invariant API-call malware detector originally proposed by Fellicious et al., using a public dataset with 250{,}533 training and 83{,}511 test samples. By implementing Unigram, Bigram, Trigram, and Combined n-gram features and training Random Forest classifiers across 14 API-call length thresholds, the study confirms the original findings while delivering consistent performance improvements at the optimal length of 2{,}500 API calls. Across three independent runs with different seeds, the results show standard deviations below 0.5% and reproduced F1-scores that exceed the original by up to 2.57% for certain variants, with Unigram achieving the highest F1 of 0.8717. The work emphasizes open data and code, demonstrates practical deployment viability of lightweight frequency-based detection, and provides a robust baseline for future research in reproducible cybersecurity ML.

Abstract

This study independently reproduces the malware detection methodology presented by Felli cious et al. [7], which employs order-invariant API call frequency analysis using Random Forest classification. We utilized the original public dataset (250,533 training samples, 83,511 test samples) and replicated four model variants: Unigram, Bigram, Trigram, and Combined n gram approaches. Our reproduction successfully validated all key findings, achieving F1-scores that exceeded the original results by 0.99% to 2.57% across all models at the optimal API call length of 2,500. The Unigram model achieved F1=0.8717 (original: 0.8631), confirming its ef fectiveness as a lightweight malware detector. Across three independent experimental runs with different random seeds, we observed remarkably consistent results with standard deviations be low 0.5%, demonstrating high reproducibility. This study validates the robustness and scientific rigor of the original methodology while confirming the practical viability of frequency-based API call analysis for malware detection.

Malware Detection based on API Calls: A Reproducibility Study

TL;DR

This paper performs an independent reproduction of a frequency-based, order-invariant API-call malware detector originally proposed by Fellicious et al., using a public dataset with 250{,}533 training and 83{,}511 test samples. By implementing Unigram, Bigram, Trigram, and Combined n-gram features and training Random Forest classifiers across 14 API-call length thresholds, the study confirms the original findings while delivering consistent performance improvements at the optimal length of 2{,}500 API calls. Across three independent runs with different seeds, the results show standard deviations below 0.5% and reproduced F1-scores that exceed the original by up to 2.57% for certain variants, with Unigram achieving the highest F1 of 0.8717. The work emphasizes open data and code, demonstrates practical deployment viability of lightweight frequency-based detection, and provides a robust baseline for future research in reproducible cybersecurity ML.

Abstract

This study independently reproduces the malware detection methodology presented by Felli cious et al. [7], which employs order-invariant API call frequency analysis using Random Forest classification. We utilized the original public dataset (250,533 training samples, 83,511 test samples) and replicated four model variants: Unigram, Bigram, Trigram, and Combined n gram approaches. Our reproduction successfully validated all key findings, achieving F1-scores that exceeded the original results by 0.99% to 2.57% across all models at the optimal API call length of 2,500. The Unigram model achieved F1=0.8717 (original: 0.8631), confirming its ef fectiveness as a lightweight malware detector. Across three independent experimental runs with different random seeds, we observed remarkably consistent results with standard deviations be low 0.5%, demonstrating high reproducibility. This study validates the robustness and scientific rigor of the original methodology while confirming the practical viability of frequency-based API call analysis for malware detection.
Paper Structure (63 sections, 2 equations, 3 figures, 5 tables)

This paper contains 63 sections, 2 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Reproduction Goals
  3. Contributions
  4. Paper Organization
  5. Related Work
  6. Behavioral Malware Detection
  7. API Call Sequence Analysis
  8. Frequency-Based and Statistical Approaches
  9. Public Datasets and Benchmarks
  10. Reproducibility in Machine Learning
  11. Positioning This Work
  12. Reproduction Methodology
  13. Dataset
  14. Feature Engineering
  15. Unigram Features
  16. ...and 48 more sections

Figures (3)

  • Figure 1: F1-score versus API call length for all four models. The X-axis uses logarithmic scale. Unigram and Bigram show ascending trends peaking at 2,500, while Trigram degrades after length 200. The Combined model provides balanced performance across all lengths.
  • Figure 2: Precision-Recall curves for all models at 2,500 API calls. Unigram and Bigram maintain high precision across all recall levels. Trigram shows rapid precision degradation at high recall, indicating low confidence in many predictions. The Combined model balances between extremes.
  • Figure 3: Receiver Operating Characteristic curves for all models at 2,500 API calls. All models achieve AUC $>$0.98, with Unigram and Bigram approaching perfect discrimination (AUC $>$0.996). The steep initial rise indicates high true positive rates achievable at very low false positive rates.