Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robustness of Selected Learning Models under Label-Flipping Attack

Sarvagya Bhargava, Mark Stamp

TL;DR

This study investigates how ten learning models—spanning classic, boosting, and deep learning categories—hold up against label-flipping attacks in malware detection. Using the Malicia dataset, the authors compare performance under simulated label corruption from 0% to 100%, with features tailored to each model class (TF-IDF opcode features for classic/boosting models and 64×64 opcode-images for CNN-based models). The results show wide variation in robustness: the Multilayer Perceptron (MLP) achieves the best balance of high initial accuracy and resilience to label flips, while SVM also demonstrates notable robustness; GBM matches MLP in robustness in some scenarios. Conversely, GNB performs poorly, CNN and MobileNet are relatively fragile to label-flipping, and DenseNet struggles due to data scarcity. Practically, the findings guide practitioners to prefer models like MLP or GBM when defending against data-poisoning threats in malware detection, and they highlight the need for defenses and further work across additional datasets and attack modalities.

Abstract

In this paper we compare traditional machine learning and deep learning models trained on a malware dataset when subjected to adversarial attack based on label-flipping. Specifically, we investigate the robustness of Support Vector Machines (SVM), Random Forest, Gaussian Naive Bayes (GNB), Gradient Boosting Machine (GBM), LightGBM, XGBoost, Multilayer Perceptron (MLP), Convolutional Neural Network (CNN), MobileNet, and DenseNet models when facing varying percentages of misleading labels. We empirically assess the the accuracy of each of these models under such an adversarial attack on the training data. This research aims to provide insights into which models are inherently more robust, in the sense of being better able to resist intentional disruptions to the training data. We find wide variation in the robustness of the models tested to adversarial attack, with our MLP model achieving the best combination of initial accuracy and robustness.

Robustness of Selected Learning Models under Label-Flipping Attack

TL;DR

This study investigates how ten learning models—spanning classic, boosting, and deep learning categories—hold up against label-flipping attacks in malware detection. Using the Malicia dataset, the authors compare performance under simulated label corruption from 0% to 100%, with features tailored to each model class (TF-IDF opcode features for classic/boosting models and 64×64 opcode-images for CNN-based models). The results show wide variation in robustness: the Multilayer Perceptron (MLP) achieves the best balance of high initial accuracy and resilience to label flips, while SVM also demonstrates notable robustness; GBM matches MLP in robustness in some scenarios. Conversely, GNB performs poorly, CNN and MobileNet are relatively fragile to label-flipping, and DenseNet struggles due to data scarcity. Practically, the findings guide practitioners to prefer models like MLP or GBM when defending against data-poisoning threats in malware detection, and they highlight the need for defenses and further work across additional datasets and attack modalities.

Abstract

In this paper we compare traditional machine learning and deep learning models trained on a malware dataset when subjected to adversarial attack based on label-flipping. Specifically, we investigate the robustness of Support Vector Machines (SVM), Random Forest, Gaussian Naive Bayes (GNB), Gradient Boosting Machine (GBM), LightGBM, XGBoost, Multilayer Perceptron (MLP), Convolutional Neural Network (CNN), MobileNet, and DenseNet models when facing varying percentages of misleading labels. We empirically assess the the accuracy of each of these models under such an adversarial attack on the training data. This research aims to provide insights into which models are inherently more robust, in the sense of being better able to resist intentional disruptions to the training data. We find wide variation in the robustness of the models tested to adversarial attack, with our MLP model achieving the best combination of initial accuracy and robustness.
Paper Structure (24 sections, 5 figures, 1 table)

This paper contains 24 sections, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related work
  3. Background
  4. Classic Models
  5. Boosting Models
  6. Deep Learning Models
  7. Experiments and Results
  8. Dataset and Data Preprocessing
  9. Baseline Results
  10. Label-Flipping Results for Classic Models
  11. Support Vector Machine Results
  12. Random Forest Results
  13. Gaussian Naı̈ve Bayes Results
  14. Label-Flipping Results for Boosting Techniques
  15. Gradient Boosting Machine Results
  16. ...and 9 more sections

Figures (5)

  • Figure 1: Baseline accuracies without label-flipping
  • Figure 2: Accuracy, precision and recall graphs for classic ML techniques
  • Figure 3: Accuracy, precision and recall graphs for boosting techniques
  • Figure 4: Accuracy, precision and recall graphs for deep learning techniques
  • Figure 5: Accuracy, precision, and recall for all models tested