SecureLearn -- An Attack-agnostic Defense for Multiclass Machine Learning Against Data Poisoning Attacks

TL;DR

SecureLearn tackles data poisoning in multiclass classification by introducing a two-layer defense that is attack-agnostic: (i) data sanitization to relabel and prune poisoned data, and (ii) feature-oriented adversarial training (FORT) to harden decision boundaries. It demonstrates robustness across RF, DT, GNB, and MLP on IRIS, MNIST, and USPS under OOP, SubP, and RLPA attacks, maintaining accuracy above $90\%$ and high recall/F1, with MLP achieving up to $97\%$ recall/F1. A novel 3D evaluation matrix assesses performance across data poisoning, sanitization, and adversarial training, and SecureLearn consistently outperforms existing defenses. The results indicate strong generalization to traditional multiclass ML and neural networks, offering practical resilience without requiring attack knowledge or extra data.

Abstract

Data poisoning attacks are a potential threat to machine learning (ML) models, aiming to manipulate training datasets to disrupt their performance. Existing defenses are mostly designed to mitigate specific poisoning attacks or are aligned with particular ML algorithms. Furthermore, most defenses are developed to secure deep neural networks or binary classifiers. However, traditional multiclass classifiers need attention to be secure from data poisoning attacks, as these models are significant in developing multi-modal applications. Therefore, this paper proposes SecureLearn, a two-layer attack-agnostic defense to defend multiclass models from poisoning attacks. It comprises two components of data sanitization and a new feature-oriented adversarial training. To ascertain the effectiveness of SecureLearn, we proposed a 3D evaluation matrix with three orthogonal dimensions: data poisoning attack, data sanitization and adversarial training. Benchmarking SecureLearn in a 3D matrix, a detailed analysis is conducted at different poisoning levels (10%-20%), particularly analysing accuracy, recall, F1-score, detection and correction rates, and false discovery rate. The experimentation is conducted for four ML algorithms, namely Random Forest (RF), Decision Tree (DT), Gaussian Naive Bayes (GNB) and Multilayer Perceptron (MLP), trained with three public datasets, against three poisoning attacks and compared with two existing mitigations. Our results highlight that SecureLearn is effective against the provided attacks. SecureLearn has strengthened resilience and adversarial robustness of traditional multiclass models and neural networks, confirming its generalization beyond algorithm-specific defenses. It consistently maintained accuracy above 90%, recall and F1-score above 75%. For neural networks, SecureLearn achieved 97% recall and F1-score against all selected poisoning attacks.

Figures (5)

• Figure 1: Architectural overview of SecureLearn illustrating a two-layer approach to secure the training pipeline of ML models irrespective of data poisoning attacks
• Figure 2: 3D evaluation matrix to evaluate SecureLearn from three different aspects
• Figure 3: Impact of OOP attack on accuracy at various poisoning levels. The first row illustrates all models trained with the IRIS dataset, the models in the second row are trained with the MNIST dataset, and in the third row, the models are trained with the USPS dataset
• Figure 4: Impact of SubP attack on accuracy at various poisoning levels. The first row illustrates models trained with the IRIS dataset, the models in the second row are trained with the MNIST dataset, and the models in the third row are trained with the USPS dataset
• Figure 5: Impact of RLP attack on accuracy at various poisoning levels. The first row illustrates models trained with the IRIS dataset, the models in the second row are trained with the MNIST dataset, and the models in the third row are trained with the USPS dataset