Noise as a Double-Edged Sword: Reinforcement Learning Exploits Randomized Defenses in Neural Networks

TL;DR

The results suggest that in some cases, noise-based defenses can inadvertently create an adversarial training loop beneficial to the RL attacker, highlighting the need for a more nuanced approach to defensive strategies in adversarial machine learning, particularly in safety-critical applications.

Abstract

This study investigates a counterintuitive phenomenon in adversarial machine learning: the potential for noise-based defenses to inadvertently aid evasion attacks in certain scenarios. While randomness is often employed as a defensive strategy against adversarial examples, our research reveals that this approach can sometimes backfire, particularly when facing adaptive attackers using reinforcement learning (RL). Our findings show that in specific cases, especially with visually noisy classes, the introduction of noise in the classifier's confidence values can be exploited by the RL attacker, leading to a significant increase in evasion success rates. In some instances, the noise-based defense scenario outperformed other strategies by up to 20\% on a subset of classes. However, this effect was not consistent across all classifiers tested, highlighting the complexity of the interaction between noise-based defenses and different models. These results suggest that in some cases, noise-based defenses can inadvertently create an adversarial training loop beneficial to the RL attacker. Our study emphasizes the need for a more nuanced approach to defensive strategies in adversarial machine learning, particularly in safety-critical applications. It challenges the assumption that randomness universally enhances defense against evasion attacks and highlights the importance of considering adaptive, RL-based attackers when designing robust defense mechanisms.

Figures (14)

• Figure 1: (a) Original image correctly classified with 100% confidence. (b) Modified image misclassified with 57.6% confidence after a single pixel modification. (c) True Class 9 image.
• Figure 2: Example of a physical-world adversarial attack on stop signs. Left: A stop sign with actual graffiti. Right: A stop sign with carefully designed stickers applied to mimic graffiti, capable of fooling image classifiers. These stickers represent a real-world adversarial attack, not digital manipulation. (Image source: eykholt2018robust)
• Figure 3: Flow chart of the RL environment, showing the interaction between the agent, the environment, and the CNN classifier.
• Figure 4: Averaged LSRs over the 43 classes for different attack scenarios across classifiers.
• Figure 5: Averaged AAF over the 43 classes for different attack scenarios across classifiers.