Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Trade-Off Between Transparency and Security in Adversarial Machine Learning

Lucas Fenaux, Christopher Srinivasa, Florian Kerschbaum

TL;DR

The paper addresses whether transparency about defense status conflicts with security in adversarial ML by combining a large-scale empirical study of transferable adversarial examples with game-theoretic analysis. It shows that attackers benefit when their surrogate-defense choice matches the defender’s, implying that revealing defense status (transparency) can harm security under transfer attacks. Through Nash and Stackelberg formulations, the work demonstrates the security cost of transparency and emphasizes that defense diversification and obfuscation can improve robustness. The findings challenge the notion that transparency always strengthens Responsible AI and highlight the need for more nuanced security strategies in adversarial settings.

Abstract

Transparency and security are both central to Responsible AI, but they may conflict in adversarial settings. We investigate the strategic effect of transparency for agents through the lens of transferable adversarial example attacks. In transferable adversarial example attacks, attackers maliciously perturb their inputs using surrogate models to fool a defender's target model. These models can be defended or undefended, with both players having to decide which to use. Using a large-scale empirical evaluation of nine attacks across 181 models, we find that attackers are more successful when they match the defender's decision; hence, obscurity could be beneficial to the defender. With game theory, we analyze this trade-off between transparency and security by modeling this problem as both a Nash game and a Stackelberg game, and comparing the expected outcomes. Our analysis confirms that only knowing whether a defender's model is defended or not can sometimes be enough to damage its security. This result serves as an indicator of the general trade-off between transparency and security, suggesting that transparency in AI systems can be at odds with security. Beyond adversarial machine learning, our work illustrates how game-theoretic reasoning can uncover conflicts between transparency and security.

On the Trade-Off Between Transparency and Security in Adversarial Machine Learning

TL;DR

The paper addresses whether transparency about defense status conflicts with security in adversarial ML by combining a large-scale empirical study of transferable adversarial examples with game-theoretic analysis. It shows that attackers benefit when their surrogate-defense choice matches the defender’s, implying that revealing defense status (transparency) can harm security under transfer attacks. Through Nash and Stackelberg formulations, the work demonstrates the security cost of transparency and emphasizes that defense diversification and obfuscation can improve robustness. The findings challenge the notion that transparency always strengthens Responsible AI and highlight the need for more nuanced security strategies in adversarial settings.

Abstract

Transparency and security are both central to Responsible AI, but they may conflict in adversarial settings. We investigate the strategic effect of transparency for agents through the lens of transferable adversarial example attacks. In transferable adversarial example attacks, attackers maliciously perturb their inputs using surrogate models to fool a defender's target model. These models can be defended or undefended, with both players having to decide which to use. Using a large-scale empirical evaluation of nine attacks across 181 models, we find that attackers are more successful when they match the defender's decision; hence, obscurity could be beneficial to the defender. With game theory, we analyze this trade-off between transparency and security by modeling this problem as both a Nash game and a Stackelberg game, and comparing the expected outcomes. Our analysis confirms that only knowing whether a defender's model is defended or not can sometimes be enough to damage its security. This result serves as an indicator of the general trade-off between transparency and security, suggesting that transparency in AI systems can be at odds with security. Beyond adversarial machine learning, our work illustrates how game-theoretic reasoning can uncover conflicts between transparency and security.

Paper Structure

This paper contains 17 sections, 4 equations, 2 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Adversarial example attacks
  4. Adversarial example defenses
  5. Related Work
  6. Experiments
  7. Setup
  8. Evaluation
  9. Observations
  10. Other notable observations
  11. Importance of the robustness of the surrogate model
  12. Game-theoretic analysis
  13. The Trade-off between Transparency and Security
  14. Attack and Surrogate Game
  15. Implications of Optimal Play
  16. ...and 2 more sections

Figures (2)

  • Figure 1: Adversarial accuracy against white-box AutoAttack of the 71 CIFAR10 and the 21 ImageNet defended models, with the worst, median, and best performing models highlighted.
  • Figure 2: Transferability of adversarial examples from surrogate models of varying robustness. Models A–J are random representatives from successive deciles of defended CIFAR-10 models ranked by white-box AutoAttack adversarial accuracy (A = worst, J = best). Each surrogate is used to attack all other defended models with nine attacks. We also include the correlation between each attack and the adversarial accuracy below the main plot.