Adversarial Attacks Leverage Interference Between Features in Superposition

TL;DR

The paper addresses why adversarial examples arise by tying vulnerability to how networks compress and encode many latent features via superposition. It introduces the linear representation hypothesis (LRH) and a controlled synthetic setup to show that interference among overcomplete feature directions dictates perturbation directions and transferability. The authors validate the mechanism in a ViT trained on CIFAR-10 with an engineered bottleneck, showing that greater superposition (smaller bottleneck) yields lower robustness and higher cross-model transfer due to shared latent geometry. They also demonstrate algorithmic brittleness even with orthogonal representations through frequency-based, gradient-free attacks and discuss implications for semantically informed defenses. Overall, the work reframes adversarial vulnerability as an emergent property of representational compression, offering a mechanistic lens for designing robust, interpretation-driven defenses against adversaries that exploit latent feature interference.

Abstract

Fundamental questions remain about when and why adversarial examples arise in neural networks, with competing views characterising them either as artifacts of the irregularities in the decision landscape or as products of sensitivity to non-robust input features. In this paper, we instead argue that adversarial vulnerability can stem from efficient information encoding in neural networks. Specifically, we show how superposition - where networks represent more features than they have dimensions - creates arrangements of latent representations that adversaries can exploit. We demonstrate that adversarial perturbations leverage interference between superposed features, making attack patterns predictable from feature arrangements. Our framework provides a mechanistic explanation for two known phenomena: adversarial attack transferability between models with similar training regimes and class-specific vulnerability patterns. In synthetic settings with precisely controlled superposition, we establish that superposition suffices to create adversarial vulnerability. We then demonstrate that these findings persist in a ViT trained on CIFAR-10. These findings reveal adversarial vulnerability can be a byproduct of networks' representational compression, rather than flaws in the learning process or non-robust inputs.

Figures (24)

• Figure 1: An adversarial attack exploiting superposition geometry ($k=7$, $m=2$). (a) The original sample. (b) The adversarially perturbed sample, whose ground truth remains the same but is misclassified. The sign and magnitude of an input perturbation is determined by the configuration of latent representations. (c) The original and adversarial sample in activation space. The arrows are the column vectors of $\mathbf{W}_e$, the latent representations of the input features.
• Figure 2: Greater input correlations create more consistent geometries between initialisations, driving attack transferability from 18% (uncorrelated) to 94% (global). Error bars show standard deviations.
• Figure 3: An adversarial attack (from class 5 to class 3) does not perturb the input features for a class represented orthogonally (class 1).
• Figure 4: Left: CIFAR-10 class representation structure remains similar between models across different seeds. Right: Attack transferability and robust accuracy as bottleneck dimension is increased.
• Figure 5: Visualisations of ae in the toy model, supplementing Figure 1 from the main paper by illustrating attack mechanisms in activation space and input space under varied conditions.

Theorems & Definitions (5)

• Definition 1: Linear Representation Hypothesis (LRH)
• Definition 2: Superposition Hypothesis
• Definition 3: Adversarial attack
• Definition 4: Adversarial robustness
• Corollary 1: Interference Drives Vulnerability