GreedyPixel: Fine-Grained Black-Box Adversarial Attack Via Greedy Algorithm

TL;DR

<3-5 sentence high-level summary> GreedyPixel introduces a fine-grained black-box adversarial attack that performs brute-force-style, per-pixel updates guided by a surrogate-derived priority map and refined by target-model query feedback. By combining a training-free, coordinate-wise greedy search with monotonic loss reduction, it achieves near white-box precision while maintaining pixel-sparsity and perceptual quality across CNNs and Vision Transformers on CIFAR-10 and ImageNet. The method bridges the gap between query-efficient, transferable black-box attacks and gradient-based white-box attacks, demonstrating strong attack success rates, robustness to defenses, and applicability to real-world vision APIs. Extensive ablations show the importance of the priority map, update cadence, and surrogate quality in balancing efficiency, sparsity, and perceptual fidelity.

Abstract

Deep neural networks are highly vulnerable to adversarial examples, which are inputs with small, carefully crafted perturbations that cause misclassification -- making adversarial attacks a critical tool for evaluating robustness. Existing black-box methods typically entail a trade-off between precision and flexibility: pixel-sparse attacks (e.g., single- or few-pixel attacks) provide fine-grained control but lack adaptability, whereas patch- or frequency-based attacks improve efficiency or transferability, but at the cost of producing larger and less precise perturbations. We present GreedyPixel, a fine-grained black-box attack method that performs brute-force-style, per-pixel greedy optimization guided by a surrogate-derived priority map and refined by means of query feedback. It evaluates each coordinate directly without any gradient information, guaranteeing monotonic loss reduction and convergence to a coordinate-wise optimum, while also yielding near white-box-level precision and pixel-wise sparsity and perceptual quality. On the CIFAR-10 and ImageNet datasets, spanning convolutional neural networks (CNNs) and Transformer models, GreedyPixel achieved state-of-the-art success rates with visually imperceptible perturbations, effectively bridging the gap between black-box practicality and white-box performance. The implementation is available at https://github.com/azrealwang/greedypixel.

Figures (9)

• Figure 1: Adversarial examples generated by GreedyPixel, which perturbs only a small, high-saliency subset of pixels. This approach preserves visual fidelity while achieving near white-box attack success rates using only black-box feedback.
• Figure 2: Pixel-wise priority map obtained by ranking gradient $g$ of the CW loss function $\ell$ with respect to input $x$ and ground-truth label $y$ within a surrogate model $Z_s$. Replacing the surrogate model with the target model yields a white-box attack.
• Figure 3: CW loss vs. number of iterations for PGD madry2018towards, GreedyFool$^*$, and GreedyPixel variants. GreedyFool$^*$ uses GreedyFool's pixel-selection heuristic (top-50% salient pixels) dong2020greedyfool but replaces its gradient-descent update with a pixel-wise exhaustive search, ensuring a fair comparison under black-box assumptions. However, GreedyFool$^*$ converges to a higher loss plateau, revealing the limitations of its pixel-selection strategy. GreedyPixel achieves the lowest final loss, with white-box access converging fastest, followed by adversarially trained (AT) surrogate and random surrogate settings. The dotted line indicates the first-iteration label change.
• Figure 4: Comparison of adversarial examples generated by different attack algorithms in terms of visual quality and perturbation appearance. The proposed GreedyPixel method achieved the highest visual quality by selectively modifying only critical pixels. The perturbations were normalized for visualization using \ref{['eq_normalization']}, with bold indicating superior performance. Notably, while the BruSLe attack vo2024brusleattack introduced fewer perturbed pixels than GreedyPixel, the visibility of these perturbations was higher due to their spatial placement.
• Figure 5: Results of GreedyPixel attacks on commercial online vision APIs. Within a strict budget of 400 queries (perturbing fewer than 0.1% of pixels, i.e.,$\leq 50$ pixels under unlimited $\epsilon$), the confidence scores of the ground-truth class were significantly reduced—sometimes to zero. These results demonstrate that GreedyPixel remains effective even in real-world, production scenarios.