Use as Many Surrogates as You Want: Selective Ensemble Attack to Unleash Transferability without Sacrificing Resource Efficiency

TL;DR

The paper identifies a fundamental trade-off in transfer-based adversarial attacks: increasing transferability by using more surrogate models typically worsens resource efficiency when models are kept identical across iterations. It introduces Selective Ensemble Attack (SEA), a method that decouples within-iteration and cross-iteration diversity by dynamically selecting diverse surrogates from a pool of pre-trained models across iterations while keeping the per-iteration model count fixed. SEA achieves higher transferability than traditional ensemble attacks and other advanced strategies, demonstrated on ImageNet and in real-world scenarios such as commercial vision APIs and LVLMs, with about an 8.5% average gains under similar resource budgets. The approach offers a practical, adaptable framework for balancing transferability and efficiency and can potentially extend to domains beyond image classification.

Abstract

In surrogate ensemble attacks, using more surrogate models yields higher transferability but lower resource efficiency. This practical trade-off between transferability and efficiency has largely limited existing attacks despite many pre-trained models are easily accessible online. In this paper, we argue that such a trade-off is caused by an unnecessary common assumption, i.e., all models should be \textit{identical} across iterations. By lifting this assumption, we can use as many surrogates as we want to unleash transferability without sacrificing efficiency. Concretely, we propose Selective Ensemble Attack (SEA), which dynamically selects diverse models (from easily accessible pre-trained models) across iterations based on our new interpretation of decoupling within-iteration and cross-iteration model diversity. In this way, the number of within-iteration models is fixed for maintaining efficiency, while only cross-iteration model diversity is increased for higher transferability. Experiments on ImageNet demonstrate the superiority of SEA in various scenarios. For example, when dynamically selecting 4 from 20 accessible models, SEA yields 8.5% higher transferability than existing attacks under the same efficiency. The superiority of SEA also generalizes to real-world systems, such as commercial vision APIs and large vision-language models. Overall, SEA opens up the possibility of adaptively balancing transferability and efficiency according to specific resource requirements.

Figures (14)

• Figure 1: Our Selective Ensemble Attack (SEA) vs. conventional ensemble attacks (Ens) given $s$ easily accessible pre-trained models but restricted resources allowing only $m$ models per iteration. Our SEA($s$,$m$) dynamically selects $m$diverse models across iterations compared to identical models in Ens($s$,$m$), leading to higher transferability. Here $m=3$ is used for illustration.
• Figure 2: Attack success rates (%) of MI transferring from an ensemble of $k$ models to eight different target models. The result for TI in the Appendix \ref{['sec:addit-exp']} shows similar patterns.
• Figure 3: Attack success rate (%) of Ens(20,$m$) vs. SEA(20,$m$) when within-iteration model $m$ varies from 1 to 20. Results are reported for eight transfer baselines and averaged over eight target models.
• Figure 4: SEA adversarial images on Google and Baidu Cloud Vision APIs. The ground truth label of the original image is "snail".
• Figure 5: Attack success rate (%) of SEA ($s$, 4) by varying $s$ from 4 to 40.