S$^4$ST: A Strong, Self-transferable, faSt, and Simple Scale Transformation for Transferable Targeted Attack

TL;DR

The paper tackles targeted transferability in data-free TTAs under strict black-box constraints by introducing two blind estimators—surrogate self-alignment and self-transferability—to evaluate basic transformations without extra data. Building on these insights, it designs S$^4$ST, a scaling-centered transformation with Base, Aug, and Block components, and optimizes its parameters via Bayesian search to maximize average self-transferability across 12 transforms. On ImageNet-Compatible, S$^4$ST achieves a peak average targeted transferability of $77.7\%$ (tSuc) and demonstrates strong transfer to real-world APIs and vision-language models, outperforming state-of-the-art data-free and data-reliant TTAs while requiring no additional data. The results highlight the unique potency of simple scaling, reveal redundancies among geometric and color transformations, and underscore the need to reevaluate data dependencies in TTAs for both offensive assessment and defense design.

Abstract

Transferable Targeted Attacks (TTAs), which aim to deceive black-box models into predicting specific erroneous labels, face significant challenges due to severe overfitting to surrogate models. Although modifying image features to generate robust semantic patterns of the target class is a promising approach, existing methods heavily rely on large-scale additional data. This dependence undermines the fair evaluation of TTA threats, potentially leading to a false sense of security or unnecessary overreactions. In this paper, we introduce two blind measures, surrogate self-alignment and self-transferability, to analyze the effectiveness and correlations of basic transformations, to enhance data-free attacks under strict black-box constraints. Our findings challenge conventional assumptions: (1) Attacking simple scaling transformations uniquely enhances targeted transferability, outperforming other basic transformations and rivaling leading complex methods; (2) Geometric and color transformations exhibit high internal redundancy despite weak inter-category correlations. These insights drive the design and tuning of S4ST (Strong, Self-transferable, faSt, Simple Scale Transformation), which integrates dimensionally consistent scaling, complementary low-redundancy transformations, and block-wise operations. Extensive experiments on the ImageNet-Compatible dataset demonstrate that S4ST achieves a 77.7% average targeted success rate (tSuc), surpassing existing transformations (+17.2% over H-Aug with only 26% computational time) and SOTA TTA solutions (+6.2% over SASD-WS with 1.2M samples for post-training). Notably, it attains 69.6% and 55.3% average tSuc against three commercial APIs and vision-language models, respectively. This work establishes a new SOTA for TTAs, highlights their potential threats, and calls for a reevaluation of the data dependency in achieving targeted transferability.

Figures (18)

• Figure 1: Comparison against existing transformation methods at incremental attack iterations. Compared with the previous SOTA transformation, H-Aug wei2023rethinking, S$^4$ST yields an absolute improvement of 14.2% on average tSuc to 83.0%, and a relative reduction of 74.3% on time consumption to 1.45s. Additionally, compared with the previous setting ($T=300$), more attack iterations benefit a lot for potent transformations like our S$^4$ST.
• Figure 2: Illustration of the original image and its transformed versions.
• Figure 3: Scatter diagrams depicting relationships between black-box transferability and (a) diversity, (b) attention deviation, and (c) gradient magnitude. As indicated by the Pearson Correlation Coefficients (PCCs) in the top left corner of each figure, all three metrics exhibit weak correlation to transferability in the cross-method case, impeding their application in identifying what transformations are effective and designing a complex transformation from scratch.
• Figure 4: Alignment between the surrogate model and black-box models on targeted AEs, which reflects the black-box transferability resulted by different transformations (see Table \ref{['rn50_trans']}). The vanilla gradient update, i.e., TMI, quickly misaligns the surrogate from black-box models and leads to poor transferability.
• Figure 5: The colored numbers indicate the cosine similarity between representations generated by the surrogate model (ResNet-50) and the black-box model (ConvNeXt), where the AE $\bm{x}^{adv}$ is created based on the TMI algorithm. On the clean sample, these two models exhibit a high degree of alignment. However, AEs with weak transferability have limited impact on the black-box model, which suggests that we can introduce an image transformation $\mathcal{T}$ to make $\bm{\phi}(\mathcal{T}(\bm{x}^{adv}))$ more closely approximate $\bm{\phi}(\bm{x})$ (self-alignment), thereby enhancing the approximation to $\bm{\psi}(\bm{x}^{adv})$ (alignment) and improving transferability.