Q-FAKER: Query-free Hard Black-box Attack via Controlled Generation

TL;DR

Q-faker introduces aQuery-free Hard Black-box Attacker that sidesteps target-model access by training a lightweight surrogate on a different dataset and applying controlled generation to craft adversarial text. The method updates the surrogate-informed output distribution of a GPT-2 generator via adversarial gradients, then samples adversarial tokens using a post-norm fusion with $\lambda=0.97$, preserving fluency while achieving high transferability. Evaluated on Advbench across eight victim models and four tasks, Q-faker achieves strong attack success with minimal queries and demonstrates robust transferability and high-quality adversarial examples, often outperforming black-box and hard black-box baselines. The approach offers practical implications for evaluating model vulnerability in restrictive environments, while also detailing limitations such as the need to know the target task and the absence of infinite-query scenarios. Overall, Q-faker provides a principled, efficient, and transferable framework for hard black-box adversarial text generation with practical security relevance.

Abstract

Many adversarial attack approaches are proposed to verify the vulnerability of language models. However, they require numerous queries and the information on the target model. Even black-box attack methods also require the target model's output information. They are not applicable in real-world scenarios, as in hard black-box settings where the target model is closed and inaccessible. Even the recently proposed hard black-box attacks still require many queries and demand extremely high costs for training adversarial generators. To address these challenges, we propose Q-faker (Query-free Hard Black-box Attacker), a novel and efficient method that generates adversarial examples without accessing the target model. To avoid accessing the target model, we use a surrogate model instead. The surrogate model generates adversarial sentences for a target-agnostic attack. During this process, we leverage controlled generation techniques. We evaluate our proposed method on eight datasets. Experimental results demonstrate our method's effectiveness including high transferability and the high quality of the generated adversarial examples, and prove its practical in hard black-box settings.

Figures (5)

• Figure 1: The process of Q-faker has three main steps: (1) Training the surrogate model using a different dataset for the same task as the target model; (2) Updating the language model using adversarial gradients from the surrogate model; and (3) Generating controlled adversarial examples from the updated language model.
• Figure 2: Results of ablation study on Jigsaw. The solid lines () represent ASR (left y-axis), the dashed lines () represent USE (right y-axis).
• Figure 3: Comparison of ASR according to the number of queries (from 20 to 1). The red star($\star$) is our method. As the number of queries accessible to the target model becomes more restricted, the ASR of baseline methods drops to near zero. This demonstrates the superiority of our method in real-world scenarios with limited queries.
• Figure 4: Consistent attack capabilities on various target models. This heatmap illustrates the difference ratio in ASR between BERT-base and other models. We compare our method (left) with CT-GAT (right). Colors closer to green indicate higher ASR on other target models, highlighting the high transferability.
• Figure 5: Comparison of ASR according to the number of queries (from 20 to 1). The red star($\star$) is our method. As the number of queries accessible to the target model becomes more restricted, the ASR of baseline methods drops to near zero. This demonstrates the superiority of our method in real-world scenarios with limited queries.