ICL-EVADER: Zero-Query Black-Box Evasion Attacks on In-Context Learning and Their Defenses

TL;DR

This work addresses the security of in-context learning under a realistic zero-query black-box threat model. It introduces three novel evasion strategies—Fake Claim, Template, and Needle-in-a-Haystack—that drastically degrade ICL classifier performance across sentiment, toxicity, and illicit promotion tasks, outperforming traditional NLP attacks that rely on model queries. The authors systematically explore defense primitives and demonstrate that a joint defense combining adversarial demonstrations, cautionary warnings, and prompt structure randomization can nearly eliminate attack success while preserving accuracy within a 5% loss. They operationalize these insights into an automated defense tool and provide open-source code to facilitate practical hardening of ICL systems, highlighting both vulnerabilities and actionable mitigations for robust ICL deployments.

Abstract

In-context learning (ICL) has become a powerful, data-efficient paradigm for text classification using large language models. However, its robustness against realistic adversarial threats remains largely unexplored. We introduce ICL-Evader, a novel black-box evasion attack framework that operates under a highly practical zero-query threat model, requiring no access to model parameters, gradients, or query-based feedback during attack generation. We design three novel attacks, Fake Claim, Template, and Needle-in-a-Haystack, that exploit inherent limitations of LLMs in processing in-context prompts. Evaluated across sentiment analysis, toxicity, and illicit promotion tasks, our attacks significantly degrade classifier performance (e.g., achieving up to 95.3% attack success rate), drastically outperforming traditional NLP attacks which prove ineffective under the same constraints. To counter these vulnerabilities, we systematically investigate defense strategies and identify a joint defense recipe that effectively mitigates all attacks with minimal utility loss (<5% accuracy degradation). Finally, we translate our defensive insights into an automated tool that proactively fortifies standard ICL prompts against adversarial evasion. This work provides a comprehensive security assessment of ICL, revealing critical vulnerabilities and offering practical solutions for building more robust systems. Our source code and evaluation datasets are publicly available at: https://github.com/ChaseSecurity/ICL-Evader .

Figures (13)

• Figure 1: Performance of ICL classifiers. Accuracy (top row) and False Positive Rate (FPR, bottom row) across different numbers of demonstrations (shots) and foundation models for the three classification tasks. Key observations: (1) Performance improves with more shots but exhibits diminishing returns, peaking around 16-32 shots rather than at the maximum (128). (2) Model performance is task-dependent, with no single model dominating across all tasks.
• Figure 2: Attack Success Rate (ASR) and Relative Attack Success Rate (rASR) across the number of fake claims for the three classification tasks. The first row shows ASR trends, while the second row shows R-ASR trends. In each plot, the two lines differ in the insertion position of the fake claims. Also, across these experiments, we use the default claim candidate that is of a medium degree of assertion.
• Figure 3: Attack Success Rate (ASR) across six fake claim options for the three classification tasks. The first row shows ASR trends when claims are inserted at the beginning of the test sample, while the second row shows ASR trends when claims are inserted at the end. Each figure contains six lines representing the five individual fake claim options and one mixed option.
• Figure 4: Effectiveness of the Template Attack across number of distracting demonstrations and varying attacking prefixes. The metrics are Attack Success Rate (ASR) and Relative Attack Success Rate (rASR) evaluated on the three classification tasks. The first row shows ASR trends, while the second row shows rASR trends. Each plot contains multiple lines representing different attacking prefix configurations. In these experiments, the label names are set to be different from the ICL template under attack. Also, the location of the original test sample is fixed at the beginning of the demonstrations ($l = 1$).
• Figure 5: Effectiveness of the Template Attack across insertion location of the original test sample, as measured by ASR and rASR. The experimental settings are identical to those in Figure \ref{['fig:asr-rasr-vs-demos']}, except that the insertion location of the original test sample is fixed at the middle rather than the beginning of the demonstrations. Notably, the Template Attack demonstrates a high degree of insensitivity to the insertion location of the test sample, consistently achieving strong attack performance.