Confidence Elicitation: A New Attack Vector for Large Language Models

TL;DR

This work introduces Confidence Elicitation Attacks (CEAttacks), a novel black-box adversarial framework for large language models that leverages elicited model confidence as feedback to guide word-substitution perturbations. By modeling uncertainty through Dirichlet aggregation and using a two-step prompting protocol to obtain verbal or numerical confidence signals, CEAttacks can efficiently find high-impact perturbations within a constrained semantic budget. Empirical results across SST-2, AG-News, and StrategyQA on LLaMa-3-8B-Instruct and Mistral-7B-Instruct show state-of-the-art hard-label attack performance with fewer queries and better semantic preservation, and even apply the approach to API models like GPT-4o with robust calibration. The findings highlight potential vulnerabilities in calibrated LLMs under black-box constraints and suggest defense avenues such as adversarial training and calibration-aware robustness techniques, while contributing open-source code to the community.

Abstract

A fundamental issue in deep learning has been adversarial robustness. As these systems have scaled, such issues have persisted. Currently, large language models (LLMs) with billions of parameters suffer from adversarial attacks just like their earlier, smaller counterparts. However, the threat models have changed. Previously, having gray-box access, where input embeddings or output logits/probabilities were visible to the user, might have been reasonable. However, with the introduction of closed-source models, no information about the model is available apart from the generated output. This means that current black-box attacks can only utilize the final prediction to detect if an attack is successful. In this work, we investigate and demonstrate the potential of attack guidance, akin to using output probabilities, while having only black-box access in a classification setting. This is achieved through the ability to elicit confidence from the model. We empirically show that the elicited confidence is calibrated and not hallucinated for current LLMs. By minimizing the elicited confidence, we can therefore increase the likelihood of misclassification. Our new proposed paradigm demonstrates promising state-of-the-art results on three datasets across two models (LLaMA-3-8B-Instruct and Mistral-7B-Instruct-V0.3) when comparing our technique to existing hard-label black-box attack methods that introduce word-level substitutions.

Figures (6)

• Figure 1: Confidence elicitation attack on an LLM, assuming a classification task (Start), $x$ has a ground truth $y = false$, we first perform inference and extract the model's prediction $\hat{y}$ and original elicited confidence $\mathbf{p}_{\mathcal{C}}$ (green, Step 1), we use the confidence as feedback (Step 2) to determine whether a perturbation $\delta$, modelled after a word substitution "use" $\rightarrow$ "utilize" (Step 3) added to the input leads to lower confidence (yellow, Step 4), we carry on adding $\delta$s to the input $x \rightarrow x_{adv}$ until we achieve a misclassification (red, End).
• Figure 2: Confidence Elicitation Attack on an LLM: Left) SSPAttack and other previous hard-label attacks first (A) perform multiple $\delta$ word substitutions, so that a heavily perturbed sample is misclassified. Then they (B)/(C) perform further optimization to improve the adversarial sample's quality. Right) In contrast, CEAttacks take a bottom-up approach by progressively perturbing the original sample with $\delta$ word substitutions until a misclassification is achieved, using model guidance through probability approximations. The adversarial perturbation is bounded by $\epsilon$ to preserve its quality.
• Figure 3: Reliability plots. Top) We show the SST2, AG-News and StrategyQA on LLama 3 8B Instruct calibration plots. Bottom) The ROC curves. The diagonal line is the optimal calibration.
• Figure 4: Ablation study on $|W|$ and $|S|$
• Figure 5: Ternary plots highlighting the attack path for the example in Table \ref{['tab:example_analysis']}. As the algorithm adds adversarial word substitutions the model's predictions and associated confidences to such predictions change leading to a different Dirichlet distribution profile