Black-box Optimization of LLM Outputs by Asking for Directions

TL;DR

This work tackles black-box attacks on text-only LLM APIs by leveraging the models' own introspective capabilities. It replaces fragile absolute confidence scores with calibrated binary comparisons to guide a hill-climbing optimization that perturbs inputs toward the attacker’s goal. The authors demonstrate the approach across three attack modes—adversarial examples for vision-LLMs, prompt injections, and jailbreaks—across multiple models, showing competitive or superior performance to proxy-based methods, with larger models often more vulnerable due to better calibration. The study highlights a security paradox and discusses defenses, underscoring the need for robust safeguards in deployed LLM systems.

Abstract

We present a novel approach for attacking black-box large language models (LLMs) by exploiting their ability to express confidence in natural language. Existing black-box attacks require either access to continuous model outputs like logits or confidence scores (which are rarely available in practice), or rely on proxy signals from other models. Instead, we demonstrate how to prompt LLMs to express their internal confidence in a way that is sufficiently calibrated to enable effective adversarial optimization. We apply our general method to three attack scenarios: adversarial examples for vision-LLMs, jailbreaks and prompt injections. Our attacks successfully generate malicious inputs against systems that only expose textual outputs, thereby dramatically expanding the attack surface for deployed LLMs. We further find that better and larger models exhibit superior calibration when expressing confidence, creating a concerning security paradox where model capability improvements directly enhance vulnerability. Our code is available at this [link](https://github.com/zj-jayzhang/black_box_llm_optimization).

Figures (9)

• Figure 1: An illustration of our optimization for adversarial examples for vision-LLMs. Moreover, our general method can be applied to many applications, including jailbreaks and prompt injections.
• Figure 2: We evaluate our text-only output optimization attack on samples from three datasets: ImageNet imagenet_cvpr09, AgentDojo debenedetti2024agentdojo, and AdvBench zou2023universal. Results are reported on both Claude and GPT models.
• Figure 3: Validation of optimization signals extracted from Qwen2.5-VL-72B-Instruct for adversarial examples. Left: Explicit confidence queries ("How confident are you this image contains a dog? Rate 1–100") yield uninformative extreme values (mostly 0, 5, or 95) that fail to distinguish between adversarial and benign samples, providing no useful optimization signal. Right: Our binary comparison approach ("Is the 2nd image less likely to contain a dog?") produces true positives that align with ground-truth; these correct signals are sufficient to guide updates in the right direction.
• Figure 4: Query templates for jailbreak and prompt injection using our binary comparison approach.
• Figure 5: Targeted adversarial examples on vision-LLMs.