LLM Whisperer: An Inconspicuous Attack to Bias LLM Responses

TL;DR

The paper investigates how inconspicuous perturbations to prompts issued by third-party providers can bias LLM outputs toward a target concept without altering the model. It develops two attack modalities—paraphrasing and synonym replacement—demonstrating just how sensitive LLMs can be to small linguistic shifts, with substantial increases in the likelihood of mentioning specific brands or societal concepts. A rigorous open-source model evaluation plus a large-scale user study show that these perturbations are largely indistinguishable to humans yet can meaningfully influence user perception and choices, highlighting risks to user autonomy in deployed chatbot systems. The work further discusses defenses, including warnings, model robustness, new bias metrics, and continuous audits, and offers an economic analysis of attack feasibility, underscoring the need for multi-pronged safeguards in prompt-based AI services.

Abstract

Writing effective prompts for large language models (LLM) can be unintuitive and burdensome. In response, services that optimize or suggest prompts have emerged. While such services can reduce user effort, they also introduce a risk: the prompt provider can subtly manipulate prompts to produce heavily biased LLM responses. In this work, we show that subtle synonym replacements in prompts can increase the likelihood (by a difference up to 78%) that LLMs mention a target concept (e.g., a brand, political party, nation). We substantiate our observations through a user study, showing that our adversarially perturbed prompts 1) are indistinguishable from unaltered prompts by humans, 2) push LLMs to recommend target concepts more often, and 3) make users more likely to notice target concepts, all without arousing suspicion. The practicality of this attack has the potential to undermine user autonomy. Among other measures, we recommend implementing warnings against using prompts from untrusted parties.

Figures (21)

• Figure 1: An unbranded chatbot service (created for illustration in the user study), closely mimicking Copilot, suggesting prompts. Popular chatbot services (e.g., ChatGPT, Meta AI, Gemini, Copilot) all employ such prompt recommendation mechanisms. Some, like Copilot copilotproduct, continuously update recommendations based on the chat history. Adversarial prompt providers may suggest specially crafted prompts. Fig. \ref{['fig:threatmodel:3stepthreat']} depicts an attack.
• Figure 2: A screenshot of a prompt library on the "Ad Campaigns" page. The prompt library explicitly asks users to "use these AI prompts to help boost your advertising strategy." Adversaries may similarly publish their prompts and execute the attack we describe in Fig. \ref{['fig:threatmodel:3stepthreat']}. This screenshot was captured at https://www.godaddy.com/resources/ai-prompts-for-ad-campaigns on Sep 10th, 2024.
• Figure 3: Pipeline of an attack where the adversaries craft prompts and persuade LLM users to try these prompts. For example, Instacart suggests prompts users can try with its ChatGPT-powered search instacart. Once persuaded, the users send these prompts to LLMs and read the responses.
• Figure 4: Pipeline of an attack where users ask adversaries to draft prompts. Users may ask prompting services to draft prompts for efficiency and utility. Users then forward the prompts to LLMs and read the responses. Companies (e.g., PromptPerfect promptperfect) offer such services.
• Figure 5: An illustration of the adversaries' goals. In this example, the adversary tries to increase the frequency of a target concept (A) through inconspicuous prompt recommendations.There are three concepts of the same category (e.g., brands of the same product): A, B, and C. Adversaries achieved this goal as concept A was recommended twice before the attack and four times after the attack. In practice, each response may recommend more than one concept.