Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can Large Language Models Change User Preference Adversarially?

Varshini Subhash

TL;DR

The paper addresses the risk that pretrained LLMs, as dialogue agents, can adversarially influence user preferences in high-stakes interactions. It surveys interpretability and safety literature, demonstrates red-teaming on ChatGPT and GODEL, and develops a white-box attention-probing prototype to study adversarial behavior. Empirical results show persuasive prompts can elicit problematic responses in ChatGPT and reveal distinctive attention patterns under adversarial prompts in GODEL, highlighting safety gaps. The work underscores the need for stronger interpretability, robust evaluation, and proactive red-teaming to mitigate user-preference manipulation in dialogue systems.

Abstract

Pretrained large language models (LLMs) are becoming increasingly powerful and ubiquitous in mainstream applications such as being a personal assistant, a dialogue model, etc. As these models become proficient in deducing user preferences and offering tailored assistance, there is an increasing concern about the ability of these models to influence, modify and in the extreme case manipulate user preference adversarially. The issue of lack of interpretability in these models in adversarial settings remains largely unsolved. This work tries to study adversarial behavior in user preferences from the lens of attention probing, red teaming and white-box analysis. Specifically, it provides a bird's eye view of existing literature, offers red teaming samples for dialogue models like ChatGPT and GODEL and probes the attention mechanism in the latter for non-adversarial and adversarial settings.

Can Large Language Models Change User Preference Adversarially?

TL;DR

The paper addresses the risk that pretrained LLMs, as dialogue agents, can adversarially influence user preferences in high-stakes interactions. It surveys interpretability and safety literature, demonstrates red-teaming on ChatGPT and GODEL, and develops a white-box attention-probing prototype to study adversarial behavior. Empirical results show persuasive prompts can elicit problematic responses in ChatGPT and reveal distinctive attention patterns under adversarial prompts in GODEL, highlighting safety gaps. The work underscores the need for stronger interpretability, robust evaluation, and proactive red-teaming to mitigate user-preference manipulation in dialogue systems.

Abstract

Pretrained large language models (LLMs) are becoming increasingly powerful and ubiquitous in mainstream applications such as being a personal assistant, a dialogue model, etc. As these models become proficient in deducing user preferences and offering tailored assistance, there is an increasing concern about the ability of these models to influence, modify and in the extreme case manipulate user preference adversarially. The issue of lack of interpretability in these models in adversarial settings remains largely unsolved. This work tries to study adversarial behavior in user preferences from the lens of attention probing, red teaming and white-box analysis. Specifically, it provides a bird's eye view of existing literature, offers red teaming samples for dialogue models like ChatGPT and GODEL and probes the attention mechanism in the latter for non-adversarial and adversarial settings.
Paper Structure (19 sections, 4 figures)

This paper contains 19 sections, 4 figures.

Table of Contents

  1. Introduction
  2. Literature Review
  3. Adversarial Training
  4. Red Teaming.
  5. Reinforcement Learning from Human Feedback (RLHF).
  6. Adversarial Evaluation.
  7. Probing
  8. Circuit Analysis
  9. Experiments
  10. Persuasiveness in ChatGPT.
  11. Personalized Persuasion.
  12. Prototype for Analyzing Adversarial Behavior in GODEL
  13. Experimental Setup.
  14. Non-Adversarial Case.
  15. Adversarial Case.
  16. ...and 4 more sections

Figures (4)

  • Figure 1: Self-attention heatmaps in GODEL for the non-adversarial example. Rows indicate layers and columns indicate multi-attention heads. The X and Y axes for each heatmap indicates token attention weights (instruction+knowledge+dialogue = 54 tokens) plotted against each other. Blue indicates minimum attention and red indicates maximum attention.
  • Figure 2: Zoomed in attention heads for the last layer of adversarial example, depicting box-like attention pattern.
  • Figure 3: Self-attention heatmaps in GODEL for the adversarial example. Rows indicate layers and columns indicate multi-attention heads. The X and Y axes for each heatmap indicates token attention weights (instruction+knowledge+dialogue = 55 tokens) plotted against each other. Blue indicates minimum attention and red indicates maximum attention.
  • Figure :