Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Large Language Models as Misleading Assistants in Conversation

Betty Li Hou, Kejian Shi, Jason Phang, James Aung, Steven Adler, Rosie Campbell

TL;DR

This paper investigates whether large language models can act as misleading assistants in conversation by simulating a reading-comprehension task between two LLMs. The authors vary Assistant configurations (Truthful, Subtle Lying, Wrong Answer) and User information access (No Passage, Summary, Excerpt) and evaluate on the QuALITY dataset across 500 trials, showing GPT-4-based assistants can significantly mislead others, with up to 23% accuracy drops. They also find that providing additional context to the User mitigates deception, while the deception remains a substantial risk across settings, underscoring safety concerns and the need for detection and mitigation of AI-driven manipulation. The work contributes a controlled methodology and empirical benchmarks for AI deception in conversational settings, informing policy, safety research, and system design.

Abstract

Large Language Models (LLMs) are able to provide assistance on a wide range of information-seeking tasks. However, model outputs may be misleading, whether unintentionally or in cases of intentional deception. We investigate the ability of LLMs to be deceptive in the context of providing assistance on a reading comprehension task, using LLMs as proxies for human users. We compare outcomes of (1) when the model is prompted to provide truthful assistance, (2) when it is prompted to be subtly misleading, and (3) when it is prompted to argue for an incorrect answer. Our experiments show that GPT-4 can effectively mislead both GPT-3.5-Turbo and GPT-4, with deceptive assistants resulting in up to a 23% drop in accuracy on the task compared to when a truthful assistant is used. We also find that providing the user model with additional context from the passage partially mitigates the influence of the deceptive model. This work highlights the ability of LLMs to produce misleading information and the effects this may have in real-world situations.

Large Language Models as Misleading Assistants in Conversation

TL;DR

This paper investigates whether large language models can act as misleading assistants in conversation by simulating a reading-comprehension task between two LLMs. The authors vary Assistant configurations (Truthful, Subtle Lying, Wrong Answer) and User information access (No Passage, Summary, Excerpt) and evaluate on the QuALITY dataset across 500 trials, showing GPT-4-based assistants can significantly mislead others, with up to 23% accuracy drops. They also find that providing additional context to the User mitigates deception, while the deception remains a substantial risk across settings, underscoring safety concerns and the need for detection and mitigation of AI-driven manipulation. The work contributes a controlled methodology and empirical benchmarks for AI deception in conversational settings, informing policy, safety research, and system design.

Abstract

Large Language Models (LLMs) are able to provide assistance on a wide range of information-seeking tasks. However, model outputs may be misleading, whether unintentionally or in cases of intentional deception. We investigate the ability of LLMs to be deceptive in the context of providing assistance on a reading comprehension task, using LLMs as proxies for human users. We compare outcomes of (1) when the model is prompted to provide truthful assistance, (2) when it is prompted to be subtly misleading, and (3) when it is prompted to argue for an incorrect answer. Our experiments show that GPT-4 can effectively mislead both GPT-3.5-Turbo and GPT-4, with deceptive assistants resulting in up to a 23% drop in accuracy on the task compared to when a truthful assistant is used. We also find that providing the user model with additional context from the passage partially mitigates the influence of the deceptive model. This work highlights the ability of LLMs to produce misleading information and the effects this may have in real-world situations.
Paper Structure (24 sections, 4 figures, 5 tables)

This paper contains 24 sections, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Assistant Configurations
  5. User Configurations
  6. API-based Models and Prompts
  7. Dataset
  8. Metrics and Baselines
  9. Results
  10. Qualitative Analysis
  11. Limitations and Future Work
  12. Conclusion
  13. Raw Results
  14. Base Prompts
  15. User Prompts
  16. ...and 9 more sections

Figures (4)

  • Figure 1: The User model attempts a reading comprehension task with limited access to the passage and asks clarifying questions to the Assistant model. The Assistant is provided the full passage as well as an answer to the question, and is instructed to sway the User towards an incorrect answer.
  • Figure 2: User accuracy on QuALITY questions across treatments with GPT-4 Assistant, with 95% confidence. * signifies 100 trials ran as opposed to 500.
  • Figure 3: Absolute drop in User accuracy in the Subtle Lying and Wrong Answer treatments from User accuracy in the Truthful treatment.
  • Figure 4: Success of the Wrong Answer Assistant configuration in persuading the User model to pick the designated wrong answer, with 95% confidence.