Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fool Me, Fool Me: User Attitudes Toward LLM Falsehoods

Diana Bar-Or Nirman, Ariel Weizman, Amos Azaria

TL;DR

This study investigates how users prefer LLM-produced falsehoods under different presentation formats, specifically marked versus unmarked truths and uninformative truth versus confident falsehoods. Four experiments using ChatGPT responses (with one- and two-phase designs) across 300 participants reveal a consistent tendency to favor unmarked and falsehood-based outputs, challenging assumptions about natural preference for truth. The findings imply that user feedback, when used to train or fine-tune models (e.g., via RLHF), could unintentionally promote false information unless preferences are verified or weighted appropriately. The work highlights ethical and practical implications for aligning LLM behavior with user preferences and suggests verification mechanisms or reliability assessments before incorporating feedback into training pipelines.

Abstract

While Large Language Models (LLMs) have become central tools in various fields, they often provide inaccurate or false information. This study examines user preferences regarding falsehood responses from LLMs. Specifically, we evaluate preferences for LLM responses where false statements are explicitly marked versus unmarked responses and preferences for confident falsehoods compared to LLM disclaimers acknowledging a lack of knowledge. Additionally, we investigate how requiring users to assess the truthfulness of statements influences these preferences. Surprisingly, 61\% of users prefer unmarked falsehood responses over marked ones, and 69\% prefer confident falsehoods over LLMs admitting lack of knowledge. In all our experiments, a total of 300 users participated, contributing valuable data to our analysis and conclusions. When users are required to evaluate the truthfulness of statements, preferences for unmarked and falsehood responses decrease slightly but remain high. These findings suggest that user preferences, which influence LLM training via feedback mechanisms, may inadvertently encourage the generation of falsehoods. Future research should address the ethical and practical implications of aligning LLM behavior with such preferences.

Fool Me, Fool Me: User Attitudes Toward LLM Falsehoods

TL;DR

This study investigates how users prefer LLM-produced falsehoods under different presentation formats, specifically marked versus unmarked truths and uninformative truth versus confident falsehoods. Four experiments using ChatGPT responses (with one- and two-phase designs) across 300 participants reveal a consistent tendency to favor unmarked and falsehood-based outputs, challenging assumptions about natural preference for truth. The findings imply that user feedback, when used to train or fine-tune models (e.g., via RLHF), could unintentionally promote false information unless preferences are verified or weighted appropriately. The work highlights ethical and practical implications for aligning LLM behavior with user preferences and suggests verification mechanisms or reliability assessments before incorporating feedback into training pipelines.

Abstract

While Large Language Models (LLMs) have become central tools in various fields, they often provide inaccurate or false information. This study examines user preferences regarding falsehood responses from LLMs. Specifically, we evaluate preferences for LLM responses where false statements are explicitly marked versus unmarked responses and preferences for confident falsehoods compared to LLM disclaimers acknowledging a lack of knowledge. Additionally, we investigate how requiring users to assess the truthfulness of statements influences these preferences. Surprisingly, 61\% of users prefer unmarked falsehood responses over marked ones, and 69\% prefer confident falsehoods over LLMs admitting lack of knowledge. In all our experiments, a total of 300 users participated, contributing valuable data to our analysis and conclusions. When users are required to evaluate the truthfulness of statements, preferences for unmarked and falsehood responses decrease slightly but remain high. These findings suggest that user preferences, which influence LLM training via feedback mechanisms, may inadvertently encourage the generation of falsehoods. Future research should address the ethical and practical implications of aligning LLM behavior with such preferences.

Paper Structure

This paper contains 24 sections, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Spreading Disinformation
  4. LLM Hallucinations Reduction
  5. Learning from Human Feedback
  6. Experimental Design
  7. Experiment A
  8. Experiment B
  9. Experiment C
  10. Experiment D
  11. Results
  12. Experiment A
  13. Experiment B
  14. Experiment C
  15. Experiment D
  16. ...and 9 more sections

Figures (5)

  • Figure 1: Example of a response pair from Experiment A.
  • Figure 2: Example of the second phase of Experiment B.
  • Figure 3: Example of a response pair from Experiment C.
  • Figure 4: An example of the second phase of Experiment D.
  • Figure 5: Example of ChatGPT's human feedback mechanism.