Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can Language Models Recognize Convincing Arguments?

Paula Rescala, Manoel Horta Ribeiro, Tiancheng Hu, Robert West

TL;DR

It is shown that LLMs perform on par with humans in these tasks and that combining predictions from different LLMs yields significant performance gains, surpassing human performance.

Abstract

The capabilities of large language models (LLMs) have raised concerns about their potential to create and propagate convincing narratives. Here, we study their performance in detecting convincing arguments to gain insights into LLMs' persuasive capabilities without directly engaging in experimentation with humans. We extend a dataset by Durmus and Cardie (2018) with debates, votes, and user traits and propose tasks measuring LLMs' ability to (1) distinguish between strong and weak arguments, (2) predict stances based on beliefs and demographic characteristics, and (3) determine the appeal of an argument to an individual based on their traits. We show that LLMs perform on par with humans in these tasks and that combining predictions from different LLMs yields significant performance gains, surpassing human performance. The data and code released with this paper contribute to the crucial effort of continuously evaluating and monitoring LLMs' capabilities and potential impact. (https://go.epfl.ch/persuasion-llm)

Can Language Models Recognize Convincing Arguments?

TL;DR

It is shown that LLMs perform on par with humans in these tasks and that combining predictions from different LLMs yields significant performance gains, surpassing human performance.

Abstract

The capabilities of large language models (LLMs) have raised concerns about their potential to create and propagate convincing narratives. Here, we study their performance in detecting convincing arguments to gain insights into LLMs' persuasive capabilities without directly engaging in experimentation with humans. We extend a dataset by Durmus and Cardie (2018) with debates, votes, and user traits and propose tasks measuring LLMs' ability to (1) distinguish between strong and weak arguments, (2) predict stances based on beliefs and demographic characteristics, and (3) determine the appeal of an argument to an individual based on their traits. We show that LLMs perform on par with humans in these tasks and that combining predictions from different LLMs yields significant performance gains, surpassing human performance. The data and code released with this paper contribute to the crucial effort of continuously evaluating and monitoring LLMs' capabilities and potential impact. (https://go.epfl.ch/persuasion-llm)
Paper Structure (26 sections, 5 figures, 4 tables)

This paper contains 26 sections, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Present Work.
  3. Related Work
  4. Demographics, beliefs, and persuasion.
  5. Argument Mining and Argument quality.
  6. Personalized misinformation and propaganda.
  7. Social sensing.
  8. Data
  9. PoliProp [PP].
  10. PoliIssues [IS].
  11. Methods
  12. LLMs considered.
  13. Prompting.
  14. Evaluation.
  15. Baselines/Benchmarks
  16. ...and 11 more sections

Figures (5)

  • Figure 1: Our approach to study LLMs' persuasiveness capabilities. We measure to which extent LLMs can reproduce human judgments on the quality and persuasiveness of arguments. Suppose LLMs can predict users' positions on stances (e.g., The death penalty should be legal) before and after reading a debate and judge who the better debater was. In that case, they would be well suited to power personalized misinformation and propaganda.
  • Figure 2: Prompt structure used in RQ1.
  • Figure 3: Prompt structure used in RQ2.
  • Figure 4: Prompt structure used in RQ3.
  • Figure 5: Inter-annotator agreement for different models in RQ2.