Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Analysing Differences in Persuasive Language in LLM-Generated Text: Uncovering Stereotypical Gender Patterns

Amalie Brogaard Pauli, Maria Barrett, Max Müller-Eberstein, Isabelle Augenstein, Ira Assent

TL;DR

This work tackles how user instruction attributes influence the style of LLM-generated persuasive language. It introduces a paired-prompt framework and an LLM-as-judge evaluation to quantify 19 categories of persuasive language across 13 LLMs and 16 languages. The key finding is robust gender-based divergence in persuasion: female-targeted prompts elicit more communal, affectionate, and pathos-driven language, while male-targeted prompts emphasize agentic, direct, and logos-driven language, with overall consistency across languages and intents. The framework's rigorous verification, including human annotations and cross-lingual assessments, demonstrates cross-context generalization and highlights ethical considerations for deploying persuasive AI systems and mitigating bias.

Abstract

Large language models (LLMs) are increasingly used for everyday communication tasks, including drafting interpersonal messages intended to influence and persuade. Prior work has shown that LLMs can successfully persuade humans and amplify persuasive language. It is therefore essential to understand how user instructions affect the generation of persuasive language, and to understand whether the generated persuasive language differs, for example, when targeting different groups. In this work, we propose a framework for evaluating how persuasive language generation is affected by recipient gender, sender intent, or output language. We evaluate 13 LLMs and 16 languages using pairwise prompt instructions. We evaluate model responses on 19 categories of persuasive language using an LLM-as-judge setup grounded in social psychology and communication science. Our results reveal significant gender differences in the persuasive language generated across all models. These patterns reflect biases consistent with gender-stereotypical linguistic tendencies documented in social psychology and sociolinguistics.

Analysing Differences in Persuasive Language in LLM-Generated Text: Uncovering Stereotypical Gender Patterns

TL;DR

This work tackles how user instruction attributes influence the style of LLM-generated persuasive language. It introduces a paired-prompt framework and an LLM-as-judge evaluation to quantify 19 categories of persuasive language across 13 LLMs and 16 languages. The key finding is robust gender-based divergence in persuasion: female-targeted prompts elicit more communal, affectionate, and pathos-driven language, while male-targeted prompts emphasize agentic, direct, and logos-driven language, with overall consistency across languages and intents. The framework's rigorous verification, including human annotations and cross-lingual assessments, demonstrates cross-context generalization and highlights ethical considerations for deploying persuasive AI systems and mitigating bias.

Abstract

Large language models (LLMs) are increasingly used for everyday communication tasks, including drafting interpersonal messages intended to influence and persuade. Prior work has shown that LLMs can successfully persuade humans and amplify persuasive language. It is therefore essential to understand how user instructions affect the generation of persuasive language, and to understand whether the generated persuasive language differs, for example, when targeting different groups. In this work, we propose a framework for evaluating how persuasive language generation is affected by recipient gender, sender intent, or output language. We evaluate 13 LLMs and 16 languages using pairwise prompt instructions. We evaluate model responses on 19 categories of persuasive language using an LLM-as-judge setup grounded in social psychology and communication science. Our results reveal significant gender differences in the persuasive language generated across all models. These patterns reflect biases consistent with gender-stereotypical linguistic tendencies documented in social psychology and sociolinguistics.
Paper Structure (46 sections, 3 equations, 20 figures, 10 tables)

This paper contains 46 sections, 3 equations, 20 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM Persuasion
  4. LLM Bias
  5. Framework
  6. Operationalizing Persuasive Language
  7. Evaluation via LLM-as-Judge
  8. Paired Prompts and Response Generation
  9. Analysing Gender Differences in Persuasive Language
  10. Setup
  11. Results
  12. Discussion
  13. Analysing Persuasive Language Differences in Intent Framing
  14. Analysing Language Differences
  15. Setup
  16. ...and 31 more sections

Figures (20)

  • Figure 1: Example of how Llama 3.3 varies persuasive language when the prompt specifies recipient gender.
  • Figure 2: Framework for evaluating differences in LLM-generated persuasive language under pairwise prompt instruction, including measures taken to verify findings.
  • Figure 3: Mean differences in persuasive language catagories $D_j$ between responses generated by Llama 3.3 under gender treatments. Grey; not significant, Blue: significantly more often male, Red; more often female.
  • Figure 4: Total Gender Gap across models, colored by model family. Top: messages; Bottom: arguments.
  • Figure 5: Persuasive language differences per model, under gender treatment. Grey: insignificant; Blue: significantly more male; Red: significantly more female.
  • ...and 15 more figures