Can AI-Generated Persuasion Be Detected? Persuaficial Benchmark and AI vs. Human Linguistic Differences

TL;DR

The paper tackles whether AI-generated persuasion is easier to detect than human-written persuasion and shows the answer depends on the generation method. It introduces Persuaficial, a multilingual benchmark (~65k texts) produced through four generation approaches and four LLMs to probe detection across English and five other languages. A zero-shot detection analysis reveals that open-ended and intensified persuasion are more detectable, while subtle rewriting significantly reduces detector performance, with patterns consistent across languages. A comprehensive StyloMetrix-based linguistic analysis across 196 features identifies distinct AI- versus human-signaling traits, such as higher lexical diversity and content density in AI texts and greater function-word usage in human texts, providing guidance for interpretable and robust detectors. The work offers a valuable resource for automated persuasion detection, cross-lingual NLP, and the study of linguistic differences between human and AI-generated persuasive content, with code, prompts, and data released for reproducibility.

Abstract

Large Language Models (LLMs) can generate highly persuasive text, raising concerns about their misuse for propaganda, manipulation, and other harmful purposes. This leads us to our central question: Is LLM-generated persuasion more difficult to automatically detect than human-written persuasion? To address this, we categorize controllable generation approaches for producing persuasive content with LLMs and introduce Persuaficial, a high-quality multilingual benchmark covering six languages: English, German, Polish, Italian, French and Russian. Using this benchmark, we conduct extensive empirical evaluations comparing human-authored and LLM-generated persuasive texts. We find that although overtly persuasive LLM-generated texts can be easier to detect than human-written ones, subtle LLM-generated persuasion consistently degrades automatic detection performance. Beyond detection performance, we provide the first comprehensive linguistic analysis contrasting human and LLM-generated persuasive texts, offering insights that may guide the development of more interpretable and robust detection tools.

Figures (6)

• Figure 1: StyloMetrix features with the highest discriminative importance in distinguishing human-written from LLM-generated persuasive text.
• Figure 2: Prompt template used for persuasive texts generation with LLMs using the Paraphrasing Generation approach.
• Figure 3: Prompt template used for persuasive texts generation with LLMs using the Rewriting Generation - Subtle Persuasion approach.
• Figure 4: Prompt template used for persuasive texts generation with LLMs using the Rewriting Generation - Intensive Persuasion approach.
• Figure 5: Prompt template used for persuasive texts generation with LLMs using the Open-ended Generation approach along with the prompt template used to obtain a summarized, factual, and non-persuasive sentence input from human-written persuasive texts.