Detecting Winning Arguments with Large Language Models and Persuasion Strategies

TL;DR

This work tackles the challenge of detecting persuasiveness in argumentative text by introducing Multi-Strategy Persuasion Scoring (MS-PS), a framework that decouples six rhetorical strategies and uses LLM-driven reasoning to produce per-strategy scores. MS-PS supports two aggregation modes: zero-shot AVG and a learned MS-PS-MLP that captures non-linear interactions among strategies, enabling robust, interpretable predictions across diverse domains. The approach is validated on the Winning Arguments (WA) dataset and its topic-annotated extension (TWA), as well as on Anthropic/Persuasion and Persuasion for Good (P4G), with MS-PS outperforming baselines and providing cross-domain generalization. A key contribution is the release of TWA for topic-aware analysis, plus extensive evidence that strategy-guided prompting enhances interpretability and robustness in argument quality assessment. Overall, the work advances persuasion detection by leveraging structured, strategy-aware reasoning and demonstrates practical benefits for argument analysis in real-world datasets and applications.

Abstract

Detecting persuasion in argumentative text is a challenging task with important implications for understanding human communication. This work investigates the role of persuasion strategies - such as Attack on reputation, Distraction, and Manipulative wording - in determining the persuasiveness of a text. We conduct experiments on three annotated argument datasets: Winning Arguments (built from the Change My View subreddit), Anthropic/Persuasion, and Persuasion for Good. Our approach leverages large language models (LLMs) with a Multi-Strategy Persuasion Scoring approach that guides reasoning over six persuasion strategies. Results show that strategy-guided reasoning improves the prediction of persuasiveness. To better understand the influence of content, we organize the Winning Argument dataset into broad discussion topics and analyze performance across them. We publicly release this topic-annotated version of the dataset to facilitate future research. Overall, our methodology demonstrates the value of structured, strategy-aware prompting for enhancing interpretability and robustness in argument quality assessment.

Figures (3)

• Figure 1: Overview of the MS-PS framework. Each of the two input messages is independently analyzed by a language model across six persuasion strategies. For each strategy, the model first generates an explanation assessing the presence of the strategy, followed by a 1–10 persuasiveness score. In the MS-PS-AVG variant (a), the more persuasive message is identified as the one with the higher average score. In the MS-PS-MLP variant (b), each message is represented by a feature vector consisting of the six individual scores plus their average, variance, and entropy, which is fed to a trained MLP classifier to predict which message is more persuasive.
• Figure 2: Description of the six persuasion strategies used in our experiments.
• Figure 3: Distribution of MS-PS strategy scores (1–10) across successful and non-successful messages. Each panel shows histograms for a different strategy, grouped by LLM model.