A Generalizable Rhetorical Strategy Annotation Model Using LLM-based Debate Simulation and Labelling

TL;DR

This work tackles the scalability and generalizability challenges of labeling rhetorical strategies by introducing a framework that uses LLMs to generate and label synthetic debate data along a four-part typology (causal, empirical, emotional, moral). A controlled debate-generation pipeline and persona-based LLM annotation yield a high-quality, topic-controlled dataset used to fine-tune transformer classifiers with strong cross-domain performance. The authors validate the approach against human annotations and external corpora, and demonstrate two applications: improving persuasiveness prediction and tracing temporal and partisan shifts in U.S. presidential debates, finding a rise in affective over cognitive strategies over time. The study provides a scalable methodology for large-scale rhetorical analysis and offers resources for analyzing persuasive discourse across domains, while acknowledging language, cultural, and ethical considerations and proposing future extensions and dataset release.

Abstract

Rhetorical strategies are central to persuasive communication, from political discourse and marketing to legal argumentation. However, analysis of rhetorical strategies has been limited by reliance on human annotation, which is costly, inconsistent, difficult to scale. Their associated datasets are often limited to specific topics and strategies, posing challenges for robust model development. We propose a novel framework that leverages large language models (LLMs) to automatically generate and label synthetic debate data based on a four-part rhetorical typology (causal, empirical, emotional, moral). We fine-tune transformer-based classifiers on this LLM-labeled dataset and validate its performance against human-labeled data on this dataset and on multiple external corpora. Our model achieves high performance and strong generalization across topical domains. We illustrate two applications with the fine-tuned model: (1) the improvement in persuasiveness prediction from incorporating rhetorical strategy labels, and (2) analyzing temporal and partisan shifts in rhetorical strategies in U.S. Presidential debates (1960-2020), revealing increased use of affective over cognitive argument in U.S. Presidential debates.

Figures (11)

• Figure 1: Overview of our proposed framework.
• Figure 2: Distribution of LLM-assigned strategy scores for utterances conditioned to use (Positive), avoid (Negative), or use a different rhetorical strategy (Others) for each target strategy. Positive utterances were generated with prompts instructing the model to use the corresponding strategy; Negative utterances were prompted to avoid it; and Others includes utterances that were prompted for one of the other three strategies.
• Figure 3: Human-labeled strategy scores for dialogues instructed to use vs. avoid each persuasion strategy. Scores range from 0 (definitely not using) to 1 (definitely using), with 0.5 indicating uncertainty.
• Figure 4: Rhetorical strategies in U.S. Presidential debates. Left: temporal trends (1960–2020). Right: partisan differences.
• Figure 5: Prompt for opposing stance generation