AutoPersuade: A Framework for Evaluating and Explaining Persuasive Arguments

TL;DR

AutoPersuade presents a modular workflow to study what makes persuasive arguments effective and why, by collecting large argument-response datasets, learning latent, interpretable persuasive features via the SUN supervised semi-NMF topic model, and estimating causal effects of these features on persuasiveness. The SUN model ties argument content and responses through shared latent topics, enabling out-of-sample prediction for new arguments and AMCE-style causal interpretation. The veganism case study demonstrates the framework’s ability to identify positively and negatively associated topics, and to generate new arguments with partial success, though optimizing for the single best argument remains challenging. The work advances interpretable causal analysis of textual persuasion and offers a foundation for data-driven generation of persuasive content, while acknowledging ethical considerations and the importance of robust response measures.

Abstract

We introduce AutoPersuade, a three-part framework for constructing persuasive messages. First, we curate a large dataset of arguments with human evaluations. Next, we develop a novel topic model to identify argument features that influence persuasiveness. Finally, we use this model to predict the effectiveness of new arguments and assess the causal impact of different components to provide explanations. We validate AutoPersuade through an experimental study on arguments for veganism, demonstrating its effectiveness with human studies and out-of-sample predictions.

Figures (7)

• Figure 1: The AutoPersuade workflow. After generating a collection of arguments, we collect reactions from respondents. Using these reactions and arguments, we fit the SUN topic model that discovers latent topics that simultaneously describe the documents and their persuasiveness. We then use the output from the SUN model to estimate causal effects from the existing sample and to predict the persuasiveness of new arguments. While our validation studies confirm the causal estimates of our case study, the current approach using average marginal component effects of topics appears less well-suited to finding the best argument. Improving the identification of optimal arguments in the last step might be the subject of future studies.
• Figure 2: Out-of-sample predictive accuracy of SUN topic model for different parameter choices, as well as benchmark models on the training data. Results were calculated using 10-fold cross-validation.
• Figure 3: Estimated effects of discovered latent topics on the persuasiveness score of arguments for veganism. Refer to Table \ref{['tab:Veganism_Estimation']} in the Appendix for more details.
• Figure 4: Out-of-sample predictive accuracy of SUN topic model for additional hyperparameter choices, as well as benchmark models on the training data. Results were calculated using 10-fold cross-validation.
• Figure 5: Topic coherence and out-of-sample predictive accuracy on 20% holdout of the training data for parameter choices $\alpha=0.5$ and the number of topics $J=10$.