Communication is All You Need: Persuasion Dataset Construction via Multi-LLM Communication

TL;DR

This work tackles the scarcity and quality concerns of persuasion data by introducing a fully automated, multi-agent framework that generates and annotates persuasive dialogues. It combines six agent roles (dialogue generation, utterance quality monitoring, language refinement, persuasiveness annotation, global regulation, and postprocessing) across a GPT-3.5/GPT-4 backbone, incorporating NormBank taboos for ethically challenging scenarios and continuous, sentence-level perspective labeling. The approach demonstrates high-quality outputs with strong naturalness, coherent dialogic flow, and diverse persuasion strategies, while enabling strategy-controlled and multi-party dialogue generation. Validation includes extensive human-in-the-loop evaluations and model-assisted analyses, underscoring the framework’s scalability, generalizability, and potential to advance research in computational and social science domains, with careful attention to ethical use and misinformation risks. The resulting resource and methodology offer a practical pathway to study persuasive communication at scale and across complex contexts.

Abstract

Large Language Models (LLMs) have shown proficiency in generating persuasive dialogue, yet concerns about the fluency and sophistication of their outputs persist. This paper presents a multi-LLM communication framework designed to enhance the generation of persuasive data automatically. This framework facilitates the efficient production of high-quality, diverse linguistic content with minimal human oversight. Through extensive evaluations, we demonstrate that the generated data excels in naturalness, linguistic diversity, and the strategic use of persuasion, even in complex scenarios involving social taboos. The framework also proves adept at generalizing across novel contexts. Our results highlight the framework's potential to significantly advance research in both computational and social science domains concerning persuasive communication.

Figures (23)

• Figure 1: Overview of our data generation and annotation framework. Prior to dialogue generation, each agent is assigned specific tasks and given predefined stances to maintain throughout the conversation.
• Figure 2: Frequency Distribution of Persuasion Strategies in Independently Generated Dialogues. The Y-axis indicates the proportion of each strategy used within the model-generated dialogues. Each bar represents the strategy distribution of a single dialogue, organized by generation topic. Our framework adapts to various persuasion topics.
• Figure 3: Heatmap displaying the cosine similarity between strategy distributions across different dialogues. Each group of 5 dialogues belongs to the same topic, with the grid indicating the different topics.
• Figure A1: When all the agents are instantiated using GPT-3.5, the framework does not expand the conversations well, resulting in very short, question-answering-styled responses. The score in front of each utterance indicates the collective perspective change of each speaker compared to their initially assigned perspectives.
• Figure A2: Using GPT-4 for all the agents yields the best generation results in both language style and logical flow. A score of 1 associated with the last utterance of the persuader indicates that the persuader is fully persuaded by the persuadee.