Breaking Event Rumor Detection via Stance-Separated Multi-Agent Debate

TL;DR

This work tackles rumor detection during breaking events, where data scarcity and controversy hinder traditional methods. It introduces S2MAD, a stance-separated multi-agent debate framework that uses LLMs to first segregate comments into supportive and opposing sets, then generates initial opinions with prompts tailored to whether the claim is subjective or objective, followed by a two-agent debate and a judge when needed to decide veracity. The framework is validated in zero-shot experiments on Twitter-COVID19 and Weibo-COVID19, showing consistent gains over strong baselines and highlighting the importance of stance separation, claim categorization, and debate. The approach demonstrates practical value for rapid, diverse, and robust rumor verification in breaking news contexts.

Abstract

The rapid spread of rumors on social media platforms during breaking events severely hinders the dissemination of the truth. Previous studies reveal that the lack of annotated resources hinders the direct detection of unforeseen breaking events not covered in yesterday's news. Leveraging large language models (LLMs) for rumor detection holds significant promise. However, it is challenging for LLMs to provide comprehensive responses to complex or controversial issues due to limited diversity. In this work, we propose the Stance Separated Multi-Agent Debate (S2MAD) to address this issue. Specifically, we firstly introduce Stance Separation, categorizing comments as either supporting or opposing the original claim. Subsequently, claims are classified as subjective or objective, enabling agents to generate reasonable initial viewpoints with different prompt strategies for each type of claim. Debaters then follow specific instructions through multiple rounds of debate to reach a consensus. If a consensus is not reached, a judge agent evaluates the opinions and delivers a final verdict on the claim's veracity. Extensive experiments conducted on two real-world datasets demonstrate that our proposed model outperforms state-of-the-art methods in terms of performance and effectively improves the performance of LLMs in breaking event rumor detection.

Figures (3)

• Figure 1: Given an input claim and its related comments, we first categorize the comments based on their stance towards the claim into supporting comments $P$ and opposing comments $N$. Furthermore, we classify the claim based on whether it is a subjective expression or not, and design corresponding guiding prompts for different types of claim (subjective and non-subjective). Utilizing LLMs as agents, we evaluate the truthfulness of the claim based on these different stances and generate an initial opinion. Subsequently, we conduct a multi-agent debate. If the agents do not reach a consensus after a maximum of $M$ rounds, a judge is designed to derive the final prediction.
• Figure 2: Comparison of different numbers of debate rounds in our proposed S2MAD approach using GPT-3.5 turbo.
• Figure 3: Early detection performance is evaluated at different checkpoints based on the post count.