Can Large Language Models Detect Rumors on Social Media?

TL;DR

This paper assesses whether large language models can detect rumors on social media by reasoning over the full propagation context (news and user comments). It introduces LeRuD, a framework that uses Rational Prompts for News, Conflicting Prompts for Comments, and a Chain-of-Propagation to manage long inputs and enable stepwise reasoning with a single LLM session. Empirical results on Twitter and Weibo show LeRuD achieving state-of-the-art zero-shot performance, with ablations confirming the importance of each component and the method’s capacity for early detection. The work also examines data leakage risks and provides explainable reasoning cues, highlighting practical implications for real-world rumor detection without requiring training data. Overall, LeRuD advances propagation-aware rumor detection by combining prompt engineering and structured reasoning over news and comments.

Abstract

In this work, we investigate to use Large Language Models (LLMs) for rumor detection on social media. However, it is challenging for LLMs to reason over the entire propagation information on social media, which contains news contents and numerous comments, due to LLMs may not concentrate on key clues in the complex propagation information, and have trouble in reasoning when facing massive and redundant information. Accordingly, we propose an LLM-empowered Rumor Detection (LeRuD) approach, in which we design prompts to teach LLMs to reason over important clues in news and comments, and divide the entire propagation information into a Chain-of-Propagation for reducing LLMs' burden. We conduct extensive experiments on the Twitter and Weibo datasets, and LeRuD outperforms several state-of-the-art rumor detection models by 3.2% to 7.7%. Meanwhile, by applying LLMs, LeRuD requires no data for training, and thus shows more promising rumor detection ability in few-shot or zero-shot scenarios.

Figures (5)

• Figure 1: Can LLMs detect rumors on social media?
• Figure 2: LLMs have trouble with too many comments.
• Figure 3: The procedure of the LeRuD approach. LeRuD divide the entire propagation information into a Chain-of-propagation with easy comment information for reasoning in each step. In each reasoning step, Rational and Conflicting Prompts, which are written in red italicized words, are performed. Key findings during LeRuD's reasoning process are highlighted with red underlined words.
• Figure 4: Early rumor detection results evaluated by accuracy with different detection deadlines.
• Figure 5: Accuracy of LeRuD under different numbers of comments. More comments, better performances.