PCoT: Persuasion-Augmented Chain of Thought for Detecting Fake News and Social Media Disinformation

TL;DR

The paper introduces Persuasion-Augmented Chain of Thought (PCoT), a zero-shot disinformation detection framework that injects persuasion knowledge into LLM reasoning via a two-stage process: first analyze persuasion strategies and then perform disinformation classification. It provides two novel post-cutoff datasets, MultiDis and EUDisinfo, to rigorously test on content unseen by models, and evaluates across five LLMs and five datasets, reporting a ~15% average F1 improvement over competitive baselines. The approach reveals that persuasion cues correlate with disinformation and that explicit explanations for persuasion decisions enhance robustness, especially for smaller models and longer texts. The work also presents comprehensive ablations, prompting-method comparisons, and a detailed annotation protocol, along with ethical considerations and data-sharing plans, highlighting the practical potential of persuasion-aware reasoning for scalable disinformation detection in real-world settings.

Abstract

Disinformation detection is a key aspect of media literacy. Psychological studies have shown that knowledge of persuasive fallacies helps individuals detect disinformation. Inspired by these findings, we experimented with large language models (LLMs) to test whether infusing persuasion knowledge enhances disinformation detection. As a result, we introduce the Persuasion-Augmented Chain of Thought (PCoT), a novel approach that leverages persuasion to improve disinformation detection in zero-shot classification. We extensively evaluate PCoT on online news and social media posts. Moreover, we publish two novel, up-to-date disinformation datasets: EUDisinfo and MultiDis. These datasets enable the evaluation of PCoT on content entirely unseen by the LLMs used in our experiments, as the content was published after the models' knowledge cutoffs. We show that, on average, PCoT outperforms competitive methods by 15% across five LLMs and five datasets. These findings highlight the value of persuasion in strengthening zero-shot disinformation detection.

Figures (12)

• Figure 1: The comparison between detecting disinformation with LLMs in a simple zero shot setting and detecting with PCoT and infused knowledge about persuasion.
• Figure 2: Persuasion strategies used in our experiments. A detailed description of the techniques associated with these strategies can be found in Appendix \ref{['sec:persuasion_strategies_techniques']}.
• Figure 3: Average percentage of persuasion strategies predicted across 5 models for disinformation (DIS) and reliable information (REL). ALL represents the percentage of instances with at least one detected persuasion strategy. Other abbreviations are explained in Figure \ref{['fig:persuasion_strategies_tax']}.
• Figure 4: Averaged percentage of persuasion strategies predicted across 5 models in predicted disinformation (DIS) and predicted reliable information (REL). ALL represents the percentage of instances with at least one detected persuasion strategy. Other abbreviations are explained in Figure \ref{['fig:persuasion_strategies_tax']}.
• Figure 5: Percentage of persuasion strategies predicted by GPT 4o mini for disinformation (DIS) and reliable information (REL). ALL represents the percentage of instances with at least one detected persuasion strategy. Other abbreviations are explained in Figure \ref{['fig:persuasion_strategies_tax']}.