Faking Fake News for Real Fake News Detection: Propaganda-loaded Training Data Generation

TL;DR

This paper targets the transfer gap between machine generated fake news detectors and human authored disinformation by generating propaganda informed training data. It introduces a two stage pipeline that first performs disinformation generation through salient sentence replacement guided by SCST and NLI, followed by propaganda augmentation via appeal to authority and loaded language, with intermediate pre training for domain adaptation. The authors release PropaNews, a 2256 article dataset, and show detectors trained on it outperform baselines on human written disinformation datasets such as Snopes and PolitiFact, with substantial F1 gains. The work also analyzes generation quality and ethical implications, offering a foundation for future multilingual expansion and broader propaganda technique coverage.

Abstract

Despite recent advances in detecting fake news generated by neural models, their results are not readily applicable to effective detection of human-written disinformation. What limits the successful transfer between them is the sizable gap between machine-generated fake news and human-authored ones, including the notable differences in terms of style and underlying intent. With this in mind, we propose a novel framework for generating training examples that are informed by the known styles and strategies of human-authored propaganda. Specifically, we perform self-critical sequence training guided by natural language inference to ensure the validity of the generated articles, while also incorporating propaganda techniques, such as appeal to authority and loaded language. In particular, we create a new training dataset, PropaNews, with 2,256 examples, which we release for future use. Our experimental results show that fake news detectors trained on PropaNews are better at detecting human-written disinformation by 3.62 - 7.69% F1 score on two public datasets.

Figures (5)

• Figure 1: Illustration of our self-critical sequence training. Given a corrupted input article $\Tilde{x}$, BART generates two sequences with Nucleus sampling and greedy decoding, respectively. The reward for each sequence is computed as the negative entailment probability $-P_{ent}$ as output from the NLI model.
• Figure 2: An example showing the NLI model predicts an entailment from the masked out sentence $y^*$ to the generated sentence $y'$.
• Figure 3: Performance comparison of RoBERTa-Large on the PolitiFact dataset when trained on Snopes and different size of PN-silver.
• Figure 4: Total counts of the propaganda techniques used in the human-written fake news we analyzed.
• Figure 5: Breakdown scores of our human evaluation. The x-axis denotes the counts of evaluators votes for a score, while the y-axis denotes different methods.