Exposing propaganda: an analysis of stylistic cues comparing human annotations and machine classification

TL;DR

This paper examines how propaganda differs stylistically from mainstream press by combining human annotations with machine-based detection on the Propagandist Pseudo-News (PPN) dataset. It uses an 11-label annotation scheme evaluated on a French subset, then compares human judgments with multiple classifiers, including RoBERTa- and CamemBERT-based models, CATS, and XGBoost, alongside VAGO-based linguistic analyses. Key findings show humans reliably distinguish propagandist from regular content, with Exaggeration and Pejorative cues strongly linked to propaganda; ML models achieve near-perfect accuracy on Ukraine-topic data, with SHAP and VAGO providing explainability and highlighting syntactic and punctuation as important cues. The work underscores topic-specific strengths and cautions about generalization, bias, and the need for multilingual, topic-general propaganda detection resources for robust, real-world deployment.

Abstract

This paper investigates the language of propaganda and its stylistic features. It presents the PPN dataset, standing for Propagandist Pseudo-News, a multisource, multilingual, multimodal dataset composed of news articles extracted from websites identified as propaganda sources by expert agencies. A limited sample from this set was randomly mixed with papers from the regular French press, and their URL masked, to conduct an annotation-experiment by humans, using 11 distinct labels. The results show that human annotators were able to reliably discriminate between the two types of press across each of the labels. We propose different NLP techniques to identify the cues used by the annotators, and to compare them with machine classification. They include the analyzer VAGO to measure discourse vagueness and subjectivity, a TF-IDF to serve as a baseline, and four different classifiers: two RoBERTa-based models, CATS using syntax, and one XGBoost combining syntactic and semantic features.

Figures (9)

• Figure 1: Description of the 11 labels used for the annotation task.
• Figure 2: Topic distribution of articles from the annotated corpus.
• Figure 3: Mean scores and agreement by label (error bars=standard error of the mean).
• Figure 4: Correlation matrix of the 11 labels used for human annotations.
• Figure 5: Pearson correlations between the VAGO mean opinion score per article and the mean scores for labels "Subjective" (left) and "Descriptive" (right). Blue data points correspond to regular articles while red data points correspond to alternative press articles.