Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Multi-Label Dataset of French Fake News: Human and Machine Insights

Benjamin Icard, François Maine, Morgane Casanova, Géraud Faye, Julien Chanson, Guillaume Gadek, Ghislain Atemezing, François Bancilhon, Paul Égré

TL;DR

This work tackles the multidimensional problem of fake news in French by introducing OBSINFOX, a 100-document multi-label corpus annotated by eight human raters across 11 labels to capture cues distinguishing human judgments from machine predictions. It combines topic/genre analysis (GATE Cloud with mBERT) and a rigorous annotation study, reporting moderate inter-annotator agreement and meaningful label relationships, particularly among Subjective, Opinions, Exaggeration, and Fake News. By applying VAGO and a neural variant VAGO-N, the authors show that linguistic subjectivity correlates with certain labels more strongly than with Fake News itself, highlighting the nuanced role of subjectivity in detecting misinformation. The dataset, while modest in size, provides a valuable resource for decoding how humans and machines perceive fake news in French and points to avenues for richer label sets and satire-aware detectors; the data is publicly available for further research.

Abstract

We present a corpus of 100 documents, OBSINFOX, selected from 17 sources of French press considered unreliable by expert agencies, annotated using 11 labels by 8 annotators. By collecting more labels than usual, by more annotators than is typically done, we can identify features that humans consider as characteristic of fake news, and compare them to the predictions of automated classifiers. We present a topic and genre analysis using Gate Cloud, indicative of the prevalence of satire-like text in the corpus. We then use the subjectivity analyzer VAGO, and a neural version of it, to clarify the link between ascriptions of the label Subjective and ascriptions of the label Fake News. The annotated dataset is available online at the following url: https://github.com/obs-info/obsinfox Keywords: Fake News, Multi-Labels, Subjectivity, Vagueness, Detail, Opinion, Exaggeration, French Press

A Multi-Label Dataset of French Fake News: Human and Machine Insights

TL;DR

This work tackles the multidimensional problem of fake news in French by introducing OBSINFOX, a 100-document multi-label corpus annotated by eight human raters across 11 labels to capture cues distinguishing human judgments from machine predictions. It combines topic/genre analysis (GATE Cloud with mBERT) and a rigorous annotation study, reporting moderate inter-annotator agreement and meaningful label relationships, particularly among Subjective, Opinions, Exaggeration, and Fake News. By applying VAGO and a neural variant VAGO-N, the authors show that linguistic subjectivity correlates with certain labels more strongly than with Fake News itself, highlighting the nuanced role of subjectivity in detecting misinformation. The dataset, while modest in size, provides a valuable resource for decoding how humans and machines perceive fake news in French and points to avenues for richer label sets and satire-aware detectors; the data is publicly available for further research.

Abstract

We present a corpus of 100 documents, OBSINFOX, selected from 17 sources of French press considered unreliable by expert agencies, annotated using 11 labels by 8 annotators. By collecting more labels than usual, by more annotators than is typically done, we can identify features that humans consider as characteristic of fake news, and compare them to the predictions of automated classifiers. We present a topic and genre analysis using Gate Cloud, indicative of the prevalence of satire-like text in the corpus. We then use the subjectivity analyzer VAGO, and a neural version of it, to clarify the link between ascriptions of the label Subjective and ascriptions of the label Fake News. The annotated dataset is available online at the following url: https://github.com/obs-info/obsinfox Keywords: Fake News, Multi-Labels, Subjectivity, Vagueness, Detail, Opinion, Exaggeration, French Press
Paper Structure (6 sections, 6 figures, 2 tables)

This paper contains 6 sections, 6 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Corpus and Labels
  3. Topic and Genre Analysis
  4. Human Annotations
  5. Ascriptions of "Fake News"
  6. Conclusion

Figures (6)

  • Figure 1: Description of the 11 labels selected for the annotation task.
  • Figure 2: Topic and genre distribution in the corpus.
  • Figure 3: Persuasion techniques by genre.
  • Figure 4: Mean scores and agreement by label (error bars=standard error of the mean).
  • Figure 5: Correlation matrix between the 11 labels (top), and percentage of a row label satisfying a column label (bottom).
  • ...and 1 more figures