What Signals Really Matter for Misinformation Tasks? Evaluating Fake-News Detection and Virality Prediction under Real-World Constraints
Francesco Paolo Savatteri, Chahan Vidal-Gorène, Florian Cafiero
TL;DR
The paper conducts an evaluation-driven analysis of fake-news detection and virality prediction under realistic constraints, comparing text-centric embeddings with lightweight numeric signals and sequence models on EVONS and FakeNewsNet. It demonstrates that textual content is a strong discriminator for fake-news detection, while numeric features aid stability when language models are unavailable; virality prediction remains substantially harder and highly sensitive to label construction. Non-linear structure appears more informative for virality than for fake-news detection, and RoBERTa vs. Mistral embeddings yield only minor differences, aiding reproducibility. The work emphasizes robust evaluation design, transparency, and practical guidance, providing reproducible splits and code to facilitate responsible deployment and further research in misinformation analytics.
Abstract
We present an evaluation-driven study of two practical tasks regarding online misinformation: (i) fake-news detection and (ii) virality prediction in the context of operational settings, with the necessity for rapid reaction. Using the EVONS and FakeNewsNet datasets, we compare textual embeddings (RoBERTa; with a control using Mistral) against lightweight numeric features (timing, follower counts, verification, likes) and sequence models (GRU, gating architectures, Transformer encoders). We show that textual content alone is a strong discriminator for fake-news detection, while numeric-only pipelines remain viable when language models are unavailable or compute is constrained. Virality prediction is markedly harder than fake-news detection and is highly sensitive to label construction; in our setup, a median-based ''viral'' split (<50 likes) is pragmatic but underestimates real-world virality, and time-censoring for engagement features is desirable yet difficult under current API limits. Dimensionality-reduction analyses suggest non-linear structure is more informative for virality than for fake-news detection (t-SNE > PCA on numeric features). Swapping RoBERTa for Mistral embeddings yields only modest deltas, leaving conclusions unchanged. We discuss implications for evaluation design and report reproducibility constraints that realistically affect the field. We release splits and code where possible and provide guidance for metric selection.