Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

WISE: Web Information Satire and Fakeness Evaluation

Gaurab Chhetri, Subasish Das, Tausif Islam Chowdhury

TL;DR

WISE benchmarks eight lightweight transformer models against two baselines to distinguish fake news from satire on a balanced 20,000-sample Fakeddit subset. Using stratified 5-fold cross-validation and a broad suite of metrics, MiniLM achieves the highest accuracy (87.58%), while RoBERTa-base attains the top ROC-AUC (95.42%). DistilBERT provides an excellent efficiency-accuracy trade-off, and overall the results show lightweight models can match or surpass larger baselines in real-world, resource-constrained deployments. The study delivers deployment-oriented insights for misinformation detection systems, illustrating how model choice trades off discrimination power, calibration, and efficiency.

Abstract

Distinguishing fake or untrue news from satire or humor poses a unique challenge due to their overlapping linguistic features and divergent intent. This study develops WISE (Web Information Satire and Fakeness Evaluation) framework which benchmarks eight lightweight transformer models alongside two baseline models on a balanced dataset of 20,000 samples from Fakeddit, annotated as either fake news or satire. Using stratified 5-fold cross-validation, we evaluate models across comprehensive metrics including accuracy, precision, recall, F1-score, ROC-AUC, PR-AUC, MCC, Brier score, and Expected Calibration Error. Our evaluation reveals that MiniLM, a lightweight model, achieves the highest accuracy (87.58%) among all models, while RoBERTa-base achieves the highest ROC-AUC (95.42%) and strong accuracy (87.36%). DistilBERT offers an excellent efficiency-accuracy trade-off with 86.28\% accuracy and 93.90\% ROC-AUC. Statistical tests confirm significant performance differences between models, with paired t-tests and McNemar tests providing rigorous comparisons. Our findings highlight that lightweight models can match or exceed baseline performance, offering actionable insights for deploying misinformation detection systems in real-world, resource-constrained settings.

WISE: Web Information Satire and Fakeness Evaluation

TL;DR

WISE benchmarks eight lightweight transformer models against two baselines to distinguish fake news from satire on a balanced 20,000-sample Fakeddit subset. Using stratified 5-fold cross-validation and a broad suite of metrics, MiniLM achieves the highest accuracy (87.58%), while RoBERTa-base attains the top ROC-AUC (95.42%). DistilBERT provides an excellent efficiency-accuracy trade-off, and overall the results show lightweight models can match or surpass larger baselines in real-world, resource-constrained deployments. The study delivers deployment-oriented insights for misinformation detection systems, illustrating how model choice trades off discrimination power, calibration, and efficiency.

Abstract

Distinguishing fake or untrue news from satire or humor poses a unique challenge due to their overlapping linguistic features and divergent intent. This study develops WISE (Web Information Satire and Fakeness Evaluation) framework which benchmarks eight lightweight transformer models alongside two baseline models on a balanced dataset of 20,000 samples from Fakeddit, annotated as either fake news or satire. Using stratified 5-fold cross-validation, we evaluate models across comprehensive metrics including accuracy, precision, recall, F1-score, ROC-AUC, PR-AUC, MCC, Brier score, and Expected Calibration Error. Our evaluation reveals that MiniLM, a lightweight model, achieves the highest accuracy (87.58%) among all models, while RoBERTa-base achieves the highest ROC-AUC (95.42%) and strong accuracy (87.36%). DistilBERT offers an excellent efficiency-accuracy trade-off with 86.28\% accuracy and 93.90\% ROC-AUC. Statistical tests confirm significant performance differences between models, with paired t-tests and McNemar tests providing rigorous comparisons. Our findings highlight that lightweight models can match or exceed baseline performance, offering actionable insights for deploying misinformation detection systems in real-world, resource-constrained settings.
Paper Structure (30 sections, 2 figures, 5 tables)

This paper contains 30 sections, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Fake News Detection
  4. Satire Detection
  5. Lightweight Transformer Models
  6. Benchmark Studies
  7. Methodology
  8. Dataset
  9. Dataset Preparation
  10. Models
  11. Lightweight Models
  12. Baseline Models
  13. Experimental Design and Training Methodology
  14. Data Splitting and Cross-Validation
  15. Preprocessing and Tokenization
  16. ...and 15 more sections

Figures (2)

  • Figure 1: Overview of the WISE framework
  • Figure 2: ROC curves for all evaluated models showing discriminative capabilities across different architectural designs. Curves are computed from out-of-fold predictions across 5-fold cross-validation.